Processing system and method for intermediate code data stream originated in object-oriented language program or multi-media data stream

Description: TECHNICAL FIELD [Detailed description of the invention]

【0001】

[Technical field to which the invention belongs]

The present invention relates to the fields of digital computer hardware and software, and particularly to object-oriented multimedia architectures for processing multimedia data in real time.

【0002】

Background of the Invention

Multimedia systems combine various sources of information such as audio, graphics, video, images, audio and full motion video with a variety of applications. In general, multimedia refers to a new combination of three historically separate industrial disciplines: computing, telecommunications, and broadcasting. What characterizes multimedia systems is the incorporation of "continuous" media such as audio, video and video. Distributed multimedia systems provide relatively long-term continuous data transmission, media synchronization, mass storage, and other advanced features, such as playout of an image signal stream from a remote camera. Technology is required.

【0003】

For multimedia systems, new and improved uses have been found for a variety of applications. For example, set top Box) and interactive televisions, multimedia libraries (databases), portable computers, gaming machines, progressive portable digital devices, mobile terminals, and worldwide web pages. The large amount of data used in multimedia applications and the need to process data in real-time or near real-time poses technical challenges for both hardware and software system designers. These challenges are addressed at the forefront of a variety of different technical disciplines, such as compression algorithms and special purpose hardware processors. The complexity of multimedia applications is a burden on all components of a computer system. Processing of multimedia data requires extremely substantial processing power for performing graphics, data conversion, data compression, and the like. It is clear that the architecture for this must have extremely large bus bandwidth and efficient I / O. The multimedia operating system should also support new data types, real-time scheduling processing, and fast interrupt processing.

【0004】

Historically, data processing technology has evolved from an environment in which only character data is embedded. Computer graphics and other multimedia elements are relatively new in this area. Traditional computer systems are also characterized by linear or "flat" processing. The computer sequentially executed a predetermined series of instructions that were operated on based on a collection of characters. In many cases batch processing was used. It is also important to be aware that computer processors have historically been general-purpose processors. That is, the computer was designed to be able to perform any specific function realized by the application program. The only relatively exception was the "dedicated processor" developed to meet special needs. Therefore, traditional computer architectures were designed to execute any set of instructions written by a programmer. Such specific applications were not known to system designers in advance, and therefore the system architecture could not be optimized for specific applications. For this reason, general-purpose computers have flexibility in their applications, but this has caused performance limitations.

【0005】

Advances in multimedia applications have stimulated development in several different hardware and software areas. For example, the large amount of data required in multimedia applications has led to advances in compression / decompression technology. We have witnessed the development of the JPEG and MPEG standards, which are standards for compressing image and audio data. MPEG2 is a video coding standard currently realized in many computers. Most recently, we have witnessed software improvements for "streaming" multimedia data. For example, the asynchronous image model of Java (registered trademark) has made it possible to send image data to the Internet. This means that the "applet" on the client machine can start working on the image once the data is available. Without this ability, the user would have to wait for the data of the operation to be displayed until the download of multimedia data is completed, but with this ability, the data can be used as it is. Although such a function can be exhibited, the Java environment is still not a real-time operation, and its interactivity is limited. This limitation in processing multimedia data is partly due to the quality of the data, and partly because many of the required operations such as decompression and graphics operations are computationally intensive. Because of the fact. There were several advantages to using a faster microprocessor. In fact, one of the reasons for the remarkable spread of the World Wide Web must be the advancement of microprocessor technology. Nevertheless, Intel X86.P5. Current microprocessors like the P6 are still general purpose processors. Such processors, if any, are optimized for multimedia applications only in very limited cases. In order to bring about further improvements, it is necessary not only to use special hardware such as a coprocessor, but also to improve the architecture so that the hardware can be organized more efficiently.

【0006】

Some special hardware has been developed to address this need. For example, digital signal processing (DSP) integrated circuits are known to process voice data in real time. DSP devices may be implemented in add-on "soundboards" for upgrading personal computers. Video Random Access Memory (VRAM) devices are known to improve the refresh rate of screen displays. In personal computers, VRAM is often realized on a "video board", which is a circuit board that improves screen refresh by improving the display bandwidth. However, since such a dedicated processor and memory are provided in the context of the conventional general-purpose processor architecture, their advantages are also limited. In industry terms, this type of coprocessor is "bolted" to an existing architecture. In such a system, it is usually processed as a flat data stream under the control of one general-purpose processor. Therefore, there is still a need for new architectures that make more efficient use of various hardware and software technologies to process multimedia applications in real time.

【0007】

We have described how the growth of multimedia applications and content and the need for real-time interactive capabilities have increased the need for more powerful computing power and wider data communication bandwidth to transmit and process multimedia data. I've been doing it. Increasing communication channel bandwidth with existing huge finite bandwidth communication infrastructures, such as increasing the use of ISDN and T-1 lines, can help with communications, but such infrastructures are expensive. And it can still be used only in a limited area. Also, the processing power of the client machine is still limited, even if communication bandwidth is practically available.

【0008】

Technological advances in hardware capabilities, compression algorithms, etc. are somewhat helpful to each, but these fragmentary advances limit the level of improvement as they do not work together. Not too much. One example of this is the fact that various "accelerators" such as video boards are simply bolted to older architectures that rely on just one general purpose processor. The conventional concept of parallel computer processing is not portable, so it is difficult to actually apply it to a multimedia environment.

【0009】

[Problems to be Solved by the Invention]

Therefore, an object of the present invention is to provide an intermediate code data stream derived from an object-oriented language program, or a processing system and method for a multimedia data stream, which enables more efficient data processing under a multimedia environment. is there.

【0010】

[Means for solving problems]

The present invention has been achieved by reassessing the entire hardware and software environment in an effort to bring about substantial advances in multimedia data processing. According to one embodiment of the invention, the improved multimedia software and hardware architecture takes full advantage of the object-oriented paradigm and covers everything from data communications to real-time multimedia content performance. It supports this paradigm throughout. In this new multimedia architecture, software objects are brought down to the execution level as threads running concurrently on a lightweight process (LWP) or multiple processors. Traditionally, object-oriented technology has been used to program or authorize multimedia applications. To reduce the data communication bandwidth required and make it portable across various hardware platforms, these object-oriented applications are compiled and low-level pseudo-languages of intermediate code (eg, Java®). It was converted to bytecode), and this intermediate code was translated at runtime and converted into an operation code that functions on the target processor, which is equivalent to the intermediate code.

【0011】

In the novel architecture of the present invention, the original "object" and thread defined in the source program are restored from the input intermediate code stream in the new type of "virtual machine" mechanism included therein. This new virtual machine contains a class library for instantiating objects and methods used in the source program and restores the "threads" of the original application program. On the hardware side, there will be multiple processors under the control of the microkernel operating system. One or more hardware processors are designed or optimized to perform certain multimedia functions or methods, such as decompression of compressed audio data or calculation of image object rotation. The virtual machine queries the class library to correlate objects and threads restored from the bytecode stream with a list of currently available hardware resources. This virtual machine organizes object methods and program threads to run concurrently on the most appropriate processor for the task wherever possible. This architecture delivers the performance of parallel processing machines while maintaining the portability of pseudocode programs across different platforms. In addition, the virtual machine can automatically select and use the hardware resources it can use, including cores (microprocessors) that are not yet commercially available. In this case as well, the portability of the application program is not impaired.

【0012】

Objects or threads that require these features that are directly supported by the hardware are sent to the corresponding processor for their execution. Other processors or "cores" may be provided to directly execute this pseudo-language or intermediate code (eg Java bytecode). Transfer a "flat" thread or code segment to a "native" processor (eg, Sun Microsystems' SUN Pico-Java engine) to which these codes fit to execute without translating it into operational code. You can also do it. Therefore, in order to run a serial stream of pseudo-language or intermediate code instructions on the desired processor, this new architecture introduces the principles of parallel processing and an object-oriented paradigm rather than simply translating as in the prior art. I tied it up and speeded up the execution. In other words, it extended it to hardware while preserving the benefits of an object-oriented environment.

【0013】

According to the object-oriented paradigm, threads and objects are executed by "calls" to resources, here selected hardware processors. Therefore, one embodiment of the present invention improves the "integrity" of software processes and hardware for execution. The system contains decentralized components (processors or cores in current technology), each of which is a virtual machine, microkernel, and hardware arranged to perform tasks associated with a particular object class method. In the sense that it is object-oriented throughout, including the hardware. This new virtual machine program can be programmed for any hardware platform, preserving the portability benefits of the object environment. The virtual machine also fits into all of the currently available platform hardware resources, thus providing cost-effectiveness flexibility.

【0014】

Preferred embodiments of the present invention include a virtual machine program stored in memory. This virtual machine has access to an object class-defined stored library, as well as hardware that acts as a buffer for receiving incoming pseudo-language instructions, or intermediate code streams, such as bytecode. Can also be accessed. This virtual machine checks that all the classes referenced in the bytecode are in the stored library, and removes all the missing classes from network resources that can be the source of the intermediate code stream. Includes a class loader to download.

【0015】

Objectives, features, and advantages of the present invention will become even more apparent with reference to the sections of embodiments of the invention described below, along with the accompanying drawings.

【0016】

BEST MODE FOR CARRYING OUT THE INVENTION

Figure 1 is a data flow diagram of a virtual machine implementation of an object-oriented programming language. Sun Microsystems' Java® environment is an example of this implementation. This Java environment contains several different parts. First, there are Java programming languages that belong to the C language family. It has a C ++-like object semantics, but with additional object features and a garbage collection feature that collects unwanted objects. Java has become famous for its support for dynamic binding, which determines where methods and member variables are referenced at runtime, for its ability to load code at runtime, and for its ability to execute code safely. In Figure 1, block 10 represents a program consisting of Java language source code.

【0017】

Java language programs can be compiled to run on various machine architectures. However, the second core piece of the Java environment is the virtual machine (abbreviated as VM) shown below the dotted line 30 in Figure 1 (Virtual Machine API). The Java Virtual Machine provides an abstract processor architecture. This virtual machine can be implemented by software on a variety of operating systems and hardware. The Java source code program 10 is compiled into a set of pseudo-code intermediate code, a set of "bytecode" based on the Java Virtual Machine instruction set. This virtual machine must be ported to each destination platform. In virtual machine 40, the runtime interpreter performs tasks similar to emulation. That is, the runtime interpreter translates the Java instruction set bytecode into operation code that can be executed on the desired platform hardware.

【0018】

As part of this process, the interpreter also loads code into the Fundation Class Library, the Java class definition, which is another component of the Java environment, by invocation. This step is a basic step in an object-oriented paradigm where a program object is just an instance of a predefined class. Other libraries that can be used by the interpreter do not necessarily have to be implemented, but they give Java various features. For example, a VM generally contains a library of C-code TCP / IP features. Anyway, once the "translated" code is obtained, it can then make a call to the desired platform operating system 44 and execute that code. At this time, the operating system 44 interfaces with the processor hardware 46. In this prior art environment, a program in the original object-oriented language is run in serial flat code for execution on the desired platform. Convert to code). Since the same source code program 10 can be executed on any platform on which the virtual machine 40 is realized, portability has been obtained in the conventional environment. For more information on implementing the Java Virtual Machine, see, for example, the article "Implementing the Java Virtual Machine-Java's Complex Instruction Set Can Be in Software; Microprocessor Report, March 25, 1996, p. 12" by Brian Case. "It is described in. Although portable is provided by Java, running a program on a client machine relies on running serial flat code on a conventional general-purpose processor.

【0019】

FIG. 2 is a conceptual diagram showing a novel multimedia architecture according to the present invention. In FIG. 2, data (here, program code and data) flows from the left side to the right side of the drawing. Data entered in the form of object-oriented pseudo-code such as Java bytecode or intermediate code is received and executed. The bytecode stream can be locally present (inside the same machine) or transferred from the server to the client machine via a local area network or wide area network. This input data stream can also be transmitted via a worldwide network such as the Internet. Bytecode streams are compiled with a predefined virtual machine API. Virtual machine 102 has a runtime interpreter, a garbage collection mechanism, and other mechanisms described below. This input stream needs to be object-oriented pseudo-code or intermediate code, but is not limited to Java "byte code". The system of the present invention is particularly useful when used with novel multimedia programming languages such as MPEG-4 "Syntax Description Language".

【0020】

"MPEG-4" is a coding standard that has emerged as a support for new (especially content-based) methods for communicating, accessing, and manipulating digital audio-visual data. This standard is not yet widespread and will take several years to be fully established. Nevertheless, the concept is clear: it is for the coding of the audio-visual object itself. In traditional "flat" representations, image and audio frames do not distinguish between objects. All parts of this frame have equal priority, and when encoded, it is possible, for example, to "steal" bits from foreground objects where detailed background data is more important. The MPEG-4 community believes that some form of object-based coding is needed to reach new levels of performance and interaction capabilities.

【0021】

The intended MPEG-4 functionality easily supports interactive functionality, high compression rates, and / or universal accessibility. Key is the concept of content-based interaction capabilities, the possibility of interacting with meaningful objects in the audio-visual scene. As a concrete example, imagine a car running in a movie scene on the screen. Normally, in order to watch a movie, even if there is no interaction, a significant amount of data (entire frame) must be continuously compressed, transferred, decompressed, displayed, and refreshed in real time. It doesn't become. That is, the entire frame data is processed. However, MPEG-4 recognizes that most of the data is static and probably unimportant. However, objects such as car wheels change (rotate). Therefore, in MPEG-4 coding, the wheel of each car is treated as an object (software object corresponding to a physical object). The MPEG-4 Syntax Description Language (MSDL) provides templates that support the interaction of both natural and artificial objects. Wheel objects described by MSDL, which are used instead of processing vast amounts of pixel data to display a car, are simply referred to as the "spin" method. See ISO / IEC JTC1 / SC29 / WG11 / N0998 Coding of Moving Pictures and Associated Audio Information (MPEG-4 Proposal Package, July, 1995). The system of the present invention disclosed herein can be easily applied to the processing of this MSDL-encoded multimedia application.

【0022】

Seeing FIG. 2 again, the virtual machine 102 also includes at least one base class library (a set of software object definitions) in the source code language, and preferably contains an extension class library. The Java foundation class has, for example, all about 15 different packages included in the Java Development Kit. For example, the JDK package has java.applet, java.awp, java.io, java.net, and java.image. As a specific example, one of the applet interfaces is used to play back audio clips. ("Interface" is a special type of Java class.) The package java.awp contains the interfaces and classes needed to build user interfaces and on-screen graphics. Examples include buttons, labels, panels, colors and the like. The image package class handles the manipulation of pixel images.

【0023】

Extension classes are more specialized classes for implementing things like commercial transactions, cryptography, banking, database APIs, and various graphics operations on the Internet, many of which are currently under construction. As described below, the multimedia architecture of the present invention can take full advantage of new extension classes in the future, and most importantly, this architecture is used to call extension class objects. The point is that a special hardware processor can be installed for execution. Such processors may include, for example, dedicated special purpose processors, RISC cores, native engines, or other types of unfinished processors. Advances in new extension classes can motivate hardware designers to provide specialized processors for the execution of these class methods.

【0024】

In its operation, the virtual machine 102 reconstructs the software object shown in the source code by mapping all the input bytecodes to (a) fields, (b) objects, and (c) threads. .. Fields are variables that are not fully disclosed at compile time. The virtual machine decomposes this field before it becomes accessible. An object is, of course, an instance of a class defined in a class library.

【0025】

A thread, also called a lightweight process, is an individual control stream that can execute its instructions individually, and a multithreaded process can execute many tasks at the same time. In other words, many threads can be executed as one process. (The terms "thread" and "lightweight process (LWP)" are sometimes used interchangeably, but in some implementations there is a one-to-one relationship between each thread and one LWP. However, the present invention seeks a method of maximizing concurrency in execution. The present invention intends to schedule individual execution for each thread, as described in detail below. The Java language implements thread classes. The base and extension classes are part of the virtual machine, which allows them to be successfully rebuilt. This is because these classes contain the same information as the library that the compiler used to generate pseudo or bytecode. This virtual machine also has a class loader mechanism that downloads arbitrary classes that are used in pseudocode but cannot be found in the library.

【0026】

The "threaded" or multithreaded programming concept is well known in the prior art as a programming methodology for improving application throughput, application responsiveness, and program structure, as well as efficiently developing usable parallel processors. there were. The use of threads for research experiments became widespread in universities and research institutes after a while. However, it has only been in the last few years that it has entered the stage of implementation in commercial operating systems. There are currently three main sets of multithreaded libraries. That is, UNIX, OS / 2, and Windows NT. It is clear that multithreading technology has made it possible to obtain widely improved performance by taking advantage of multiprocessor (SMP) machines. On a single processor machine, these advantages are less obvious, but can still be dramatic in applications where many tasks must be performed at the same time.

【0027】

In a typical multitasking operating system such as VMS or UNIX, there is a farm partition line between user space and kernel space. This division is enhanced by hardware. The user program is executed in the user space. User space includes user code, global data, program counters, and stack pointers. The data that a program or process can directly access and modify is limited to the data in user space. When the user program needs something from the kernel (for example, reading a file or reading the current time), the user program must make a system call. This system call is a library function that sets up some arguments and then executes a spatial interrupt instruction. This instruction causes the hardware to interrupt the kernel and then take control of the machine. The kernel determines what action is required and decides whether to perform that action. Finally, the kernel performs the desired task and returns some information back to the user process.

【0028】

Since the operating system has full control over I / O memory, processors, etc., it is necessary to maintain data for each process in progress. From this data, the operating system can recognize what the state of the process is, that is, which files are open, which users are running them, and so on. Therefore, the concept of multitasking processes is extended to the kernel, and this information is maintained in the process structure in the kernel. In a multitasking system, many processes can be run in parallel. Each process has its own memory space and its own stack program counter and the like. Neither of the two processes can see or change each other's memory without setting up a special shared memory segment. Therefore, each program has one stack, one program counter, and one set of CPU registers per process. Therefore, each of these programs or processes can only do one thing at a time, which is single-threaded.

【0029】

A process can do the same thing by running multiple threads, just as a multitasking operating system can do multiple things at the same time by running multiple processes. Each thread is a different flow of control that can execute its instructions individually, so a multithreaded process can perform many tasks at the same time. For example, one thread can run the GUI, while a second thread can perform I / O and a third thread can do the calculations. A thread is similar to a process, but it has a set of data, code, kernel state, and CPU registers. However, processes are kernel-level entities that have things like virtual memory maps, file descriptors, user IDs, etc., and each process has its own collection of these. Therefore, the only way for a program to access data in a process structure and query or change its state is by calling the system.

【0030】

In the prior art, threads were primarily user-level entities. The thread structure existed in user space and could be accessed directly by thread library calls. This thread library call is just a user-level function. All registers (stack pointers, program counters, etc.) were part of a thread, and each thread had its own stack, but the code it executed was not part of the thread. The actual code (functions, routines, signal handlers, etc.) was global and could be executed in any thread. The important point is that all threads in a process share the state of that process. Threads exist in exactly the same memory space, refer to the same function, and use the same data. When one thread changes a process variable, all other threads will recognize the change the next time it accesses it. When one thread opens a file for reading, all other threads can read from it. Such an organization raises the need for some synchronization and scheduling, which has the advantage of being able to perform many tasks without the kernel overhead of actual process switching.

【0031】

Looking back at Figure 2, the reconstructed objects and threads follow operating system API 104. Operating system 106 includes a real-time microkernel operating system that transfers objects and threads and Java bytecode to a multi-core or parallel processor 112, as described in detail later. An important aspect of the present invention is to improve the "alignment" between program threads and "on the fly" hardware resources. Since these threads run simultaneously, they can access the same user memory space, and programmers must be careful to harmonize or "synchronize" their operations. One thread cannot read data at the same time while another thread is modifying the data. Therefore, threads need to have exclusive access to objects, at least temporarily. Several techniques are known for thread synchronization. In the simplest case, Mutual Exclusion Lock or mutex causes only one thread to temporarily execute a given piece of code, such as code that modifies global data. In addition, conditional variables are known that prohibit thread execution until a given conditional test is true.

【0032】

In Java, for example, threads are supported as an integral part of the language. Java has a set of synchronization methods defined in the class protocol. When a class with synchronized methods is "initialized", a new object is given a "monitor". To call a synchronization message on an object, the thread needs to monitor that object. If it is possible to acquire a monitor, the thread enters the synchronization method, but no other thread can call the synchronization method in the object while it has the monitor. If a thread calls a synchronization method within an object and the object's monitor is acquired by another thread, the calling thread is blocked until the other thread releases the monitor. When the original thread exits the synchronization method to acquire the monitor, ownership of the monitor is transferred to the blocked thread, which is then able to enter the blocked method. Virtual machine 102 identifies application layer threads and separates them for execution in parallel, as described below.

【0033】

The virtual machine then generates code for execution on the host platform. The important point is that the virtual machine first looks at the list of hardware resources available on the platform and compares it to the object with the identified threads and hardware list. That is, the virtual machine is aware of all the hardware resources in the machine (even if the application programmer does not know it). Appropriate addresses and variables are placed on the in-memory stack for each thread of execution or LWP, depending on the hardware resources available. The executable code is located and stored in memory for each thread. In the process of producing what is executable, the operation code for those threads is translated and is not transferred to the native processor engine. The virtual machine then calls the microkernel operating system 104 to schedule thread execution.

【0034】

The identification of threads executed by a virtual machine is not limited to threads specifically written as such by the application programmer. Virtual machines can find specific processors in the hardware resource list, such as those that add a "base boost" to audio files. If the virtual machine finds a called "base boost" object, the virtual machine organizes the call as a thread running on the "base boost" hardware. This is because this hardware feature supports software methods. In general, a "thread" in a new virtual machine can be a thread specially written by the application programmer, or any object or code fragment written by the virtual machine as a thread for execution. Therefore, a virtual machine or kernel can "split" a thread into two if it has the appropriate hardware resources available to handle the smaller thread, which causes it to run. Can increase the concurrency of. Figure 5 shows a rebuilt thread T2 that has been split (see routes 324 and 326) to run on two different processors.

【0035】

For the microkernel operating system 104, RTOS, an operating system of this type on the market known as a real-time microkernel operating system, can be used. ISI Integrated Systems Solutions (Mountain View, CA, USA) has launched such a product called pSOS. MicroWare (Des Moines, Iowa, USA) has also launched such a product called OS-9. Wind River (Oakland, CA, USA) has also launched the VxWorks operating system. The microkernel interfaces with hardware resource 106. That is, the microkernel sorts objects and threads and sends them to the corresponding hardware processor 108. The microkernel handles the scheduling of hardware resources to the extent necessary. For example, if 20 different tasks are required, but there are less than 20 processors, or if the processor mix does not correspond to the type of thread and object being executed, the microkernel will traditionally Schedule tasks using well-known techniques. The microkernel is also involved in network file systems (NFS), networking operations, peripheral drivers, virtual memory management, user interfaces and the handling of other tasks for which other operating systems are still responsible.

【0036】

Two additional memory management tasks are also assigned to the microkernel. The first is "garbage collection," the task of freeing memory allocated to an object or thread whose code no longer has a reference. The details of garbage collection are known as examples in the Java environment. The garbage collection task itself is a process and can be assigned to the appropriate processor. Preferably, the operating system should be written in the Java language and garbage collection tasks should be able to run in native code on Java native processors. Another spatial memory management task is the management of distributed memory across all native processors, such as PicoJava® processors.

【0037】

FIG. 3 shows an embodiment of the multimedia architecture according to the invention, which shows further details of hardware resource 108. Each of these hardware resources, generally the processor core, exchanges data with the microkernel 104, as shown by line 124 as an example. The following types of processors are examples of current preferred embodiments of the present invention. However, this architecture can be applied to a combination of many different hardware resources to optimize performance for a particular multimedia application. One of the advantages of the present invention is that various processors can be arbitrarily added or removed without changing the virtual machine. This provides extremely high flexibility and portability for improved performance. As an example in FIG. 3, the hardware resource 108 includes a video background processor 120, a DSP audio processor 122, a video decompression processor 124, a two-dimensional object rotation processor 126, a stereo audio synchronization processor 128, and a three-dimensional rendering processor 130. It has a graphical user interface tool processor 132, an additional video background processor 134, and one or more native byte code processors 136.

【0038】

Some or all of these processors are ARM-8s manufactured by RISC Machines (Cambridge, UK). Commercially available processor cores such as RISC processors can be used. This type of processor is compact and consumes less power, providing high processing power on the order of 100 MIPS. In addition, one or more processors can be native bytecode cores. These are processors designed to directly execute virtual machine pseudo-language instructions. A code segment or thread that is not compatible with one of the available dedicated processors is transferred by the virtual machine to the native bytecode core for its execution, as described below. In another form, some or all of the above processors can be native bytecode cores. In this case, translation of the operation code is not required for code generation in the virtual machine. The particular choice and combination of hardware resources involves a trade-off between cost and performance in other engineering design choices. Preferably, some or all of the processors are implemented on a single integrated circuit chip to minimize size, cost, and power consumption. FIG. 3 shows how the software objects are rebuilt in the virtual machine 102 and delivered to the corresponding hardware processor in the two embodiments described above. The examples shown here are the video background object indicated by the dashed line 152 and the three-dimensional video object indicated by the dashed line 150.

【0039】

FIG. 4 is a diagram showing the virtual machine program 102 in more detail. This virtual machine is preferably implemented in software and interacts with a hardware buffer, pipeline or the like 200 that receives the stream of input bytecode 100 described above. The first operation of the virtual machine is shown in rebuild step 200. Rebuilding step 200 includes a class loader that determines which class is referenced in the input bytecode and retrieves that class in the class library 204. The class library stored in the virtual machine contains the application programming language library. For example, in the case of the Java language, the basic library includes AWT classes (Abstract Windows Toolkit), network classes, utilities, graphics, sound, video classes, etc. An example of an extension class was given earlier. Preferably, the applicable extension classes are stored in the virtual machine to minimize the increase in digital class linking. Rebuild step 202 identifies any missing class, that is, a class that is referenced in bytecode but is not in the stored library, from the source of input stream 100, as indicated by arrow 206 in the drawing. Download the missing class. By parsing (operand parsing) through bytecode and referencing the class library, rebuild step 202 rebuilds the original software objects, threads, and code fragments used in the source program (10 in Figure 1). Can be done. A list of these objects, threads, and bytecode segments is given to the association process 210 as indicated by arrow 208. This association step 210 limits the hardware resource list 212.

【0040】

The hardware resource list is formed through interaction between the hardware resource 108 and the interfaceing microkernel 104. Process 210 then correlates the objects, threads, and bytecode that need to be executed with the currently available hardware resources. The process combines threads with the most appropriate available resources for each object. An example of this has been described earlier with reference to FIG. Next, in step 216, the variables (fields) required by the virtual machine and other references are decomposed by using the class library as needed. The virtual machine organizes all the parameters and variables needed to perform a given task. There are four essential types of tasks or processes that need to be performed. That is, threads, objects, applets, and flat code segments. The cogeneration process 220 is similar to the traditional code generation process. In this process, the operation code is translated from the virtual machine pseudo-language into an executable machine language. This can vary from process to process if specified for execution on different hardware processors that do not have the same instruction set. Certain processes, especially flat code segments, can run directly on the native pseudocode processor without translation into operation code. For example, code segments of Java bytecode pseudo-languages can be executed directly on the PicoJava native code processor manufactured by Sun Microsystems, California, USA. The cogeneration process 220 appropriately places operands, variables, pointers, etc. on the stack for each process to be executed, and then "passes" the process to the microkernel 104. In this process, each executable process by cogeneration step 220 is associated with the above-mentioned step 2 It is directed to the appropriate processor determined in 10. At the bottom of FIG. 4, for example, the execution process shown on line 230 runs on the corresponding processors 120, 122, and 136. The various types of processors are summarized earlier in relation to Figure 3.

【0041】

Execution threads do not necessarily have to be tied to hardware resources on a one-to-one basis. Some applications can achieve higher performance by providing a "smart" kernel that can rebuild "in progress" threads based on available resources. Preferably, a sufficient number of processors are provided, i.e. up to 20, 30, or 100 processors. In this way, many threads are executed in parallel. Moreover, the whole process is continuous and dynamic. Therefore, the hardware allocation changes constantly at the end of thread execution, and new threads are allocated. An implementation that integrates the entire system into a single silicon chip provides high performance at low cost.

【0042】

In order to further improve the performance, a "smart controller" is installed in the system. Smart controllers are effectively located between the microkernel and hardware resources to control access to the bus. This controller sends addresses to the processor to keep bus traffic low and thus reduce contention and delay.

【0043】

FIG. 5 shows a broad concept of the present invention. In FIG. 5, the source program 300 includes several program threads indicated by T1, T2, and T3. The source program 300 is compiled with the appropriate compiler 302 to form a series of pseudo-language bytecode 304. Compiler 302 includes a class library for the particular object-oriented programming language used in 300. Bytecode 304 can be stored, transferred, or distributed by various means, such as using a machine-readable medium such as a magnetic disk or optical disk, via a network, the Internet, or the like.

【0044】

At the bottom of FIG. 5, the bytecode received on the client machine is processed by the virtual machine 306 as described above. The virtual machine reconstructs the program thread and object 310 and then is transferred to the corresponding special purpose processor, commonly designated by reference numeral 320. The object-orientation of the original program 300 is therefore rebuilt through the hardware processor 320 and carried to allow concurrent execution while retaining the portability, compactness, and other benefits of the object-oriented paradigm. You will get the benefits. By using the appropriate hardware processor outlined here, real-time processing of multimedia data is possible, which is much more effective than the architecture of the present invention and conventional solutions.

【0045】

Although the principle of the present invention has been described with reference to preferred embodiments, it will be clear that the organization and details can be changed in various ways without departing from the scope of the present invention.

【0046】

Further, the software program according to the embodiment of the present invention can be stored, distributed, and used by storing it in an arbitrary storage medium.

【0047】

[Effect of the invention]

From the above, the present invention provides an intermediate code data stream derived from an object-oriented language program, or a processing system and method for a multimedia data stream, which enables more efficient data processing in a multimedia environment.

[Simple explanation of drawings]

[Figure 1]

A simplified data flow diagram showing a traditional object-oriented virtual machine environment used in multimedia applications.

[Figure 2]

It is a figure which showed the novel object-oriented multiprocessor architecture for processing a multimedia application by this invention.

[Fig. 3]

It is the figure which showed the architecture of FIG. 2 in more detail.

[Fig. 4]

It is a mixed diagram of a data flow diagram and a hardware block diagram showing the virtual machines of FIGS. 2 and 3 in more detail.

[Fig. 5]

It is a functional block diagram which showed the handling of the program thread in the new architecture of FIG. 2 and FIG.

[Explanation of symbols]

Source code consisting of 10 Java languages 20 Java language bytecode 30 Virtual Machine API 40 runtime interpreter 42 Basic classes 44 operating system 46 processor 100 input bytecode stream 102 virtual machine 104 operating system or microkernel 106 hardware 108 core (processor) group 110 Thread data flow 120 video background processor 122 DSP audio processor 124 lines 126 2D object rotation processor 128 stereo audio synchronization processor 130 3D rendering processor 132 Graphical User Interface Tool Processor 134 Additional video background processor 136 Native bytecode processor 150 3D video object 152 Video background object 300 Source Program Object 302 compiler 304 bytecode 306 virtual machine 310 Reconstructed program object 320 hardware processors