Method of realizing combination of multi-sets of multiple digital images and bus interface technique

FIELD OF THE TECHNOLOGY

0001The invention generally relates to technology of transmitting or recurring any combined images, more specially, to a technique for implementing combination of multi-sets of multiple channels digital images and bus interface thereof. The invention can be used in the multipoint control units (MCU) of a videoconference system or image editing equipment.

BACKGROUND OF THE INVENTION

0002In a videoconference or image editing service, usually it is necessary to combine moving image (sub-image) signals of multiple channels to a single channel signal (one channel or one set of channels) in real time. The combined digital video image signal should include contents of original signals of multiple channels, and is same format and physical interface as a single digital image channel signal. Controlling the moving image of multiple channels, switching the data of the moving image of multiple channels and updating flexibly combination mode of the moving image of multiple channels need to be implemented through software in one hardware device. For example, images of multiple channels can be flexibly divided into sets and every set is combined in different modes.

0003The present conventional image divider (such as the product of US AD company) satisfies the requirement mentioned above, but only has a single fixed combination mode or several limited combination modes.

0004There are mainly two disadvantages of multiple channels images combination equipment at present:

00051. Modes switching is mainly done by hardware, so the combination modes of multiple channels video images cannot be flexible;

00062. Limited by performance and modality of the hardware interface, when a combination mode is more complex, the frame rate of combined original sub-images cannot be kept, so quality of the combined image is not good, and frame rate of combined image is low.

SUMMARY OF THE INVENTION

0007Objective of the invention is to provide a method for implementing image combination of the digital signal of multiple channels and bus interface thereof. With this method and bus interface, digital image signals of multiple channels can be flexibly switched and combination modes can be flexibly updated, multiple channels images (sub-images) can be flexibly grouped into sets and every set can have different combination mode.

0008The invention, a method for combining image data of multi-sets of multiple channels, is implemented by the scheme comprising the following steps:

0009A. integral video data of m channels from an original video channel are inputted to m sub-image channels accordingly, and respectively compressed to m-channel sub-image pixel data, the m-channel sub-image pixel data are stored in m FIFO sub-image frame memories;

0010B. bus strobe signals of m-channel sub-image pixel data control the moment when m-channel sub-image pixel data are outputted to a specified set bus;

0011C. sub-image pixel data on n-set video data buses are respectively combined to form n combined images, and integral video data are outputted from n-set combining image channels.

0012In said step A, generating sub-image pixel data with same scanning sequence as the original video data, but less column pixels and row pixels. The generated sub-image pixel data are stored at the low address of the FIFO sub-image frame memories.

0013In said step B, only one channel of sub-image pixel data is outputted at a specified moment. The sub-image pixel data of m channels are selectively switched to video data buses of n sets by said bus strobe signals.

0014In said step B, computing the corresponding sub-image channel address of every pixel of one field or one frame of combining image as a pixel address; storing the computed pixel addresses of one field or one frame to the memory according to the image scanning sequence to form a pixel address table; reading pixel address from the pixel address table under the control of video sync signal; selecting a set of video data buses and reading the m^thsub-image pixel data, which are read by the n^thset image-processing unit for combining images.

0015In the odd field and even field mode, there are two memories to store pixel address tables for odd field pixel data and even field pixel data, respectively. In the frame mode, there is one memory to store the pixel address tables of common frames.

0016In duration when the image combination mode is stable, the pixel address of every pixel for one field or one frame of combining image is computed only once and repeatedly read by every field or every frame.

0017The implementing scheme of the invention includes a bus interface for combination of multi-sets of multiple channels digital images. The implementing scheme includes an image-processing unit for m-channel sub-images, an image-processing unit for n-set combining images, video data buses of n sets, address and control buses, a pixel data read control and bus switching logic module and a clock and sync signal generating module;

0018m sub-image channels outputs of the image-processing unit for m-channel sub-images are synchronously connected to n-set video data buses, the n-set video data buses are synchronously connected to sub-image data inputs of the image-processing unit for n-set combining images, or fixedly connected to sub-image data inputs of the image-processing unit for less than n sets combining images;

0019said the address and control buses are synchronously connected to the image-processing unit for n-set combining images, the pixel data access control and bus switching logic module and the clock and sync signal generating module.

0020Every channel of said image-processing unit for m-channel sub-images is consisted of sequentially connecting a sub-image generating module, a FIFO sub-image frame memory, and a sub-image pixel data switching and allocating module. Every set of the image-processing unit for n-set combining images is consisted of connecting pixel address control module and combining control logic module. Said pixel address control module and said access control and bus switching logic module are connected with said address and control bus. Said pixel data access control and bus switching logic module is connected with m sub-images pixel data switching and allocating module. Said m sub-images pixel data switching and allocating module m and said n combining control logic module are connected to n sets video data buses.

0021Said pixel data access control and bus switching logic module, which is consisted of decoders, outputs a bus strobe signal to control a sub-image pixel data to be outputted to one set of video data buses. The bus strobe signal is decoded by a pixel address signal, which is outputted from the pixel address control module of a relating combining image, and video sync signal.

0022Said pixel address control module is consisted of connecting a CPU or DSP and a memory. The CPU or DSP computes corresponding sub-image address of every pixel of one field or one frame of combining image. The memory stores the computed pixel address to form a pixel address table for a field or frame.

0023Said address and control bus includes address bus, clock and sync signal bus and control bus. Said address bus is connected with the pixel address control module and said pixel data access control and bus switching logic module. Said clock and sync signal bus is connected with the image-processing unit for m-channel sub-images and image-processing unit for n-set combining images. Said control bus is connected with said m sub-image pixel data switching and allocating modules and the pixel data access control and bus switching logic module.

0024The invention, a method and an interface for combination of multiple sets of multiple channels digital image, has the following advantages: <ul id="ul0001" list-style="none"><li id="ul0001-0001" num="0000"><ul id="ul0002" list-style="none"><li id="ul0002-0001" num="0025">1) The invention takes hardware as a signal-processing platform, and the functions of the platform are implemented by software. Since software is easier to update or change, so the combination mode is flexible. In practice, there are several hundred modes of combination can be switched rapidly under control of software.</li><li id="ul0002-0002" num="0026">2) The invention can flexibly divide sub-image channels into different sets and combination modes, so image-processing ability of the system is greatly raised, which satisfies multimedia image communication and image editing requirement.</li></ul></li></ul>

BRIEF DESCRIPTION OF THE DRAWINGS

0027<figref idref="DRAWINGS">FIG. 1</figref> shows ten typical image combination modes: <figref idref="DRAWINGS">FIGS. 1</figref><i>a</i>, <b>1</b><i>b</i>, <b>1</b><i>c</i>, <b>1</b><i>d</i>, <b>1</b><i>e</i>, <b>1</b><i>f</i>, <b>1</b><i>g</i>, <b>1</b><i>h</i>, <b>1</b><i>i </i>and <b>1</b><i>j. </i>

0028<figref idref="DRAWINGS">FIG. 2</figref> shows a hardware block diagram for the digital image combination of multi-set signals of multiple channels in the invention.

0029<figref idref="DRAWINGS">FIG. 3</figref> shows connection between buses for multi-sets combination of the multiple channels digital image signals and interface modules in the invention.

EMBODIMENTS OF THE INVENTION

0030The invention will be described in more detail, hereinafter, with reference to drawings and embodiments.

0031In principle, number of sub-images that can take part in an image combination is unlimited. Nevertheless, limited by drive capacity of a bus interface chip and signal quality, the sub-images taking part in a combination process are no more than 16, in general. With special processing for signals drive and buses match, number of sub-image channel can be increased to 25 or more.

0032In theory, with software flexibility the combination modes of an image are almost unlimited. <figref idref="DRAWINGS">FIG. 1</figref> shows ten typical combination modes. <figref idref="DRAWINGS">FIG. 1</figref><i>a </i>is a combination mode consisted of two sub-images. <figref idref="DRAWINGS">FIG. 1</figref><i>b </i>is a combination mode consisted of three sub-images. <figref idref="DRAWINGS">FIG. 1</figref><i>c </i>is a combination mode consisted of four sub-images. <figref idref="DRAWINGS">FIG. 1</figref><i>d </i>is a combination mode consisted of nine sub-images. <figref idref="DRAWINGS">FIG. 1</figref><i>e </i>is a combination mode consisted of six sub-images. <figref idref="DRAWINGS">FIG. 1</figref><i>f </i>is another combination mode consisted of four sub-images. <figref idref="DRAWINGS">FIG. 1</figref><i>g </i>is a combination mode consisted of 16 sub-images. <figref idref="DRAWINGS">FIG. 1</figref><i>h </i>is a combination mode consisted of eight sub-images. <figref idref="DRAWINGS">FIG. 1</figref><i>i </i>is a combination mode consisted of 13 sub-images. <figref idref="DRAWINGS">FIG. 1</figref><i>j </i>is another combination mode consisted of 13 sub-images.

0033The embodiment takes combination mode consisted of 16 sub-images as an example. The 16 (m=16) original images (16 sub-images) are divided into n sets (in general, n=1 to 4). Every set of sub-images applies one set of video data bus; so n-set sub-images respectively apply n-set video data buses to form combination image. During combination, there is one limitation that is every original image (it is output data of one sub-image physical channel) cannot take part in two or more than two image combination processing at a moment. Conforming to this limitation, original images from every channel can be grouped into many kinds of set or combined many kinds of modes. For example, original images of the first, second and third channel form the first set, original images of the forth, fifth, sixth and seventh channel form the second set, and original images of the eighth to sixteenth channel form the third set and so on.

0034Refer to <figref idref="DRAWINGS">FIG. 2</figref>, it shows the hardware structure for m (m=16) sub-images to be combined to n (n=4) sets. The hardware structure includes image-processing unit for multiple channels sub-image <b>10</b>, video data bus of multiple images (the video data buses of n sets) <b>20</b>, address and control bus <b>30</b>, image-processing unit for multiple sets combining images <b>40</b>, pixel data access control and bus switching logic module <b>50</b> and clock and synchronous signal module <b>60</b>. The integral video data, coming from m-channel original video, are inputted to m sub-image channel and combined to n sets. The integral video data is outputted from n sets of combining image channel.

0035In the image-processing unit for multiple channels sub-image <b>10</b>, each sub-image channel includes: a sub-image generating module <b>11</b>, a FIFO (first-in-first-out) frame buffer memory module <b>12</b> and a switching and allocating module for sub-image pixel data <b>13</b>. The modules <b>11</b>, <b>12</b> and <b>13</b> are connected sequentially.

0036The sub-image generating module <b>11</b> compresses one channel of the integral digital image data corresponding to original video channel in pixel to generate the sub-image digital data. The integral digital image data in original can take two formats: one is the standard digital video format including CCIR-601 or CCIR-656 formats, another is common intermediate format which is same as scanning sequence of display and isn't distinguished odd field and even field, such as the CIF format. Using DSP (digital signal processor), original image data with these two formats can be compressed in pixel to generate a sub-image data, which has the original scanning sequence but less row pixels and column pixels than original image. For the first format, there is a special chip for image compression.

0037The FIFO frame buffer memory module <b>12</b> is used for storing temporarily compressed original image data. When each frame or field of original image has been compressed, it is difficult to synchronize the sub-images with different size, so the compressed data are stored in the FIFO frame buffer memory and each frame or field of original image can refresh the FIFO frame buffer memory. The compressed sub-image data are stored in low address of the FIFO frame buffer memory.

0038There are two functions of the switching and allocating module for sub-image pixel data <b>13</b>. One is to guarantee that at any moment there is only one sub-image pixel data is outputted to the same set of video data buses of multiple images <b>20</b>, which is a tri-state bus. Another is to switch selectively every channel of sub-image data to different set data bus (such as one of the four set buses), when there are multiple sets (such as four sets). In a period of time, which can be several hours or several minutes, the grouping of m sub-images is definite. Therefore, allocation relation of the video data buses of multiple images <b>20</b> is also definite and is controlled by the system CPU. The address and control bus from image-processing unit for multiple sets combining images <b>40</b> and CPU of the system implemented jointly the two functions.

0039The pixel data access control and bus switching logic module <b>50</b> is consisted of several decoders. Input signals of module <b>50</b> come from the address and control bus <b>30</b> including pixel address signal sent by pixel address control module in image-processing unit for multiple sets combining images <b>40</b> and video synchronous signal. Output signals of module <b>50</b> are the strobe signal of sub-image pixel data of every channel, which are used to control the sub-image pixel data to be outputted to a set of the video data bus at a specified moment. The module <b>50</b> can be put on the side of the image-processing unit for multiple channels sub-image <b>10</b>, such as <figref idref="DRAWINGS">FIG. 2</figref>, or on the side of the image-processing unit for multiple sets combining images <b>40</b>, such as <figref idref="DRAWINGS">FIG. 3</figref>.

0040Refer to <figref idref="DRAWINGS">FIGS. 2 and 3</figref>, the video data buses of multiple images <b>20</b> are consisted of four-set video data buses: <b>201</b>, <b>202</b>, <b>203</b> and <b>204</b>. <figref idref="DRAWINGS">FIG. 3</figref> shows typical connections between the four video data buses and interface modules.

0041The video data buses of multiple images <b>20</b> can be divided into one set or several sets. In general, it is four sets or less than four sets, and it is four sets in this embodiment. Every set of the video data buses, <b>201</b>, <b>202</b>, <b>203</b> or <b>204</b>, is connected to the outputs of all sub-image channels in the units <b>10</b>, i.e., the outputs of all channels of switching and allocating module for sub-image pixel data <b>13</b>. At the same time, every set of the video data buses, <b>201</b>, <b>202</b>, <b>203</b> or <b>204</b>, is also connected to the inputs of all images combination units i.e. the combining control logic module for combining images <b>41</b> in the unit <b>40</b>. Of course, they can be respectively connected to one image combination unit as shown in <figref idref="DRAWINGS">FIG. 3</figref>. Every set of video data buses <b>201</b>, <b>202</b>, <b>203</b> or <b>204</b> comprises 16 or 8 video data lines to transfer color difference, brightness signals etc. of the YUV data.

0042It is necessary to explain that the video data buses of multiple images <b>20</b> can be one set buses or multiple sets buses. It is unnecessary that the inputs of image-processing unit for multiple sets combining images <b>40</b> should be circularly connected with all multiple sets of the video data buses. The input of image-processing unit for multiple sets combining images <b>40</b> can be fixedly connected with only one set of video data buses. Module <b>50</b> can be put on the side of the image-processing unit for multiple channels sub-image <b>10</b> (such as <figref idref="DRAWINGS">FIG. 2</figref>), or on the side of the image-processing unit for multiple sets combining images <b>40</b> (such as <figref idref="DRAWINGS">FIG. 3</figref>).

0043In order to have a good quality signals in the buses, it is necessary to have impedance match for each video data signal. In general, the ABT type (one kind of bus drive chips) interface chip is used for the bus drive.

0044The address and control bus <b>30</b> in <figref idref="DRAWINGS">FIG. 2</figref> is control bus <b>301</b>, clock and synchronous bus <b>302</b> in <figref idref="DRAWINGS">FIG. 3</figref>. The address and control bus <b>30</b> includes video synchronous signals, video pixel address signal/sub-image strobe signals. The video synchronous signals include a 27 MHz main clock, a 13.5 MHz or 6.75 MHz pixel clock, a field-synchronizing signal, a line-synchronizing signal and mixed blanking signal etc. The video synchronous signals are provided to the image-processing unit for multiple channels sub-image <b>10</b> and image-processing unit for multiple sets combining images <b>40</b>. The signal type of video pixel address signal/sub-image strobe signal depends on where the pixel data access control and bus switching logic module <b>50</b> is put on. The pixel address control module for combining images <b>42</b> in every set of the image-processing unit for multiple sets combining images <b>40</b> sends out a image channel number (address) corresponding to the pixel data to read at every pixel clock after the pixel data access control and bus switching logic module <b>50</b> decodes the image channel number (address), the module <b>50</b> generates a strobe signal that selects the m^thchannel sub-image data to be read by the n^thcombining image process unit. The video data is read from the FIFO <b>12</b> of the m^thsub-image channel.

0045In the invention, the multiple images video buses can be a general name for video data buses of multiple channels <b>20</b>, control bus <b>301</b> or clock and synchronous signal bus <b>302</b>. In the <figref idref="DRAWINGS">FIG. 3</figref> embodiment, the pixel data access control and bus switching logic module <b>50</b> is put on the position where it is near the image-processing unit for multiple sets combining images <b>40</b>.

0046The pixel address control module of combining image <b>42</b> is consisted of CPU or DSP and memory. According to the background requirement, the CPU or DSP computes a corresponding sub-image channel number (address) of every pixel data of a field or frame of combining image as a pixel address. Then, a field or frame of the pixel addresses is stored in the memory according to the image scanning sequence. In the odd field and even field mode, two memories are needed to respectively store pixel addresses table of odd field data and pixel addresses table of even field data. In the frame mode, only one pixel addresses table of a common frame is needed.

0047The memory can be a FIFO frame buffer memory or a SRAM (static RAM) frame buffer memory. In duration, such as several minutes or several hours, combination mode of multiple images is relatively stable and don't need be switched continually. Therefore, for a definite mode, pixels address needs to be computed only once and is stored in the memory. Then, under control of video sync signal, every field or frame of the pixels address is read from the memory repeatedly. This belongs to clock operation of hardware. Pixel address table is obtained by large of computation. If every field or frame is computed in real time, high speed DSP is applied and the cost is high. So it is unnecessary to compute pixel address table of every field or frame.

0048The combining control logic module for combining images <b>41</b> and the pixel address control module of combining image <b>42</b> constitute the image-combining sub-module. When every sub-image video data outputs to video data buses of multiple channels <b>20</b> in a specified sequence, a data format and sequence of a combined image has been formed. Nevertheless, in order to enhance function of system, every image combining sub-module must implement the following two functions: selecting one set video data buses from the multiple set video data buses (or fixed connection), substituting pixels and adding captions and boarders. The function of substituting pixels is to pad a fixed color to the blank sub-image area in the combined image. Padding color in an area can be implemented directly in the video data bus with push up or pull down mode, or implemented by other sub-module accompanying with the captions. Since they do not belong to the invention, so they are not described in more detail.

0049Special video devices or logic circuits can make the clock and synchronous signal module <b>60</b>.

0050The invention, implementing image combination and bus interface thereof, has been tested in a MCU system of videoconference, and it is proved that object of the invention has been reached. With its powerful processing ability for multiple image combination and better performance, the invention can be used in the multimedia image communication.