Synchronized transmission of audio and video data from a computer to a client via an interface

The present invention broadly relates to a device that communicates through a network. In particular, the present invention relates to data flow management between devices that transmit and receive data at different communication speeds. More specifically, the present invention relates to controlling data flow through a buffer by monitoring the buffer and adjusting data transmission based on the state of the buffer.

A "bus" is a set of signals that interconnects at least two electrical devices, allowing one device to transmit information to one or more other devices. There are many different types of buses used in computers and computer-related products. Peripheral Component Interconnect (PCI) bus, Industry Standard Architecture (ISA) bus and Universal Serial, to name a few. There is Bus (USB). Bus operation is usually defined by standards, which specify a variety of things. For example, the electrical characteristics of the bus, how data should be transmitted through the bus, how to notify that a data request has been received, and so on. Performing work using a bus, such as data transmission, data request, etc., is commonly referred to as "cycle" execution. Standardization of bus protocols helps ensure effective communication between such devices, even if the devices connected to the bus are manufactured by different manufacturers. Any company that wants to manufacture and sell equipment that should be used on a particular bus will provide that equipment with an interface specific to the bus to which the equipment is connected. Designing equipment to a particular bus standard means that the equipment will be suitable with such other equipment, even if all other equipment connected to the same bus is manufactured by different manufacturers. Ensure that you can communicate. So, for example, an internal fax / modem designed to run on the PCI bus (ie, built into a personal computer) will be PCI even if each device on the PCI bus is manufactured by a different manufacturer. You can send data to other devices on the bus and receive data from other devices.

Currently, there is a market demand for accepting various forms of consumer electronic devices with a bus interface, which makes it possible to connect such devices to other devices with a corresponding bus interface. For example, digital cameras, digital video recorders, digital video discs (DVDs), and printers are available using the IEEE 1394 bus interface. The Institute of Electrical and Electronics Engineers (IEEE) 1394 bus allows, for example, a digital camera to be connected to a printer or computer, which allows images captured by the camera to be printed on the printer or in the computer. Can be stored electronically. In addition, digital televisions can also be connected to a computer or computer network via the IEEE 1394 bus.

<p> However, there are many devices that do not have an IEEE 1394 interface of any kind. This presents the problem that such devices cannot be connected to other devices as described above. There is an urgent need to overcome this problem and provide connectivity to the device. Otherwise, you will not be able to connect to the IEEE 1394 bus.</p>

<p> The present invention controls the transmission of data from a computer to a video client via an interface device. The interface device buffers the transmitted data frames and propagates them to computers and video clients using different protocols. In one embodiment, the invention performs a first poll on the interface, measures the size of the buffer on the interface, receives a first buffer size value from the interface, and puts video and audio data into the buffer on the interface. Send multiple frames so that there is a delay period between the transmissions of each frame, do a second poll on the interface, measure the buffer size after the frames are sent to the interface, and make a second from the interface. Provides a method of performing data transmission flow control by receiving a buffer size value of. If the second buffer size value is greater than the optimal size and greater than the first buffer size value, the delay period between the transmission of frames from the computer to the interface is increased.</p><p> In another embodiment, the invention performs a first poll on the interface, measures the size of the buffer on the interface, receives the first buffer size value from the interface, and puts video and audio data in the buffer on the interface. Send multiple frames of, so that there is a delay period between the transmissions of each frame, do a second poll on the interface, measure the buffer size after the frames are sent to the interface, and then from the interface Provides a way to perform data transmission flow control by receiving a buffer size value of 2.</p><p> If the second buffer size value is smaller than the optimal size and smaller than the first buffer size value, the delay period between the transmission of frames from the computer to the interface is reduced.</p><p> Many other features and effects of the present invention will be understood by reading the following detailed description with reference to the accompanying drawings.</p>

Looking at FIG. 1, in block diagram format, components are shown that are connected to transmit audio and video data between computer 100 and client 102 and are connected to interface 106 by bus 104. The computer 100 in a preferred embodiment is a computing device capable of processing video and audio data and displaying it in a user-recognizable format. Such devices include desktop, laptop, and palmtop computers. The client 102 referred to herein is a video consumer or producer, including devices such as digital cameras and video storage devices, such as linear and random access devices. The bus 104 referred to herein includes a physical connection between the computer 100 and the interface 106, in addition to the serial protocol that the device communicating on the bus 104 complies with. In a preferred embodiment, the bus 104 is an IEEE known as Firewire. It uses the 1394 serial bus protocol. Interface 106 accepts both analog and digital inputs from client 102 and converts these inputs into scanned lines, which can be used by an audio / video player running on computer 100. In another embodiment, interface 106 accepts a digitally compressed / uncompressed signal from client 102 and transmits the entire or subset of that signal. In one embodiment, interface 106 divides the input into frames 108 on bus 104 to computer 100.

The format of frame 108 is shown in Figure 2. Frame 108 includes a frame header 110, a video block 112, an audio block 114, and optionally an audio header 116. The audio data in the audio block 114 is sampled with respect to the video data in the video block 112. Audio sample count per frame is ANSI / SMPTE Although it varies depending on the number specified in the 272M specification, this specification shall be incorporated herein by reference in its entirety. The audio sample count cadence requires splitting an integer number of samples per second over the NTSC frame rate (29.97fps). Similarly, the size of frame 108 can be varied to accommodate various video formats such as PAL or NTSC, and 8 or 10 bit video data, and audio formats such as 48Khz and 96Khz, 16 and 24 bits, etc. it can. Similarly, the frame size of the compressed data can be varied to accommodate the compressed format. In one embodiment, the video block 112 and the audio block or compressed block are of a predetermined size and create a simple parsing frame 108 with a small processing overhead by the application, eg, a direct memory access program. Only request. If not all of the video block 112 or audio block 114 is completely full of data, the rest of blocks 112, 114 may be filled with zeros. In one embodiment, the data contained in the video block 112 and the audio block 114 is uncompressed, which, of course, requires the processing overhead of a decompression program running on computer 100. The processing overhead on interface 106 is also further reduced.

Interface 106 translates the input received from client 102, translates it into scan lines, organizes it into frames 108, and then sends the frames during each vertical blanking period (interval). And provide synchronization with computer 100. The computer 100 can derive a vertical blanking period from the frequency of the received frame and synchronize itself with the audio and video data of the incoming frame 108 received from the interface 106. Thus, when it is not necessary to perform synchronization on each frame upon reception, processing resources are saved, which provides higher quality performance of audio and video display on the computer 100. To.

Figures 3A and 3B show the formats of the first and second data packets, respectively.

Figures 4A and 4B show the composition of video data within a data packet. Figures 5A and 5B show the composition of audio data within a data packet.

FIG. 6 shows the contents of the frame header 110. Includes are format flags 130 indicating how many bits per sample, SMPTE timecode 132, increasing frame counter 134, audio cycle count 136, audio sample count 138, channel count 140, block size byte count. 142, audio format flag 144, and video format flag 146. The audio sample count 138 indicates the number of samples, which depends on the cadence. The value at the audio cycle count 136 indicates the position within this pace. The pace of the frame forms a cycling pattern. In another embodiment, some of the contents of the frame header 110 can be moved or copied to the optional audio header 116. An alternative diagram of frame header 110 is shown in FIG. 7, showing byte count, data length, and frame bits.

As shown in FIG. 8, the frame 108 is composed of a plurality of packets 150 having a predetermined size. Associated with each packet is a 1394 isochronous packet header. Data transmission according to the present invention utilizes synchronization bits to find the beginning of a frame. The first packet in frame 108 is marked with a synchronization bit. This further reduces the processing overhead by allowing the stream of data to be identified by the computer 100 upon reception, allowing the computer 100 to synchronize with the flow of frames received from interface 106.

In another embodiment of the invention, a frame conforming to the serial digital interface (SDI) standard can be used as shown in FIGS. 9A-9E. In these embodiments, the bus 104 conforms to the IEEE 1394B serial bus protocol and conforms to the data rate limits set forth by the SDI standard. As mentioned above, interface 106 creates scan lines from the received input, performs deinterlacing, packetizing, and creates fixed-size SDTI frames for audio and video data. Form a frame. Various modifications can be made to SDTI frames, depending on the processing resources available on computer 100, interface 106, client 102, or other equipment. As mentioned above, the transmission of SDTI frames transmitted over bus 104 is synchronized with the vertical blanking period of the accepted signal.

As shown in FIG. 9A, the SDTI frame 160 generally has two components, a vertical blanking section 162 and a horizontal retrace 164. Instead, in another embodiment (FIG. 9B), the SDI frame header 166, a header with synchronization bits and frame count, is used for further synchronization and error detection purposes, such as recovery or bus from data lost during transmission. Added to SDTI frame 160 due to reset occurrence. In this embodiment, a frame count synchronization bit is included in the SDTI frame header 166, which synchronizes the SDTI frame header 166 with the vertical blanking section 162. For example, in an application where interface 106 cannot read compressed data, or where an excessive upgrade to interface 106 is required, SDTI frame 160 may be transmitted to computer 100. Here, the processing on the SDTI stream is executed by the software in the non-real-time method. Alternatively, as shown in FIG. 9C, the SDTI frame 160 can also be configured without the horizontal return line 164, further reducing processing overhead. Although there is no horizontal return, an SDTI frame configured with header 166 can also be used in one embodiment, as shown in FIG. 9D. In yet another embodiment, as shown in FIG. 9E, the SDTI frame can be split between the plurality of channels to include the SDTI frame header 166 as well. In this embodiment, the transmitter splits the SDTI stream in half, transmitting half of the lines through channel A and the other half through channel B. The header attached to each subframe can be used to help rejoin the frame data.

In another aspect of the invention, external clocking can be used to synchronize data transmissions between computer 100, interface 106 and client 102. In one embodiment, the client 102 has a high quality reference clock 180 (FIG. 1), which is used to synchronize the clock 182 on the interface 106 and prevent the buffer 184 on the interface 106 from overflowing. Can be. In this embodiment, the value of the reference clock 180 on the client 102 is derived from the frequency at which data is transmitted from the computer 102 to the interface 106 on the interface 106. Cycles are skipped between frame transmissions to perform flow control. The skipped cycle increases the amount of time between frame transmissions and slows down the data rate of frame transmissions. Looking at FIG. 10, at reference number 200, the computer polls interface 106 and reads the size of buffer 184. For illustrative purposes, buffers are referred to by terms such as "greater than" and "less than", but for fixed-size buffers, larger and smaller are related to buffer filling. It should be understood that there is. Next, at reference number 202, computer 100 transmits a plurality of frames to interface 106. At reference number 204, computer 100 polls interface 106 again and measures the size of buffer 184. If the size of buffer 184 is larger than that of the last poll (judgment at reference number 206), control proceeds to reference number 208, where computer 100 is sending frames to interface 106. Increase the delay between. In one embodiment, the delay between transmitted frames is 125 milliseconds. In another embodiment, the fractional delay is by adjusting the delay over a large number of frames. Achieved. For example, if an interframe delay of 2.5 x 1.25 microseconds is required, interspersed with alternating frame delays of 2 and 3 cycles (of 125 microseconds). Control then returns to reference number 202, where frames are sent to interface 106 with additional delay between frames. However, returning to the decision at reference number 206, if the size of buffer 184 was not larger than that size of the last poll, control shifts to the decision at reference number 210. If the judgment at reference number 210 is that the size of buffer 206 has been reduced, control is transferred to reference number 212, where the delay between frames transmitted from computer 100 to interface 106 is reduced. In one embodiment, the amount of this reduction is also 125 Ms. Control then shifts to reference number 202, where frames are transmitted from computer 100 to interface 106 with reduced delay between frames. Returning to the decision at reference number 210, if the size of buffer 184 was not smaller than the size of buffer 184 for the last poll, no adjustment for inter-frame delay is required and control shifts to reference number 202. If an inter-frame delay of 25 microseconds is required, interspersed with alternating frame delays of 2 and 3 cycles (of 125 microseconds). Control then returns to reference number 202, where frames are sent to interface 106 with additional delay between frames. However, returning to the decision at reference number 206, if the size of buffer 184 was not larger than that size of the last poll, control shifts to the decision at reference number 210. If the judgment at reference number 210 is that the size of buffer 206 has been reduced, control is transferred to reference number 212, where the delay between frames transmitted from computer 100 to interface 106 is reduced. In one embodiment, the amount of this reduction is also 125 Ms. Control then shifts to reference number 202, where frames are transmitted from computer 100 to interface 106 with reduced delay between frames. Returning to the decision at reference number 210, if the size of buffer 184 was not smaller than the size of buffer 184 for the last poll, no adjustment for inter-frame delay is required and control shifts to reference number 202. If an inter-frame delay of 25 microseconds is required, interspersed with alternating frame delays of 2 and 3 cycles (of 125 microseconds). Control then returns to reference number 202, where frames are sent to interface 106 with additional delay between frames. However, returning to the decision at reference number 206, if the size of buffer 184 was not larger than that size of the last poll, control shifts to the decision at reference number 210. If the judgment at reference number 210 is that the size of buffer 206 has been reduced, control is transferred to reference number 212, where the delay between frames transmitted from computer 100 to interface 106 is reduced. In one embodiment, the amount of this reduction is also 125 Ms. Control then shifts to reference number 202, where frames are transmitted from computer 100 to interface 106 with reduced delay between frames. Returning to the judgment at reference number 210, if the size of buffer 184 was not smaller than the size of buffer 184 for the last poll, no adjustment for inter-frame delay is required and control shifts to reference number 202.

The interface 106 has a serial unit 300 for enabling communication over the entire bus 104. The serial unit 300 includes a unit directory 302 as shown in Table 1.

<tables num="1"><img file="JP4846589B2_D0001.tif" /></tables>

The Unit_Spec_ID value defines the configuration responsible for the structural definition of the serial unit 300. The Unit_SW_Version value, which works with the Unit_Spec_ID value, defines the unit's software interface. The Unit_Register_location value specifies the offset of the serial unit register in the initial address space of the target device. The Unit_Signals_Supported value specifies which RS-232 signals are supported, as shown in Table 2. If this entry is omitted from serial unit directory 302, none of these signals will be supported.

<tables num="2"><img file="JP4846589B2_D0002.tif" /></tables>

The serial unit 300 also includes a serial unit register map 304, making it easier to refer to the registers contained in the serial unit 300. The configuration of the serial unit register map 304 is shown in Table 3.

<tables num="3"><img file="JP4846589B2_D0003.tif" /></tables>

The serial unit register map 304 makes it easy to refer to the login register. A device attempting to communicate with the serial unit 300 is referred to herein as an initiator. For example, the initiator may be computer 100, or another node connected to the network via a high speed serial bus and communicating with interface 106. The initiator writes the 64-bit address of the base of the serial register map to the login register and logs in to the serial unit 300. If another initiator is already logged in, serial unit 300 returns a conflict error response message. The upper 32 bits of the address are written to the login address and the lower 32 bits are written to login +4. The serial unit register map also makes it easier to see the logout register. The initiator writes an arbitrary value to this register and logs out of the serial unit. After every bus reset, the initiator must write its (possibly modified) node ID to the reconnect register. If the initiator fails to do so within 1 second of the bus reset, it will be automatically logged out. The 16-bit node ID is written to the least significant 16 bits of this register, and zero is written to the most significant 16 bits. Reading the TxFIFO size register returns the size of the serial unit's transmit FIFO in bytes. Reading the RxFIFO size register returns the size of the receive FIFO of the serial unit 300 in bytes. Reading the status register returns the current state of CTS / DSR / RI / CAR (if supported). The status register is configured as shown in Table 4.

<tables num="4"><img file="JP4846589B2_D0004.tif" /></tables>

Writing to the control register sets the DTR and RTS states (if supported). The structure of the control register is shown in Table 5.

<tables num="5"><img file="JP4846589B2_D0005.tif" /></tables>

Writing any value to the Flush TxFIFO register causes the serial unit 300 to flush its transmit FIFO, discarding all bytes currently in it. Writing any value to the flash RxFIFO register causes the serial unit to flush its receive FIFO, discarding all bytes currently in it. Writing an arbitrary value to the break transmit register causes the serial unit 300 to set the break state on its serial port after transmitting the current contents of the TxFIFO. Writing to the set baud rate register sets the baud rate of the serial port of the serial unit 300. The set baud rate register is configured as shown in Table 6.

<tables num="6"><img file="JP4846589B2_D0006.tif" /></tables>

The set Char size register sets the bit size of the characters sent and received. The configuration of the set Char size register is shown in Table 7. 7-bit characters are filled up to 8 bits by adding a pad bit as the most significant bit.

<tables num="7"><img file="JP4846589B2_D0007.tif" /></tables>

The set stop size register indicates the number of stop bits. The set stop size registers are configured as shown in Table 8.

<tables num="8"><img file="JP4846589B2_D0008.tif" /></tables>

The set parity register sets the parity of the serial port. The configuration of the set parity register is shown in Table 9.

<tables num="9"><img file="JP4846589B2_D0009.tif" /></tables>

The set flow control register sets the type of flow control used by the serial port. The configuration of the set flow register is shown in Table 10.

<tables num="10"><img file="JP4846589B2_D0010.tif" /></tables>

The data transmit register is used when the initiator sends a block write request to this register to write a character in the transmit FIFO. The block write must not be larger than the transmit FIFO size specified by the TxFIFO size register. If the Tx FIFO does not have enough room to write all the blocks, a conflict error response message will be returned and the characters will not be copied to the FIFO.

The serial unit 300 also includes an initiator register map having a plurality of registers, which is configured as shown in Table 11.

<tables num="11"><img file="JP4846589B2_D0011.tif" /></tables>

When the serial unit 300 detects a break condition on its serial port, it writes any value to this register. When the serial unit 300 detects a framing error on its serial port, it writes the received character to the framing register. When the serial unit 300 detects a parity error on its serial port, it writes the received character to the parity error register. When the receive FIFO of the serial unit 300 overflows, the serial unit 300 writes an arbitrary value to the RxFIFO overflow register. When the serial unit 300 detects a change in the state of some of the CTS / DSR / RI / CAR, it writes to the status change register and indicates the new serial port signal state. The structure of the status register is shown in Table 12.

<tables num="12"><img file="JP4846589B2_D0012.tif" /></tables>

When the serial unit 300 receives a character from its serial port, it writes the received character to a receive data register that has a block write transaction. It does not write more bytes than the receive FIFO size specified by the Rx FIFO size register. If the initiator is unable to receive all the characters sent, it responds with a conflict error response message and does not receive any of the characters sent.

FIG. 11 shows a register memory map for an interface device according to an embodiment of the present invention. FIG. 12 shows the configuration of the A / V global register included in the interface of the present invention. FIG. 13 shows the configuration of the global status register included in the interface device of the present invention. FIG. 14 shows an isochronous control register included in the interface device of the present invention. FIG. 15 shows the configuration of the flow control register included in the interface device of the present invention. FIG. 16 shows the configuration of the isochronous channel register included in the interface device of the present invention.

In another embodiment of the invention, the synthesized vertical blanking signal is derived by polling the vertical blanking register on interface 106. The vertical blanking signal calls code for a program running on computer 100. In one embodiment, timing information may also be provided to a program running on computer 100 in combination with or in place of the called code. In one embodiment of the invention, interface 106 includes a register that keeps the counter in a state indicating the current progress in the frame, from which the next vertical blanking interval is estimated, or even If not, it can be derived. By deriving boundaries on frame transmission, it is possible to locate and access other data within the frame and in sync with the occurrence of the vertical blanking period, for example for sampling operations. In addition, one embodiment of the invention derives frame boundaries to locate data that is consistent with the vertical blanking period but does not contain information about vertical blanking. In one embodiment, the present invention is used to obtain valid data, eg, timecode contained within a frame, for a period after the occurrence of a video blanking period, which can be read in various processing applications. And can be used. In one embodiment, the computer 100 can schedule an interrupt to start at this estimated time, thereby sending a frame.

<figref num="1">In block diagram format, the main components used in connection with the embodiments of the present invention are shown.</figref><figref num="2">The format of the frame according to the embodiment of the present invention is shown.</figref><figref num="3A">Shows the format of the first data packet.</figref><figref num="3B">The format of the following data packet is shown.</figref><figref num="4A">The structure of the video data in the data packet according to the embodiment of the present invention is shown.</figref><figref num="4B">The structure of the video data in the data packet according to the embodiment of the present invention is shown.</figref><figref num="5A">The configuration of audio data in a data packet according to the embodiment of the present invention is shown.</figref><figref num="5B">The configuration of audio data in a data packet according to the embodiment of the present invention is shown.</figref><figref num="6">The elements of the header included in the frame according to the embodiment of the present invention are shown.</figref><figref num="7">The elements of the header included in the frame according to the embodiment of the present invention are shown.</figref><figref num="8">It shows a set of packets combined to form a frame according to an embodiment of the present invention.</figref><figref num="9A">It shows another embodiment of the present invention in which a modification of the SDTI frame according to the embodiment of the present invention is used.</figref><figref num="9B">It shows another embodiment of the present invention in which a modification of the SDTI frame according to the embodiment of the present invention is used.</figref><figref num="9C">It shows another embodiment of the present invention in which a modification of the SDTI frame according to the embodiment of the present invention is used.</figref><figref num="9D">It shows another embodiment of the present invention in which a modification of the SDTI frame according to the embodiment of the present invention is used.</figref><figref num="9E">Another embodiment is shown in which the transmitter splits the SDTI stream across multiple channels.</figref><figref num="10">In the form of a flow chart, the actions performed to provide external timekeeping between a computer and a hardware interface according to an embodiment of the present invention are shown.</figref><figref num="11">A register memory map for an interface device according to an embodiment of the present invention is shown.</figref><figref num="12">The configuration of the A / V global register included in the interface of the present invention is shown.</figref><figref num="13">The configuration of the global status register included in the interface device of the present invention is shown.</figref><figref num="14">The isochronous control register included in the interface device of the present invention is shown.</figref><figref num="15">The configuration of the flow control register included in the interface device of the present invention is shown.</figref><figref num="16">The configuration of the isochronous channel register included in the interface device of the present invention is shown.</figref>

Code description

100 computers 102 Client 104 bus 106 interface 108 frames