Video communication terminal

Description: TECHNICAL FIELD [Detailed description of the invention]

【0001】

[Technical field to which the invention belongs]

The present invention relates to video data transmission control of a multimedia communication terminal capable of communicating video data with each other.

【0002】

[Conventional technology]

Nowadays, in addition to audio information, a multimedia communication service that multiplexes video data and audio data by equipping communication terminals with a camera that shoots images and a display device such as an LCD is available to communicate so-called multimedia information with each other. It has been realized.

【0003】

In the multimedia communication service described above, the video data provided is handled by encoding the video input from the camera housed in the communication terminal with a video coding device in real time and transmitting the video data to the other party. It is composited with a video coding device in real time and projected onto a display device, and the size and number of frames of the video data transmitted at that time are defined based on preset information.

【0004】

[Problems to be Solved by the Invention]

However, in the conventional multimedia communication service, constant video data is always encoded according to preset information, and the correspondence to the size of the video data to be displayed and the number of displayed images is not taken into consideration. , There is a problem that it cannot cope with a change in the size of the video data to be displayed and a change in the number of displayed images.

【0005】

In addition, since the allocation rate to video data and the number of video frames in the transmission rate on the network are defined only by the ability of the communication terminal, there is a problem that the user needs for video data are not finely tuned. ..

【0006】

Therefore, in consideration of these problems, the present invention considers the display form of the video data during communication, that is, the change in the display mode on the receiving side with respect to the display mode, and the transmitting side recognizes it, and if necessary. The purpose is to provide a video communication service that can respond to user requests by finely controlling the coding when motion is required by changing the video coding process or when image quality is required. ..

【0007】

[Means for solving problems]

In order to achieve the above object, in the present invention, in a video communication terminal that performs multimedia communication that transmits and receives multimedia information including at least information about video, an input means for inputting video information and a video input from the input means. Multimedia communication is performed between a display means for displaying information, an instruction means for instructing a processing form of video information displayed by the display means, and video processing information based on the image processing form of the video information instructed by the instruction means. It is characterized in that it is provided with a transmission means for transmitting to a communication partner terminal.

【0008】

Further, in a video communication terminal that performs multimedia communication that transmits and receives multimedia information including at least information about video, an input means for inputting video information, a display means for displaying video information input from the input means, and a video. Based on the receiving means for receiving the video processing information regarding the video processing form of the information and the video processing information received by the receiving means, the input means inputs and controls the video processing of the video information displayed on the display means. It is characterized by having a control means.

【0009】

Further, in a video communication terminal that performs multimedia communication that transmits and receives multimedia information including at least information about video, a video receiving means that receives video information from the communication terminal that performs multimedia communication and a video received by the video receiving means. The video is based on a display means for displaying information, a video processing information receiving means for receiving video processing information related to a video processing form of video information from the other terminal, and video processing information received by the video processing information receiving means. It is characterized by comprising a control means for controlling the processing of video information received by the receiving means and displayed on the display means.

【0010】

Further, in a video communication terminal that performs multimedia communication that transmits and receives at least multimedia information including information related to video, a video processing information receiving means that receives video processing information related to a video processing form of video information from a partner terminal that performs multimedia communication. It is characterized by including a control means for controlling the processing of the video information transmitted to the other party terminal based on the video processing information received by the video processing information receiving means.

【0011】

According to the above configuration, the present invention makes it possible to manage the information related to the current video display on the own terminal, and notifies the communication partner of the video display mode information when the video display mode of the displayed video is changed. Is possible.

【0012】

In addition, it serves as an index for performing optimal processing for performing video coding for a video communication partner, and the state of the video display capability of the receiving side can be known in advance on the transmitting side, so that a wireless line in particular is used. In a network where the transmission rate changes significantly depending on the wireless environment, the request for video display on the receiving side can be clearly understood, so that unnecessary processing can be omitted on the transmitting side, and the user's operation can be performed. Since it is possible to take detailed measures, the convenience of the user is greatly improved.

【0013】

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, examples of the present invention will be described with reference to the drawings. FIG. 1 shows the relationship between the network to which the terminal capable of multimedia communication is applied and the multimedia terminal MT (hereinafter referred to as MT) capable of multimedia communication in one embodiment of the present invention.

【0014】

Examples of the MT include an analog TV telephone terminal 1, an ISDN TV telephone terminal 2, a mobile wireless terminal 3, a PHS terminal 4, and the like, as shown in FIG. In addition, PC5 that supports multimedia communication connected to PHS terminal 4 which is MT, and PC6 that supports multimedia communication connected to multimedia server 7 via LAN can perform multimedia communication. is there.

【0015】

The analog TV telephone terminal 1 is connected to the public network 18 via the general public line 10. The ISDN TV telephone terminal 2 is connected to the public network 18 via the INS64 line 11.

【0016】

The mobile phone terminal 3 is connected to the mobile wireless network 13 via the public base station CS12, and the mobile wireless network 13 is connected to the public network 18. The PHS terminal 4 and the PC5 connected to the mobile wireless terminal are connected to the PHS network 15 via the public base station CS14, and this PHS network is connected to the public network 18.

【0017】

The MTs mentioned above can communicate with each other. Further, each MT can also communicate with the multimedia server 8 connected to the digital leased line 17 via the public network 18.

【0018】

In addition, the multimedia compatible PC 6 and access server 7 that support multimedia communication connected to the LAN are connected to the public via a digital dedicated line (INS1500) 16 and the multimedia server 8 is connected to the public via a digital dedicated line (INS1500) 17, respectively. It is connected to the network 18 and can communicate with each other.

【0019】

In addition, each MT is equipped with a display device such as an LCD and, in some cases, a camera that captures images. The camera may be removable. FIG. 2 shows the configuration of the PHS terminal 4 with the PHS terminal 4 shown in FIG. 1 in the present embodiment as an MT.

【0020】

The public wave radio section (hereinafter referred to as the RF section) 22 transmits / receives data to / from the PHS base station 14 via the public antenna 21. The wireless communication control unit 24 performs protocol conversion of data from the RF unit 22, management of control channels, control of multimedia communication protocol, and the like, and relays a call control message to the terminal control unit 26.

【0021】

Also, during communication, multimedia information including video data and audio data is relayed to the multiplex separation processing unit 25. The terminal control unit 26 processes call control messages such as call connection / release, input processing from the key control unit 27, and after the call connection, requests the wireless communication control unit 24 to negotiate multimedia terminal control and multiple separation units. 25, video processing unit 29, audio processing unit 30, data processing unit 31, and display control unit 28 are controlled.

【0022】

The display control unit 28 executes video output control to the display device 33 composed of an LCD, video input control from the camera 34, etc. at the request of the terminal control unit 26, and inputs / outputs video to the video processing unit 29. ..

【0023】

The video processing unit 29 relays the video data required for the multimedia information multiplex separation processing performed by the multiplex separation processing unit 25 to the display control unit 28. Similar to the video processing unit 29, the data processing unit 30 relays text data between the multiplex separation processing unit 25 and the display control unit 28.

【0024】

The audio processing unit 31 relays audio data between the multiplex separation processing unit 25 and the PCM Codec 32. The PCM Codec 32 digitally encodes / combines the audio input from the microphone 35 and the audio output to the speaker 36 between the audio processing units 31.

【0025】

The key control unit 27 relays the interrupt signal from the key 37 stored in the terminal to the terminal control unit as input data. The procedure for starting video communication on the PHS terminal 4 in the configuration described above is shown below with reference to FIG.

【0026】

First, the terminal control unit 26 that has received the dial input from the key control unit 27 and the "connection key" requests the communication control unit 24 to perform call connection processing based on the input destination number, and the communication control unit 24 communicates. Call processing is performed from the RF unit 22 to the PHS base station 14 via the interface 23.

【0027】

If there is a response from the other party MT or the multimedia server 8, the terminal control unit 26 receives a call connection status change notification from the communication control unit 24, and realizes multimedia communication such as video and audio. The procedure of is instructed to the communication control unit 24.

【0028】

Here, the procedure for realizing multimedia communication is that the terminal equipped with the multimedia communication function exchanges specifications and multiplexing methods used for encoding each multimedia information of audio, video, and data to be transmitted. It is an indispensable negotiation between terminals for actual multimedia communication, such as ITU-T H.324 protocol for analog TV phones, H.320 protocol for ISDN lines, and LAN accommodation. If it is a terminal, H.323 is common, but negotiation between terminals is not limited.

【0029】

Next, after making a call connection with the other party MT and following the procedure for realizing the above multimedia communication, when starting the multimedia communication, the terminal control unit 26 performs each actual data (video data and audio). Notification of each processing method based on the result of terminal-to-terminal negotiation to the video processing unit 29, the data processing unit 31, the audio processing unit 31, and the multimedia separation unit 25 that performs multiplexing / separation processing in order to code / combine the data). And control operations such as start and end.

【0030】

FIG. 3 shows an outline of the image display mode and its mode change in the MT of the present invention. First, in the case of video display mode 1, as a video display at the time of communication connection, in the full-screen display mode of a single video that displays either the own video or the other party's video on the display device 33 on the entire display device. is there.

【0031】

In the case of video display mode 2, as a video display at the time of communication connection, two screens simultaneously or singlely display both the own video and the other party's video or only the other party's video on the display device 33 on a fixed size area of the display device. This is the video reduction screen display mode.

【0032】

In the case of video display mode 3, as a video display at the time of communication connection, one of both the own video and the other party's video is displayed on the display device 33 to fill the display device, and the other is displayed on a fixed size area. This is a simultaneous inset video display mode for two screens that are inset and displayed.

【0033】

In each mode, the image display during a call can be changed by operating the key 37 of the terminal, the image display of the own image and the other party's image can be changed, and the own image display can be turned ON / OFF. It has become like.

【0034】

That is, when the process of changing the display mode A is instructed while the video display mode 1 of the MT is set, the MT changes the video display mode to the video display mode 2.

【0035】

Further, in the state where the video display mode 1 is set, when the process of changing the display mode 1 is instructed, the display of the own video and the display of the other party's video can be switched.

【0036】

When the MT video display mode 2 is set and the process of display mode change B is instructed, the MT changes the video display mode to the video display mode 3. Further, in the state where the video display mode 2 is set, when the process of changing the display mode 2 is instructed, the display of the own video can be switched ON / OFF.

【0037】

When the process of changing the display mode 3 is instructed while the MT image display mode 3 is set, the display size of the own image and the other party's image can be switched. In the display mode of the own image and the other party's image in the present embodiment, as shown in the image display mode 2, it is also possible to display the other party's telephone number and the time during communication under the control of the terminal control unit 26. ..

【0038】

Further, the switching of the video display mode is not limited to the present embodiment, and for example, the switching from the video display mode 3 to the video display mode 1 or the switching of the video display mode is not limited.

【0039】

The MT described in the present embodiment displays a video by referring to a mode setting change table that stores information about the video display mode when displaying a video in each video display mode or changing the video display mode. .. Figure 4 shows an example of the mode setting change table referenced by MT.

【0040】

Figure 4 (a) is a table showing the setting status of each display mode related to video display on the MT terminal. Number of images displayed in each image display mode in the table, display size (ratio to full screen display), display position on display device 33 (the upper left corner is (0,0) in X and Y coordinates), It consists of the number of display frames (the number of images encoded per second).

【0041】

The video ID 1 indicates the display size of the other party's video, the display position on the display device 33, and the number of display frames in each video display mode. The video ID 2 indicates the display size of the own video, the display position on the display device 33, and the number of display frames in each video display mode.

【0042】

In the video display mode 1, since the own video is not displayed (OFF), the value does not exist. In the present embodiment, these values have default values set in advance by MT and are determined by the video display mode at the start of video communication, but the present invention is not limited to this, and other determination methods may be used. Here, each value shown in FIG. 4A indicates a value in the currently displayed video display mode or a value set when the video display mode was last selected.

【0043】

Figure 4 (b) shows the preset default values in each of the above-mentioned video display modes and the video display mode changed by the user's key input, or when requested, the other party should change it, that is, the video frame to notify the other party. It is a table in which the change rate of the number is set.

【0044】

When the video display mode is changed as shown in Fig. 3, the value of the display video frame setting table for each mode is notified to the control unit of the own terminal (MT) and to the other MT terminal that is the communication partner. The value of the number of frames in Fig. 4 (a) will be changed.

【0045】

The mode setting status table (a) and the mode-specific display video frame setting table (b) shown in FIG. 4 are stored in the storage unit (not shown) of the MT, and the storage form thereof does not matter.

【0046】

Next, when the user changes the video display mode during communication in the own MT terminal of the present embodiment based on the mode setting status table shown in FIG. 4, and notifies the other MT terminal and the multimedia server 8. It operates as shown in the processing flow shown in Fig. 5. In this processing flow, the case of communication between MT terminals will be described as an example.

【0047】

It is assumed that the own MT terminal is in a call (step 50), and the video display mode 3 is selected (step 51). In the image display mode 3, as shown in FIG. 3, either one of the self-image taken by the own terminal on the display device 33 and the other party's image transmitted by the other party's MT terminal is displayed on the entire display device 33. It is a simultaneous inset screen display mode of two screens that displays in and displays the other in an area of a certain size.

【0048】

When the image display mode 3 is set, according to Fig. 4 (b), the number of frames between the other party's image and the own image is "10 frames" for the other party's image and "4 frames" for the own image by default. The MT terminal encodes and transmits video data at a rate of 10 frames / second (step 52).

【0049】

In this case, since the own MT terminal displays the own video in addition to the other party's video transmitted by the other MT terminal (step 53), it seems to be a little overloaded as the real-time coding and compositing process of the video. It is possible that delays and roughness will be noticeable in the video display.

【0050】

Therefore, the terminal control unit 26 refers to the display video frame setting table for each mode shown in FIG. 4 (b), and when the own video is displayed, it means that the number of frames of the other video is reduced by 2. "-2" Since it is ", change the number of video frames of the other party to 10-2 = 8 frames (step 54).

【0051】

That is, since the number of frames is reduced, the processing load of the video display is reduced, and the video communication environment is improved for both transmission and reception. In addition, when the user simultaneously requests to increase the display size of the own image (step 55), according to the mode-specific display image frame setting table shown in FIG. 4 (b), the change in the number of frames when the image size is changed. The values are "-2" to "+2".

【0052】

At this time, when the user expands the display size of the own image by one step, the number of frames is changed to "-1" and 4-1 = 3 frames, and the current mode setting display status table is shown as shown in Fig. 4 (a). Three frames are stored as the value of the number of frames in the video display mode 3 shown in (step 56).

【0053】

The terminal control unit 26 issues a request to the remote terminal MT to change the video coding frame rate via the communication control unit 24 for the updated mode setting display state video table information (step 57).

【0054】

Next, it is assumed that the user again changes from the video display mode 3 to the video display mode 2 by operating a key during the video display (step 58). In the changed video display mode 2, as can be seen from Fig. 4 (b), the number of frames for the other party's video and the own video is "10 frames" for the other party's video and "4 frames" for the own video, respectively, by default. "It has become.

【0055】

For example, suppose that the size of the other video is changed by increasing the number of frames by two steps and the size of the own video is changed by increasing the number of frames by two steps. In this case, the number of frames of the other party's video is changed from the default value of 10 frames to the change in the number of frames when displaying the own video-2 frames, and the size of the other party's video is increased by 2 steps. 10-2 + 2 = 10 frames . In addition, the number of frames of the own image is "4 + 2 = 6 frames" because the size of the own image is changed by increasing the number of frames by 2 steps from the default value of 4 frames.

【0056】

When the above change process is performed and the number of frames of the own video partner video is changed, the changed value is stored in the mode setting status table as shown in FIG. 4 (a).

【0057】

Further, when the change process is performed, the terminal control unit 26 analyzes the input request from the user (step 59), and if the request is related to the video display mode change (step 60), the request event or message. Is transmitted to the terminal control unit 26 (step 61).

【0058】

Next, the processing of the partner MT terminal on the receiving terminal side that has received the request for changing the video display mode will be described with reference to FIG. When it is determined that the terminal is in a call (step 70), the video display mode setting information according to the size of the display device 33 is acquired (step 71), and the display video frame setting table for each mode shown in FIG. 4 (b) is acquired. According to this, the content is notified to the video processing unit 29, and video coding is started (step 72).

【0059】

On the other hand, when the terminal control unit 26 of the transmitting terminal receives the request message transmitted by the MT terminal (step 79), the terminal control unit 26 analyzes the content of the request message (step 80) and determines whether or not the request message is a video display mode change. Determine if (step 81).

【0060】

When it is determined that the video display mode has been changed, the terminal control unit 26 generates a processing request signal (event) to the terminal control unit 26 in order to perform the video display mode change processing.

【0061】

If it is determined that the code rate of the current transmitted video frame needs to be changed for the request message received from the other party, it is determined whether or not there is a change request in the video display setting (step 73), and the request is made. It is analyzed whether or not the video coding rate and the number of frames are changed according to the above (step 74).

【0062】

If it is determined that it is necessary to change the video coding rate or the number of frames (step 75), reset the contents in the video processing section of the own terminal to switch to the video display mode according to the request, and then set the video code. Resume conversion (step 76).

【0063】

After that, it is determined whether or not the own terminal is in a call (step 78), and it is determined whether to continue or end the process. Next, in the present embodiment, when the video display mode is changed in the own terminal MT and the mode setting status table is updated, which of the data in the updated mode setting status table is updated in the other MT terminal with which communication is being performed. It will be described with reference to FIG. 7 whether or not the terminals are exchanged with each other. The configuration of the terminal MT shown in FIG. 7 is the same as the configuration shown in FIG. 2, and the description thereof will be omitted.

【0064】

The own terminal MT that has changed the video display mode multiplexes the change instruction data such as the frame rate, that is, the data stored in the mode setting status table shown in FIG. 4A, together with the video data and the audio data in the multiplex separator. User data A area Data used in ITU-T H.324 protocol for analog TV phones, H.320 protocol for ISDN lines, and H.323 protocol for LAN-accommodated terminals Set to and send to the other MT terminal.

【0065】

In the partner MT terminal that receives the change instruction, the data for issuing the change instruction such as the frame rate is analyzed from the area Data of the user data A separated by the multiple separation processing unit, and the terminal control unit of the terminal ( Figure 2) will change the settings.

【0066】

The negotiation between terminals is not limited to the above-mentioned protocol. As described above, in the embodiment of the present patent, the notification method for the number of frames of the cell image displayed on the display device has been described, but the information to be notified to the partner MT terminal is not only the frame rate but also the change in the display area. No particular mention is made as long as it is information or instructional information for adjusting the transmitted video frame rate such as the number of videos to be displayed.

【0067】

[Effect of the invention]

As described above, according to the video communication terminal of the present invention, it is possible to cooperate with each other for the video coding process between the communication terminals according to the video display mode, so that the video is received on the receiving side. Even when the display mode is switched and displayed, unnecessary coding processing can be prevented on the transmitting terminal, the limited network can be used effectively, and the user's needs for video display can be flexibly met. The effect is obtained.

【0068】

Further, in the present invention, since the state of the image display capability of the receiving side terminal can be known in advance on the transmitting side terminal, a requirement for the image display on the receiving side is required especially in a network such as a wireless line whose transmission rate changes remarkably depending on the wireless environment. Is clearly understood, so unnecessary processing can be omitted on the sending side, and the user's operation can be dealt with in detail, which greatly improves user convenience. Is obtained.

[Simple explanation of drawings]

[Figure 1]

FIG. 1 is a system configuration diagram showing the relationship between the multimedia terminal and the network in this embodiment.

[Figure 2]

FIG. 2 is a diagram showing each processing unit of the MT terminal in this embodiment.

[Fig. 3]

FIG. 3 shows an outline of the video display mode and mode change in the MT terminal in this embodiment.

[Fig. 4]

FIG. 4 shows an example of each video display mode in this embodiment and a mode setting change table referred to when the mode is changed.

[Fig. 5]

FIG. 5 is a diagram showing a processing flow when the video display mode is changed in the MT terminal in this embodiment.

[Fig. 6]

FIG. 6 is a diagram showing a processing flow of the partner MT terminal on the receiving terminal side that has received the request for changing the video display mode.

[Fig. 7]

FIG. 7 is a diagram illustrating a protocol exchange of data for instructing a change of a transmission coding rate / frame rate between multimedia terminals.

[Explanation of symbols]

1 ... analog TV phone terminal, 2 ... ISDN TV phone terminal, 3 ... mobile phone terminal, 4 ... PHS terminal, 5 ... PC connected to mobile wireless terminal, 6 ... on LAN Multimedia PC, 7 ... Access server, 8 ... Multimedia server, 10 ... General public line, 11 ... INS64 line, 12 ... Public base station CS, 13 ... Mobile wireless Line network, 14 ... Public base station CS, 15 ... PHS line network, 16, 17 ... Digital dedicated line (INS1500), 18 ... Public network