Data communication system, electronic conference system, data communication method, data communication program and recording medium

The present invention is a server / client type data communication in which screen data displayed by one information device can be shared by another information device or can be remotely controlled from another information device. Regarding the method. The present invention also relates to an electronic conferencing system using the communication method. It also relates to data communication methods, data communication programs and storage media.

In recent years, electronic conferencing systems that hold conferences with distant users have begun to spread due to the speeding up of processing of computer systems and the speeding up of networks. The following software is available for use in electronic conferencing systems.

1.Microsoft NetMeeting An electronic conferencing application that is installed as standard on some operating systems of the Windows (registered trademark) series sold by Microsoft Corporation in the United States, and allows the entire desktop screen of the server or the screen of the specified application to be displayed from the client. It can be viewed or operated remotely.

2.VNC (Virtual Network Computing) Screen sharing software developed by AT & T Corporation in the United States, which basically shares the entire desktop screen of the server. Using a protocol called RFB (Remote Frame Buffer) protocol, the updated part of the screen area of the server is encoded and sent to the client. Some software developed based on this VNC has been improved so that only the screen area of the specified application can be shared, or only the specified rectangular area of the desktop screen of the server can be shared. Some screen data encoding methods with strong compression have been developed. As such improvements, for example, software such as TightVNC, eSVNC, and VDACC VNC is known.

Other software whose main function is screen sharing includes PC Anywhere and LapLink Gold from Symantec, and NetOp Remote Control from Danware and Remote Administrator from Radmin as remote management tools for PCs. In addition, as educational software using the screen sharing function, there are NetOp School of Danware and Remote Control Club for School developed by Cross Tech.

3.MulticastVNC This is an improved version of the client software that runs on Java (registered trademark) of the VNC, and has the function of relaying the screen data of the server to multiple clients. As a result, it is possible to reduce the decrease in screen update speed due to a large number of clients connecting to one server.

4. Windows Remote Desktop Windows Xp Professional, one of Microsoft's operating systems, has a screen sharing function called Remote Desktop. Similarly, the Remote Desktop server screen data can be received by the Remote Desktop client for viewing and remote control from the client. However, since it simply sends screen data, it has been pointed out that when you try to view a video image with software such as Windows Media Player on a client device, frame dropping occurs.

Further, Patent Document 1 describes a teleconference (by electrical communication means) between a conference attendee connected via a connectionless network and a conference attendee connected via a connection-oriented network having a bridge. The technology related to the teleconference device capable of holding a conference) is disclosed.<patcit num="1"><text>Japanese Unexamined Patent Publication No. 10-257053</text></patcit>

<p> However, the transmission of screen data uses TCP packets, which is a data stream type communication method. In TCP / IP communication, the reliability of communication is guaranteed, but it is necessary to reply to the reception of the packet, that is, to return the ACK packet. On the other hand, in UDP / IP communication, which is a datagram type communication method, it is not necessary to return an ACK in response to packet reception. As a result, the reliability of communication is not guaranteed, but the communication speed is improved. In addition, the header of the UDP packet is 64 bits, while that of TCP is 182 bits, and the overhead of this header may cause a decrease in communication speed. In particular, when multiple clients view the document displayed on the server or edit the document remotely from the client by screen sharing for one server as in an electronic conference system, one client It is necessary to establish a TCP connection every time, which causes a decrease in communication speed, slows down the screen update of the client, and increases the possibility that the user feels stress or performs an erroneous remote operation. For example, if 10 or more clients are connected to a single server, the screen update speed will slow down with existing software, making it unusable for practical use.</p><p> In order to solve this, it is possible to sandwich a relay device like MulticastVNC, but since the screen update is delayed due to the processing in the relay device, the client connected via the relay device is that. As the number of units increases, the screen update speed decreases accordingly.</p><p> The present invention has been made in view of the above circumstances, and provides a data communication method, an electronic conference system, a data communication method, a data communication program, and a storage medium for performing data communication by switching between TCP and UDP as needed. The purpose is to do.</p>

<p> In order to solve the above problems, the invention according to claim 1 comprises an information device that functions as a server and an information device that functions as a client of the server as components, and the server relates to at least a part of the display screen thereof. The client has a means for transmitting data to the client via a network that communicates using the IP protocol, and the client displays the screen data received from the server on a display device to display the display screen of the server. The server is a data communication method in a network system having a means for sharing the data, and the server determines the screen data depending on the type or display condition of the data in the screen area to be transmitted, the communication environment with the client, or the instruction given by the user. When transmitting, at least a part of the screen data to be transmitted is characterized by having a means for switching between transmission using a TCP packet and transmission using a UDP packet.</p><p> The invention according to claim 2 is characterized in that the server has means for transmitting screen data to a plurality of the clients by a broadcast communication method using UDP multicast or UDP broadcast packets.</p><p> In the invention according to claim 3, the server has a means for registering whether to transmit screen data in a TCP packet or a UDP packet for each application program, and is registered to transmit in a TCP packet. The screen data of the application program is transmitted in TCP packets, and the screen data of the application program registered to be transmitted in UDP packets is characterized in that it is transmitted in UDP packets.</p><p> The invention according to claim 4 is a case where the server has a means for determining whether or not a window of an application program is displayed in the foreground, and transmits screen data of the window displayed in the foreground. Is transmitted as a TCP packet, and screen data of a window or desktop screen that is not in the foreground is transmitted as a UDP packet.</p><p> According to the invention of claim 5, the server has a means for determining whether or not the application program window is active, and when transmitting screen data of the active window, the server transmits the screen data in a TCP packet. The screen data of the inactive window or the desktop screen is characterized by being transmitted as a UDP packet.</p><p> According to the invention of claim 6, the server has a means for instructing a user to transmit screen data in a TCP packet or a UDP packet to a running application program, and uses a TCP packet. The screen data of the application program registered to be transmitted is transmitted in a TCP packet, and the screen data of the application program registered to be transmitted in a UDP packet is transmitted in a UDP packet.</p><p> According to the invention of claim 7, the server has means for instructing the user whether to transmit screen data in TCP packets or UDP packets for any area of the shared screen area. However, the screen data of the screen area registered to be transmitted by the TCP packet is transmitted by the TCP packet, and the screen data of the screen area registered to be transmitted by the UDP packet is transmitted by the UDP packet.</p><p> According to the invention of claim 8, the server has a means for switching a protocol for transmitting screen data according to a user's instruction, and when the user instructs to transmit the TCP protocol, the server transmits the screen data in a TCP packet. When the user instructs to transmit the UDP protocol, the screen data is transmitted by the UDP packet.</p><p> According to the invention of claim 9, the server has a means for managing the number of connected clients and a means for switching between transmission of screen data by the TCP protocol or transmission by the UDP protocol according to the number of connected clients. However, when the number of connected clients is less than a predetermined number, screen data is transmitted in a TCP packet, and when the number of connected clients is larger than a predetermined number, screen data is transmitted in a UDP packet.</p><p> According to the invention of claim 10, the server has a means for managing whether the number of connected clients is connected by a wired LAN or is connected via a wireless LAN on the way, and is connected via the wireless LAN. The feature is that screen data is transmitted by TCP packet to the client, and screen data is transmitted by UDP packet to the client connected only by the wired LAN.</p><p> According to the invention of claim 11, the server has a means for managing whether the number of connected clients is connected by LAN or via WAN, and the server is connected to clients via WAN. On the other hand, the screen data is transmitted by a TCP packet, and when the client connected by LAN is instructed to transmit the UDP protocol, the screen data is transmitted by the UD packet.</p><p> According to the invention of claim 12, the server has means for measuring the remaining amount of screen data to be transmitted or the transmission rate of screen data, and the remaining amount of screen data transmission or the transmission rate of screen data is predetermined. If it is smaller than the value, the screen data is transmitted in a TCP packet, and if the remaining amount of the screen data transmission or the transmission rate of the screen data exceeds a predetermined value, the screen data is transmitted in a UDP packet.</p><p> According to the thirteenth aspect of the present invention, the server has means for measuring the communication speed of the network between the server and the connected client, and when the communication speed is smaller than a predetermined value, a UDP packet is used. The screen data is transmitted, and when the communication speed exceeds a predetermined value, the screen data is transmitted in a TCP packet.</p><p> The invention according to claim 14, wherein the server has means for measuring the data capacity (MTU) that can be transmitted in one packet between the server and the connected client, and the MTU is smaller than a predetermined value. In this case, the screen data is transmitted in a UDP packet, and when the MTU exceeds a predetermined value, the screen data is transmitted in a TCP packet.</p><p> The invention according to claim 15, wherein the server has a means for measuring the QoS of the network between the server and the connected client, and when the QoS is smaller than a predetermined value, screen data is displayed in a TCP packet. Is transmitted, and when the QoS exceeds a predetermined value, screen data is transmitted as a UDP packet.</p><p> According to the invention of claim 16, the server has a means for managing the priority of the connected client, and sends screen data in a TCP packet to the client to which the priority is given, and the priority is given. It is characterized by transmitting screen data in a UDP packet to a client that has not been given.</p><p> The invention according to claim 17, wherein the server receives an input event from a client and transfers the received input event to the server, and whether the client transmits the input event to the server within a certain period of time. A client that has a means to manage and sends screen data as a TCP packet to a client that has sent an input event to the server within a certain period of time, and has not sent an input event to the server within a certain period of time. The feature is that screen data is transmitted as a UDP packet.</p><p> According to the invention of claim 18, when the server transmits screen data, whether or not the screen data in the transmission area includes character data is determined by whether or not the OS is an area in which a font drawing command is issued. The screen data of the area including the character data is transmitted by the TCP packet, and the screen data of the area not including the character data is transmitted by the UDP packet.</p><p> According to the invention of claim 19, the server has means for determining whether or not the CPU load of the server exceeds a predetermined load amount, and when the CPU load is equal to or less than the predetermined load amount, a TCP packet is used. The screen data is transmitted, and when the CPU load exceeds a predetermined load amount, the screen data is transmitted by a UDP packet.</p><p> The invention according to claim 20 comprises an information device that functions as a server and an information device that functions as a client of the server, and the server provides data regarding at least a part of its display screen to the client in an IP protocol. A network system having means for transmitting data via a network for communicating with the server, and the client having means for sharing the display screen of the server by displaying screen data received from the server on a display device. In the data communication method in the above, the network system includes a relay device that relays screen data received from the server to the client, and the relay device relays screen data received from the server as a TCP packet in a UDP packet. It is characterized by having a means for transmitting to the client.</p><p> The invention according to claim 21 is characterized in that the relay device has means for transmitting screen data to a plurality of the clients by a broadcast communication method by using UDP multicast or UDP broadcast packet.</p><p> The invention according to claim 22 is characterized in that the relay device includes means for recording and reproducing screen data received from the server.</p><p> The invention according to claim 23 includes an information device that functions as a conference server and an information device that functions as a conference client of the conference server, and the conference server collects data relating to at least a part of its display screen. It has a means of transmitting to a conference client via a network that communicates using an IP protocol, and the conference client displays screen data received from the conference server on a display device to display a display screen of the conference server. The conference server is an electronic conference system having a means for sharing the screen data, and the conference server transmits the screen data depending on the type or display condition of the data in the screen area to be transmitted, the communication environment with the conference client, or the instruction given by the user. When transmitting, at least a part of the screen data to be transmitted is characterized by having a means for switching between transmission using a TCP packet and transmission using a UDP packet.</p><p> The invention according to claim 24 comprises an information device that functions as a conference server and an information device that functions as a conference client of the conference server, and the conference server collects data relating to at least a part of its display screen. The conference client has a means of transmitting to the conference client via a network that communicates using the IP protocol, and the conference client displays the screen data received from the conference server on the display device to display the display screen of the server. An electronic conference system having means for sharing, the network includes a conference relay device that relays screen data received from the conference server to the conference client, and the conference relay device is a TCP packet from the conference server. It is characterized in that it has a means for transmitting the screen data received in the above to the conference client as a UDP packet.</p><p> According to the invention of claim 25, the conference server has a means for managing the priority of the connecting conference client, and transmits screen data in a TCP packet to the priority-granted conference client. , It is characterized in that screen data is transmitted by UDP packet to a conference client to which priority is not given.</p><p> According to the invention of claim 26, the conference server receives an input event from the conference client and transfers the received input event to the conference server, and the connection conference client transmits the input event within a certain period of time. It has a means to manage whether it has been sent to the server, and to the conference client that sent the input event to the conference server within a certain time, screen data is sent by TCP packet and the input event is sent to the conference server within a certain time. It is characterized in that screen data is transmitted as a UDP packet to a client that has not transmitted to.</p><p> The invention according to claim 27 is a data communication method in a network system including an information device functioning as a server and an information device functioning as a client of the server as components, and at least one of the display screens of the information device in the server. The display screen of the server by transmitting the data related to the unit to the client via a network that communicates using the IP protocol and displaying the screen data received from the server by the client on the display device. When the server transmits the screen data, the data is transmitted depending on the type or display condition of the data in the screen area to be transmitted, the communication environment with the client, or the instruction given by the user. It is characterized by having a step of switching whether at least a part of the screen data to be transmitted is transmitted using a TCP packet or a UDP packet.</p><p> The invention according to claim 28 is a network including an information device that functions as a server, an information device that functions as a client of the server, and a relay device that relays screen data received from the server to the client. A data communication method in a system, in which the server transmits data regarding at least a part of the display screen to the client via a network that communicates with the client, and the server in the client. A step of sharing the display screen of the server by displaying the screen data received from the server on a display device, and a step of transmitting the screen data received from the server in a TCP packet to the client in a UDP packet in the relay device. It is characterized by having.</p><p> The data communication program according to claim 29 is characterized by causing a computer to execute the data communication method according to claim 27 or 28.</p><p> The computer-readable storage medium according to claim 30 is characterized by storing the data communication program according to claim 29.</p>

<p> According to the present invention, as is clear from the description of the following examples, it is a data stream type communication method in which it is necessary to return an ACK packet in response to a normally received packet to a transmitted packet. Not only communication by a certain TCP packet, but it is fast because there is no need to reply ACK, but using UDP packet, which is a datagram type communication method that does not guarantee reliability or packet reception order, screen data It enables transmission, high-speed screen sharing, and communication between the screen sharing server and screen sharing client.</p>

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

FIG. 1 is a schematic diagram showing a network configuration of a data communication system according to an embodiment of the present invention. In the data communication system, an information processing device (server 1) that functions as a server and an information processing device (clients 2a, 2b, ..., 2n) that function as a client of the server 1 are connected via the IP network 10. Has been done. As an IP network, for example, there is an Ethernet (registered trademark) LAN defined by IEEE802.3.

Server 1 includes a display device, all or part of the display contents displayed on the display device as a shared screen, IP networks to transmit the shared screen data to the client 2 connected to the over click 10. By receiving the shared screen data, the client 2 can display the shared screen of the server 1 on the display device of the client. As the method of sharing the screen, the conventional technique can be used as it is, so detailed description thereof will be omitted. As a general rule, the server application detects the updated portion of the screen data of the server, divides and encodes the screen update area by a certain algorithm, and sends it to the client. The receiving client decodes the screen data of the updated area and updates the data of the shared screen.

In the conventional invention, the transmission of screen data is transmitted by TCP packet. TCP is a connection-oriented (CO-type) protocol, and is a highly reliable protocol that can cover the loss of IP packets in connectionless (CL-type) IP. However, there is a problem that the data transfer efficiency is lowered. On the other hand, the present invention has a function of transmitting screen data as a UDP packet. UDP is a CL-type protocol, and although its data reliability is inferior to that of TCP, it has the advantage of being capable of high-speed data transfer.

FIG. 2 is a diagram showing a processing flow of a method of screen sharing by transmitting only UDP packets. In the initial state, server 1 is in a state where the UDP port is opened and a socket for UDP is waiting for a connection from a client. The client 2 sends a UDP packet to request the server 1 to share the screen (step S1). When the server that receives the packet permits sharing, it sends a packet notifying the permission to the client that requested sharing, and also remembers the network address (IP address) of the client that allowed sharing (steps S2, S3). .. As a result, screen data can be transmitted as a UDP packet to the client that has made the sharing request, and an input event from the client is received to enable remote operation.

The client that receives the permission from the server 1 remembers the network address of the server and prepares to receive and display the screen data from the server (step S4). That is, the UDP port is opened, and the socket for UDP listens for the communication packet from the server 1.

It should be noted that authentication may be performed instead of requesting / permitting by one round-trip UDP packet communication in steps S1 and S2. Further, although omitted in the figure, the client and the server may include communication for adjusting mutual settings regarding the size of the screen to be shared and the encoding method.

Then, when ready, server 1 sends the initial screen data to client 2 via UDP (step S5a). After that, every time the shared screen area of server 1 is updated, the screen data is transmitted as a UDP packet (steps S5b, S5c, S5d). In addition, when the client performs remote control, the input event is sent to the server as a UDP packet (step S6).

When terminating screen sharing, the client sends a disconnection request to the server as a UDP packet (step S7). The server that receives the disconnection request sends a confirmation of disconnection to the client as a UDP packet, and cancels the setting of transmission / reception to the client (steps S8 and S9). On the other hand, the client that receives the disconnection OK cancels the setting for sending and receiving to the server (step S10).

Here, in the case of the communication method shown in FIG. 2, since the server and the client unilaterally transmit data to each other by UDP packets, they communicate by when the other party terminates the screen sharing application or by a network error. It is not possible to detect whether it is disabled or the packet of disconnection OK has not arrived. Therefore, the other party is monitored by the algorithm shown in Fig. 3.

When the connection is established, the server and the client use a timer to send a confirmation packet for confirming the connection of the other party at predetermined time intervals, for example, every minute (steps S11 and S12).

On the other hand, at the same time, a timer is used to monitor whether or not any packet from the other party is received. For example, check with a 2-minute timer. When the packet reception is confirmed (step S21) and the data or confirmation packet is received (step S22 / Yes), the timer is started (step S23). If any packet is received within 2 minutes (step S24 / Yes), the timer is restarted (step S23). If it is not received within 2 minutes (step S24 / No, S25 / Yes), it is determined that the other party has disconnected, and the setting for sending and receiving with the other party is canceled (step S26).

In this embodiment, since it is transmitted by UDP packets, packets such as screen data and input events may be lost on the network, and the communication is not reflected in the destination. In addition, the transmission order of packets on the transmitting side and the receiving order of packets on the destination may be different. Therefore, according to the present invention, in a short-distance network such as a LAN, when the server and the client are connected on a network line with a large amount of data communication, the shared screen is updated more frequently than the certainty of the screen data. It is preferable to use it in such cases.

In FIG. 1, a desktop PC or a workstation equipped with a large screen display device is used as the server 1. Further, although a racktop type PC (notebook PC) is used as the client 2, the configuration is not necessarily limited to such a configuration. That is, in the present invention, the server is a computer having screen data shared in the network, and therefore, a desktop PC may be used as a client even if it does not have a large screen display device.

In addition, specific examples of networks are not limited to Ethernet (registered trademark) wired LANs, such as wireless LANs, IrDA, AppleTalk, ISDN, telephone exchange networks, digital leased lines, FR networks, and so on. It may be connected by such as, or may be connected by a combination of these.

The second embodiment is a modification of the method described in the first embodiment. FIG. 4 is a diagram showing a processing flow of a method in which a TCP / IP connection is used for the connection between the server and the client, and a UDP packet is used for the transmission of screen data and input events. Both server 1 and client 2 provide two sockets, TCP and UDP. When connecting with TCP packets (steps S31, S32, S33), screen data and input events are sent and received using UDP sockets with the same network address (steps S34a to d, S35). When the TCP connection is requested to be disconnected and the TCP connection is disconnected, the UDP socket is also terminated, and the listening for screen data and input events is terminated (steps S36, S37, S38). Since TCP is a connection-type communication protocol, the management of the connection with the other party is left to the connection management of the TCP connection, and it is not necessary to implement the monitoring function on the application side (upper layer) in particular.

Example 3 is an example in which UDP multicast or UDP broadcast is further applied to the method of Example 2, and the processing flow is shown in FIG. Table 1 is an example of network addresses in the network configuration.

<tables num="1"><img file="JP2005348262A_D0001.tif" /></tables>

&lt;1. Broadcast&gt; Server 1 is a UDP socket, listening for input events from (133. 139. 100. 111) and (133. 139. 100. 222). Also, the screen data is broadcasted within the same subnet with the broadcast address (255. 255. 255. 255). UDP packets are used as packets. Clients 2a and 2b are waiting for the broadcast of screen data from (133. 139. 100. 100). It also sends an input event to (133. 139. 100. 100) via UDP unicast. The client receives the screen data broadcast from the server and shares the screen.

&lt;2. In the case of multicast&gt; When the connection between the server and the client is established, the server 1 notifies all the clients of the multicast address. Server 1 sets the range of the multicast so that it reaches the client. That is, set the TTL (Time To Live) value appropriately.

Server 1 is a UDP socket that listens for input events from (133. 139. 100. 111) and (133. 139. 100. 222). In addition, the screen data is multicast-transmitted at the multicast address (233. 233. 233. 233). UDP packets are used as packets. Clients 2a and 2b are listening for screen data from the multicast address (233. 233. 233. 233) specified by server 1. It also sends an input event to (133. 139. 100. 100) via UDP unicast.

The fourth embodiment is an example of a method of switching between TCP and UDP depending on the type of application of the server, and FIG. 6 is an example of displaying the screen of the server. When the server 1 transmits the screen data 11 of the application set to be transmitted by the TCP packet in advance, the server 1 transmits by the TCP packet, and the other screen area (UTD shared application screen 12, etc.) uses the UDP packet. And send.

For example, consider a case where only the word processor software Wordproc and the spreadsheet software Matrixcalc are transmitted as TCP packets as an application, and the screen is surely displayed on the client side sharing the screen. Register the path of the executable file of the application you want to send by TCP packet in the screen sharing server program. For example, register C: \ Program Files \ Wordproc \ Wordproc.exe C: \ Program Files \ Matrixcalc \ Matrixcalc.exe.

FIG. 7 is a diagram showing an algorithm for a method of transmitting by either TCP or UDP for each application of the server. When the display screen of the server 1 is updated, the screen sharing program divides the area by a certain algorithm and encodes it, but whether or not the divided screen area is the window area of the registered application. Is determined (step S51). If it is the window area of the registered application (step S52 / Yes), it is instructed to send it as a TCP packet after encoding (step S53). On the other hand, in the case of other areas (step S52 / No), an instruction is given to transmit in a UDP packet (step S54).

FIG. 8 illustrates a processing flow for a method of switching and transmitting a TCP packet and a UDP packet in such a case. Similar to the processing flow shown in Example 2, connection management is performed by TCP / IP (steps S61 to S63 and S68 to S70), and screen data is switched between TCP packets and UDP packets as necessary and transmitted. (Steps S64 to S66).

The fifth embodiment is an example of a method of switching between TCP and UDP depending on whether or not the window of the server is located in the foreground, and FIG. 9 is an example of displaying the screen of the server. In FIG. 9, when transmitting the screen data of the front window 13, the TCP packet is used for transmission, and the other screen areas (rear window 14, etc.) are transmitted using the UDP packet.

FIG. 10 is a diagram showing an algorithm for a method of switching between TCP and UDP depending on whether or not the server window is located in the foreground. When the display screen of the server 1 is updated, the screen sharing program divides the area by a certain algorithm and encodes it. Here, whether or not the divided screen area is the window area in the foreground. Determine if (step S71). If it is the frontmost window area (step S72 / Yes), it is instructed to send it as a TCP packet after encoding (step S73). On the other hand, in the case of other areas (step S72 / No), an instruction is given to transmit in a UDP packet (step S74).

The sixth embodiment is an example of a method of switching between TCP and UDP depending on whether or not the window of the server is active, and FIG. 11 is an example of displaying the screen of the server. For example, when the desktop area is clicked with the mouse, the window in the foreground is not always the active window, but even if it is inactive, TCP packets can be sent if it is in the foreground. Or, if it is inactive, it is possible to send it as a UDP packet even if it is in the foreground.

In FIG. 11, when transmitting the screen data of the active window 15, the TCP packet is used for transmission, and the other screen areas (inactive windows 16a, 16b, etc.) are transmitted using the UDP packet.

FIG. 12 is a diagram showing an algorithm for a method of switching between TCP and UDP depending on whether the server window is active or not. When the display screen of the server 1 is updated, the screen sharing program divides the area by a certain algorithm and encodes it, but determines whether or not the divided screen area is the active window area. (Step S81). If the divided screen area is the active window area (step S82 / Yes), it is instructed to send it as a TCP packet after encoding (step S83). On the other hand, in the case of other areas (step S82 / No), an instruction is given to transmit in a UDP packet (step S84).

The seventh embodiment is an example of a method of switching between TCP and UDP depending on whether or not the window of the server is a document specified by the user, and FIG. 13 is an example of displaying the screen of the server. In FIG. 13, when transmitting the screen data of the user-designated document window 17, the TCP packet is used for transmission, and the other areas (non-user-designated document window 18 and the like) are transmitted using the UDP packet.

FIG. 14 is a diagram showing an algorithm for a method of switching between TCP and UDP depending on whether or not the server window is a document specified by the user. When the display screen of the server 1 is updated, the screen sharing program divides the area by a certain algorithm and encodes it, but whether or not the divided screen area is the window area of the document specified by the user. Is determined (step S91). If the divided screen area is the active window area (step S92 / Yes), it is instructed to send it as a TCP packet after encoding (step S93). On the other hand, in the case of other areas (step S93 / No), an instruction is given to transmit in a UDP packet (step S94).

For example, the window to be sent by TCP is managed for each document, and the document path is registered in the screen sharing program. For example, register as c: \ My Documents \ Statistics \ 2003Jan.pdf. Then, in the window of the application (for example, Acrobat Reader (registered trademark)) that opens this document, the screen data of the window area in which the document is opened is transmitted as a TCP packet.

As a modification of the seventh embodiment, it is possible to specify it not for each document but for each window. For example, specify the window of the program Acrobat Reader® that opens the document c: \ My Documents \ Statistics \ 2003Jan.pdf. For example, in the screen sharing program, a function to select the window to send the screen by TCP packet is provided, and when this function is executed, the window clicked with the mouse first after execution is registered to send the screen data by TCP packet and registered. Try to end the function.

Example 8 is an example of a method of switching between TCP and UDP depending on whether or not the window of the server is a screen area specified by the user, and FIG. 15 is an example of displaying the screen of the server. As shown in FIG. 15, when transmitting the screen data of the user-designated area 19, the TCP packet is used for transmission, and the other areas are transmitted using the UDP packet.

FIG. 16 is a diagram showing an algorithm for a method of switching between TCP and UDP depending on whether or not the server window is a screen area specified by the user. When the display screen of the server 1 is updated, the screen sharing program divides the area by a certain algorithm and encodes it, but whether or not the divided screen area is the screen area specified by the user. Is determined (step S101). If the screen area is specified by the user (step S102 / Yes), after encoding, an instruction is given to send as a TCP packet (step S103). On the other hand, in the case of other areas (step S102 / No), an instruction is given to transmit as a UDP packet (step S104).

An example of modification of Example 8 is shown below. The screen sharing program provides a function to select the screen area for screen transmission by TCP packet, and when this function is executed, the window that was first clicked with the mouse after execution is set as the upper left corner, and the window is moved while the mouse is clicked. The position where the mouse click is released is registered so that the screen data is transmitted by TCP packet in the rectangular area with the lower right vertex, and the registration function is terminated. Acquire and register the coordinates of the first click and the coordinates of the last released location. For example, if you start clicking at (100,200), drag it to (400,700), and release the mouse, the rectangular area with these two points diagonally as two vertices will be the TCP transmission area.

Further, in Examples 4 to 8, it is possible to give priority to the transmission in the TCP packet over the transmission in the UDP packet. As a result, it is possible to quickly and surely display the screen data of the required part on the screen of the client.

In the ninth embodiment, a means for the user to switch between a TCP packet and a UDP packet in transmitting screen data will be described. FIG. 17 is a diagram illustrating a screen for instructing the switching of the communication protocol, and provides a function for selecting TCP / UDP and instructing the switching in the screen sharing program of the server.

FIG. 18 is a diagram showing a flow of communication protocol switching operation. In this way, it is assumed that the initial default setting is set to send screen data in TCP packets to the client that has established a TCP connection (step S111). When instructed to transmit in a UDP packet by the communication means switching function of the screen sharing program as shown in FIG. 17 (step S112 / Yes), the screen data is transmitted in the UDP packet (step S113). It is also possible to return to a TCP packet (step S114 / Yes, S111). For example, when playing a video image on a server, when performing a slide show of a presentation using abundant animation effects, when the amount of screen data transmitted is large and speed is required rather than certainty, UDP packets Switch to the transmission mode with. On the other hand, when sharing data containing only characters, it is a problem if it is not displayed reliably, so switch to using TCP packets.

Further, as a modification of Example 9, instead of switching from the program on the server side, switching between TCP and UDP may be requested from the client side.

In the tenth embodiment, a means for selecting whether to use a TCP packet or a UDP packet for each user will be described. FIG. 19 is a diagram illustrating a screen for instructing the switching of the communication protocol for each user, and it is possible to select whether to use TCP or UDP for each client. The screen sharing server has a function of changing the settings for each client by using the IP address of the client, the host name, or an arbitrary identification name (for example, the name of the user) exchanged by the screen sharing program. It has a function to manage and display whether screen data is transmitted to the currently connected client using TCP or UDP, and to change the setting.

In Fig. 19, the checked "Ai Kamio" and "Saga Suzeso" are set as the clients to send by TCP, and the unchecked "Kukako Kaki" is in UDP. It is set as a sending client. Further, as shown in FIG. 19, by pressing the "Everyone TCP" button, all members can send in TCP packets, and by pressing the "Everyone UDP" button, all members can send in UDP packets. If there are a large number of clients participating in the network, such means are also effective.

The eleventh embodiment is an example of a method of switching between TCP and UDP depending on the number of connected clients, and FIG. 20 is a diagram showing the algorithm. The screen sharing program on server 1 monitors the number of connected clients on the network (step S121). When it is detected that the number of connected devices exceeds a predetermined number (step S122 / Yes), the communication method used for transmitting screen data is switched from TCP to UDP (step S124). As a result, when the number of clients is large, the amount of communication increases and the delay in the screen display of each client is reduced. If the number of clients connected to the network is less than or equal to the specified number (step S122 / No) during UDP communication, the communication method is switched from UDP to TCP (step S123). ). When the number of connections is small and the amount of data communication on the network is small, communication is performed using highly reliable TCP.

Further, as a modification of the eleventh embodiment, when the number of connected devices exceeds a predetermined number, the dialog shown in FIG. 19 may be displayed and a client to be switched to UDP may be specified, or all clients may be switched to UDP. You may.

Example 12 is an example of a method in which TCP and UDP are switched and transmitted by a communication interface of a client, and FIG. 21 is a diagram showing the algorithm. The screen sharing program on the server 1 has a function of acquiring whether the client is connected to the network by a wired LAN or is connected via a wireless LAN. The client can obtain information about its own communication interface from the system information of the OS. The client screen sharing program transmits this communication interface information when connecting to the server 1. On the other hand, the server 1 determines whether or not it is a wireless LAN from the information of the communication interface sent from the client. For example, it is judged from the information such as the name of the interface or, in the case of wireless LAN, which standard of IEEE 802.11.

As shown in FIG. 21, when the connection from the client is accepted, the communication interface information of the client is acquired and it is determined whether it is a wired LAN or a wireless LAN (step S131). In the case of a wired LAN (step S132 / No), the screen data is set to be transmitted in a UDP packet (step S134), while in the case of a wireless LAN (step S132 / Yes), the screen data is transmitted in a TCP packet. (Step S133).

Further, as a modification of the twelfth embodiment, when the server 1 itself communicates with the wireless LAN interface, the default setting may be to transmit the screen data in a TCP packet. In addition, the target of the interface set to transmit by TCP may include not only wireless LAN but also IrDA and Bluetooth (registered trademark).

As another example of modification, the screen sharing program of the server 1 and the client is provided with a function for exchanging information about the communication interface between the server 1 and the client, and this information is exchanged when the connection between the server 1 and the client is established. May be good. If it is determined that the client is using the wireless interface, the screen data is set to be transmitted by TCP packet as in the above example, and if it is determined that the client is using the wired LAN, the UDP packet is set to be used.

Example 13 is an example of a method in which TCP and UDP are switched and transmitted depending on the connection path of the client, and FIG. 22 is a diagram showing the algorithm. The screen sharing program on the server 1 has a function of acquiring whether the client is connected via LAN or via WAN.

There are the following methods to determine whether the client has a LAN connection or a WAN connection. (Method 1) The client can obtain information about its own communication interface from the system information of the OS. The client screen sharing program sends this communication interface information when connecting to server 1. On the other hand, the server 1 makes the first judgment from the information of the communication interface sent from the client. If the interface is a modem, it is determined that you are connected via WAN.

(Method 2) Check the network information of the server 1 itself with the IP address of the client, and if the IP address of the client is a global IP, it is determined that the client is connecting from the WAN.

(Method 3) Ask the client for routing information using the traceroute command, which is one of the network commands. Therefore, if it is confirmed that the client is out of the LAN, it is determined that the client is connecting from the WAN.

As shown in FIG. 22, when the connection from the client is accepted, it is determined whether the client is a LAN connection or a WAN connection (step S141), and in the case of a LAN connection (step S142 / No), the screen data is displayed in a UDP packet. Set to send (step S144). On the other hand, in the case of WAN (step S142 / Yes), the screen data is set to be transmitted by TCP packet (step S143).

Example 14 is an example of a method of switching between TCP and UDP depending on the remaining amount of encoded data to be transmitted, and FIG. 23 is a diagram showing the algorithm. The screen sharing program on the server 1 has a means for measuring the amount of data required to encode and transmit the update area when the screen is updated. This amount is obtained from the part of the screen sharing program that encodes the update area (step S151).

At some point, if the remaining amount of encoded data to be transmitted exceeds a certain level, it will be transmitted as a UDP packet. For example, set when the remaining amount exceeds 500 kilobytes. As shown in FIG. 23, when the update amount exceeds a predetermined threshold value (step S152 / Yes), the screen data is transmitted in the UDP packet with priority given to speed (step S153), and otherwise (step S152). / No) is sent as a TCP packet while confirming the reception of the client (step S154). In addition, when UDP communication is performed and the remaining amount becomes less than the threshold value, the transmission is automatically returned to TCP packet transmission.

As a modification of Example 14, it may be determined only by the amount of bitmap data that needs to be updated before encoding. For example, set when the remaining amount exceeds 1 megabyte.

Example 15 is an example of a method of switching between TCP and UDP depending on the transmission data rate, and FIG. 24 is a diagram showing the algorithm. The screen sharing program on the server 1 has a means for measuring the data transmission rate (the amount of data transmitted per unit time) transmitted per unit time when the screen is updated. For example, it is obtained from the transmission bit rate of the OS system information (step S161).

If the data rate is less than a predetermined threshold (step S162 / Yes), the screen data is transmitted in UDP packets with priority given to speed (step S163), otherwise (step S162 / No) is a TCP packet. Send while confirming the reception of the client in (step S164). In addition, when the data rate becomes higher than the threshold value during UDP communication, the transmission is automatically returned to the TCP packet.

Example 16 is an example of a method of switching between TCP and UDP depending on the communication speed between the server and the client, and FIG. 25 is a diagram showing the algorithm. The screen sharing program started on the server 1 and the client measures the communication speed between the server 1 and the client when the client connects to the server 1. As a method of measuring the communication speed between the server 1 and the client, a conventional measurement method can be used. For example, there is a way to measure the time required by sending several megabytes of data, similar to what is done on websites on the Internet.

As shown in FIG. 25, when the connection from the client is accepted, the server 1 measures the communication speed between the server 1 and the client (step S171), and when the communication speed is equal to or higher than a predetermined value (step S172 /). Yes) is set to send screen data in TCP packets (step S173), and if the value is less than a predetermined value (step S172 / No), screen data is set to be sent in UDP packets (step S174). Here, the reference value for this determination can be, for example, 1 megabyte / second (Mbps).

Example 17 is an example of a method in which TCP and UDP are switched and transmitted by the MTU between the server and the client, and FIG. 26 is a diagram showing the algorithm. The screen sharing program started on the server 1 has a means for measuring the MTU between the server 1 and the client when the client connects to the server 1. A conventional measurement method can be used as a method for measuring the MTU between the server and the client. For example, you can use one of the network commands, the ping command, to find out the maximum amount of data that is not fragmented. The principle is shown below.

ping -f -l &lt;packet size&gt; &lt;[destination host name] or [destination IP address]&gt; However, -f ... option to prohibit packet division -l ... option to specify the packet size

By sending the ping command as described above, it is possible to confirm to the client how many bytes of the packet are transmitted without being divided.

As shown in FIG. 26, when receiving a connection from the client, server 1 measures the MTU between the server and the client (step S181), and if the MTU is greater than or equal to a predetermined value (step S182 / Yes). , Set to send screen data in TCP packet (step S183), and if it is less than the specified value (step S182 / No), set to send screen data in UDP packet (step S184). Here, the reference value for this determination can be, for example, 1 megabyte.

In the above algorithm, when the MTU is small, the amount of data that can be sent in one packet is small, and if a TCP ACK reply is performed, the communication efficiency deteriorates, so the data is transmitted in a UDP packet.

Example 18 is an example of a method in which TCP and UDP are switched and transmitted by QoS between the server and the client, and FIG. 27 is a diagram showing the algorithm. The screen sharing program that starts on server 1 measures QoS when a client connects to server 1, and measures bandwidth, minimum guaranteed speed, maximum delay time, packet interval, peak speed, network delay and jitter, and packet loss. It has a means to measure and evaluate the communication quality.

As shown in FIG. 27, when the connection from the client is accepted, the server 1 measures the QoS between the server and the client (step S191), and the communication quality meets a certain standard (step S192 /). Yes) is set to send screen data in UDP packets (step S194), and if it is determined that it is not satisfied (step S192 / No), screen data is set to be sent in TCP packets (step S193). .. However, it is not always necessary to measure all the parameters.

Example 19 is an example of a method in which TCP and UDP are switched and transmitted according to the priority of the client, and FIG. 28 is a diagram showing the algorithm. The screen sharing program started on the server 1 has a function of prioritizing clients in a screen sharing system in which a plurality of clients can connect and share screens at the same time. For example, when the screen of server 1 is updated, first, the screen data is surely transmitted by TCP packet to the client to which the priority is given (steps S201, S202 / Yes, S203), and the client without the priority has no priority. After that, the screen data is transmitted as a UDP packet without confirmation of reception (steps S201, S202 / No, S204).

In the 20th embodiment, a method of registering the priority right and the control right of the remote control on the shared screen in association with each other is further described. As shown in Fig. 29, only the client who has the control right (client "Ai Kamio (192. 168.0. 2)") can remotely control the client who has the control right. Therefore, the screen data is transmitted by TCP packet before the client that does not own it. On the other hand, a client without control only browses the shared screen, and the input event is not accepted by the server. In addition, after transmitting the screen data to a client without control right, the screen data is received by UDP packet without confirmation of reception.

In the 21st embodiment, in a screen sharing system in which a plurality of clients can be connected to share a screen at the same time and remotely controlled, a method of giving a higher priority to screen data transmission to a client being remotely controlled Explaining. The server 1 has a function of determining whether or not the client is in remote control. As shown in FIG. 30, when the client performs a remote operation, the server 1 turns on the remote operation flag of the client (step S211). Therefore, the timer is started (step S212), and it is determined whether or not the client is in operation based on whether or not the client has performed an operation within a certain period of time, for example, within 1 minute. For example, if there is no input event within 1 minute, Server 1 determines that the client is not performing remote control (S213 / No, S214 / Yes) and turns off the remote control flag (step S215). ).

In addition, when the screen data is updated, as shown in Fig. 31, the server 1 checks the ON / OFF of the remote operation flag (step S221), and for the client with the flag ON, it is better than the client with OFF. First, the screen data is transmitted by TCP packet (step S222). On the other hand, for the client whose flag is OFF, after the screen data is transmitted to the client which is ON, the screen data is transmitted as a UDP packet without confirmation of reception (step S223). As a result, it is possible to give priority to quality and speed to the person performing the operation, transmit screen data, and not feel inconvenience in remote control.

As a modification of Example 21, for example, as in Example 20, it is possible to combine management of control rights. In that case, the screen data is always transmitted as a UDP packet to the client without the control right without the control right.

Example 22 is an example of a method of switching between TCP and UDP depending on whether the shared screen data is font data or image data, and FIG. 32 is a diagram showing the algorithm. When the screen is updated, first, the image data of the part where the characters are displayed is surely transmitted by the TCP packet, then the image data of the other area is transmitted by the UDP packet, and the characters are displayed on the client screen. Make the part of.

When the shared screen is updated, the screen sharing program started on the server 1 divides the update area, encodes it, and sends the screen data to the client. Here, the OS is in the divided screen area. It is determined whether or not the font drawing command is issued (step S231), and if it is the area where the font drawing command is issued (step S232 / Yes), the screen data is transmitted by the TCP packet (step S233). On the other hand, for the part where the font drawing command is not issued (step S232 / No), the screen data is transmitted by the UDP packet (step S234). Further, as the transmission order, it is desirable to transmit the font drawing portion first. As a method of detecting a font drawing command, for example, a screen sharing program started on the server 1 examines a GDI (Graphic Device Interface) command in a divided area and executes it depending on whether or not a font drawing command is issued.

Example 23 is an example of a method of switching and transmitting TCP and UDP according to the load status of the CPU, and FIG. 33 is a diagram showing the algorithm. The screen sharing program on the server 1 has a means for measuring the load amount of the CPU when the screen is updated. For example, it is acquired from the CPU load information of the OS system information.

As shown in FIG. 33, when the CPU load is measured (step S241) and the CPU load exceeds a predetermined threshold (step S242 / Yes), the screen data is sent to the UDP packet with priority given to speed. If it is transmitted (step S244), otherwise (step S242 / No), it is transmitted while confirming the reception of the client by the TCP packet (step S243). Also, when the CPU load becomes less than the threshold value, it automatically returns to TCP packet transmission. As a threshold setting, for example, if the CPU load exceeds 80% for 1 second or more, switch to UDP packet.

When the CPU load is heavy, the amount of screen updates is large, so it often takes time to encode the screen data of the updated part. Further, when the CPU load is large, the encoding process of the screen data of the update portion becomes slow, so that it is necessary to reduce the CPU load of the communication processing portion, that is, to increase the communication speed. Therefore, by switching from TCP packets to UDP packets, it is possible to reduce the CPU load and the delay of the client screen display.

Further, by appropriately combining the above Examples 1 to 23, a high-speed screen sharing system can be realized. For example, for a read-only client without control rights, screen data is unconditionally transmitted by broadcast communication by UDP multicast, and for a client with control rights, the active window in the foreground is displayed. The data of the part where the character drawing is preferentially transmitted by the TCP packet, and the other part is transmitted by the UDP packet. Alternatively, the screen data is unconditionally sent as a UDP packet to the client connected by the wired LAN, and only the specified application is prioritized by the TCP packet to the client connected by the wireless LAN. It is possible to make a combination such as sending to the client and sending the other parts as UDP packets.

In Examples 24 to 27, a case where a relay device (proxy) is provided in the network will be described. FIG. 34 is a schematic diagram showing a network configuration of a data communication system according to an embodiment of the present invention. The data communication system includes an information processing device (server 1) that functions as a server, an information processing device (clients 2a, 2b, ..., 2n) that functions as a client of the server 1, and a server 1 and a client 2. It is configured to include a relay device 3 in between. The server 1, the client 2, and the repeater 3 are, for example, personal computers, and these are connected by, for example, a network by an Ethernet (registered trademark) LAN defined by IEEE802.3.

The server 1 receives the entire Desktop screen displayed on its display device or a part of the screen displayed on the desktop screen by transmitting the shared screen data to the connected client 2. The client can display the shared screen of the server 1 on the display device. In this example, a relay device (hereinafter abbreviated as proxy) exists between server 1 and client 2, receives screen data received from server 1, and transfers screen data to the client connected to the proxy. To do. This makes it possible to reduce the load on the server and improve the communication efficiency when a large number of clients are connected to the server 1.

Further, as in the case of the first embodiment, the event from the input device such as the mouse or keyboard of the client is transmitted to the server 1 in the shared screen displayed on the client 2, so that the server is remote from the client. The operation is also possible, but this may also be performed through a proxy, or a communication method may be adopted in which the input event is sent directly to the server without going through the proxy.

The principle will be described below. As for the screen sharing method using a proxy, the conventional technology such as Multicast VNC, VNC Proxy, or VNC Reflector can be used as it is, so the details will be omitted. As a general rule, the server application detects the updated part of the screen data of the server, divides the updated area of the screen by a certain algorithm, encodes it, and sends it to the proxy. The proxy starts as a kind of screen sharing client. The proxy also functions as a server that transfers the received screen data to the connected client, and the receiving client decodes the screen data in the updated area and updates the shared screen data. This makes it possible to display the screen of the server.

A processing flow of a method of screen sharing via a proxy will be described with reference to FIG. 35. It is assumed that the connection management between the server 1 and the proxy 3 and the connection management between the proxy 3 and the client 2 are performed by the TCP connection management as in the second embodiment. That is, the proxy 3 makes a TCP connection request to the server 1 (step S301), the server 1 responds, and the TCP connection between the server and the proxy is established (steps S302 and S303). Further, the client 2 makes a TCP connection request to the proxy 3 (step S304), the proxy 3 responds, and a TCP connection between the client and the proxy is established (steps S305 and S306). When the proxy 3 receives the screen data from the server 1 while confirming the reception with the TCP packet (steps S307 and S308), the proxy 3 sends the screen data to the connected client in the UDP packet (step S309). ..

As an example of changing the processing flow of FIG. 35, for example, the proxy 3 may transfer the screen data received from the server 1 in the UDP packet to the client 2 in the UDP packet. Further, for example, the screen data received from the server 1 in the UDP packet may be transferred to the client 2 in the TCP packet. Alternatively, there is also a method of transferring the screen data received from the server 1 in a TCP packet to the client 2 in a TCP packet.

Further, as an example of the configuration change shown in FIG. 34, for example, as shown in FIG. 36, the client 2 does not necessarily have to receive the screen data from the server 1 through the proxy 3, and the client directly connected to the server 1. 2a, 2b, ..., 2i and clients 2j, ..., 2n connected via a proxy may coexist. In this case, it is possible to improve the communication efficiency by connecting directly to the server 1 only for the client having the priority to perform remote control, and connecting via the proxy 3 for the client who only browses the shared screen. ..

Example 25 is an example in which UDP multicast or UDP broadcast is applied by a proxy, and FIG. 37 shows the processing flow. Table 2 is an example of network addresses in the network configuration.

<tables num="2"><img file="JP2005348262A_D0002.tif" /></tables>

&lt;1. For broadcast&gt; Server 1 is listening to input events from (133. 139. 100. 111) and (133. 139. 100. 222) on the UDP socket. Also, the screen data is broadcasted within the same subnet with the broadcast address (255. 255. 255. 255). Clients 2j and 2n are waiting for the broadcast of screen data from (133. 139. 100. 100). It also sends an input event to (133. 139. 100. 100) via UDP unicast. Server 1 makes a TCP connection with proxy 3 and sends screen data as a TCP packet to the address of proxy 3 (133. 139. 100. 55). On the other hand, the TCP socket listens for input data from proxy 3. Proxy 3 is a TCP socket and listens for screen data from the address of server 1 (133. 139. 100. 100).

When the proxy 3 receives the screen data from the server 1, the proxy 3 broadcasts the received screen data to the same subnet at the broadcast address (255. 255. 255. 255). UDP packets are used as packets. Proxy 3 is a UDP socket that listens for input events from the client addresses (133. 139. 100. 111) and (133. 139. 100. 222). When the proxy 3 receives the input event from the client, it forwards the input event to the server 1 in a TCP packet. Clients 2j and 2n are waiting for the broadcast of screen data from (133. 139. 100. 55). It also sends an input event to (133. 139. 100. 55) via UDP unicast. The client receives the screen data broadcast from the proxy 3 and shares the screen.

&lt;2. For multicast&gt; When the connection between proxy 3 and client 2 is established, server 1 notifies all clients of the multicast address. Proxy 3 sets the range of the multicast so that it reaches the client. That is, set the TTL (Time To Live) value appropriately. Server 1 makes a TCP connection with proxy 3 and sends screen data as a TCP packet to the address of proxy 3 (133. 139. 100. 55). On the other hand, the TCP socket listens for input data from proxy 3.

Proxy 3 is a TCP socket and listens for screen data from the address of server 1 (133. 139. 100. 100). When the proxy 3 receives the screen data from the server 1, the proxy 3 multicasts the received screen data with the multicast cast address (233. 233. 233. 233). UDP packets are used as packets. Proxy 3 is a UDP socket that listens for input events from the client addresses (133. 139. 100. 111) and (133. 139. 100. 222). When the proxy 3 receives the input event from the client, it forwards the input event to the server 1 in a TCP packet.

Clients 2j and 2n are listening for screen data from the multicast cast address (233. 233. 233. 233) specified by the proxy. It also sends an input event to (133. 139. 100. 100) via UDP unicast.

In the 26th embodiment, the session recording and playback functions in the proxy will be described. The proxy 3 adds the screen data received from the server 1 to the file in the order in which the server 1 sends the TCP packet together with the time information, and accumulates the screen data. For example, the time recording starts from the recording start time. When the client 2 connects to the proxy 3 and requests the playback of the recorded screen data of the server 1, the proxy 3 sequentially sends the recorded screen data to the client in accordance with the instruction of the elapsed time from the recording start time. Send with. FIG. 38 shows the situation. After the connection between the server 1 and the proxy 3 is completed, the client 2 can connect to the proxy 3 and view the screen sharing asynchronously. Needless to say, client 2 can only browse during playback, and proxy 3 does not accept input events from client 2.

It is also possible to view the screen data after the client 2 connects to the proxy 3 and the proxy 3 and the server 1 start the connection while the proxy 3 is recording the screen data from the server 1 with a time lag. is there.

Further, as a modification of the 26th embodiment, a function of fast-forwarding the screen data recorded in the proxy may be provided. When the proxy 3 transmits the screen data to the client 2 based on the recording time data, it is possible to shorten the transmission interval by setting the elapsed time to, for example, half the recorded value.

In Examples 27 and 28, an electronic conferencing system to which the data communication system described in Examples 1 to 26 is applied will be described. For example, in the network configuration shown in FIG. 1, an electronic conference system is constructed in which the server 1 is a conference server and the client 2 is a participant terminal used by a participant. The conference server has a screen data transmission function using UDP multicast of the screen sharing server 1 as described in the third embodiment. The conference server sets the TTL so that multicast packets reach only the network in the conference room. In addition, the conference server manages the control right, and basically sends screen data to the participant terminals that do not have the control right by multicast UDP. On the other hand, when transmitting the screen data to the participant terminal having the control right, the screen data in the character area of the active window in the foreground is preferentially used by the TCP packet, and the participant terminal having the control right has the control right. Send for quick and accurate display. Data other than the character area is transmitted in UDP packets with a lower priority in the transmission order.

Further, for example, in the device configuration shown in FIG. 36, the server 1 is a conference server, the client 2 is a participant terminal used by a participant, and the relay device (proxy) 3 is capable of recording / reproducing screen data and multicasting. Configure a conference system as a conference proxy that can transfer screen data using UDP packets. The conference server sets the TTL so that multicast packets reach the directly connected participant terminals. The conference proxy sets the TTL so that multicast packets reach the participant terminals connected to the proxy.

As shown in FIG. 36, some of the participant terminals (2a, 2b, ..., 2i) exist in the conference room, are directly connected to the conference server, and receive the multicast UDP packet directly from the conference server. On the other hand, some of the other participant terminals (2j, ..., 2n) are in other conference rooms, are connected via the conference proxy, and receive the multicast UDP packet from the conference proxy.

By managing the communication in this way, the communication efficiency of the network is improved, and when used for a remote conference, the screen can be shared smoothly. In addition, participants who want to join the meeting from the middle can connect to the proxy, play the screen data of the shared screen from the beginning of the meeting, skip unnecessary parts as appropriate, or fast forward the meeting. If you want to follow the content and share the screen of the conference server in real time, you can join the conference by switching the connection to the conference server, viewing the synchronized shared screen, and performing remote control.

(Effect) As is clear from the above explanation, only communication by TCP packet, which is a data stream type communication method, in which it is necessary to return an ACK packet in response to the successful reception of the packet to the transmitted packet. It is fast because there is no need to reply ACK, but it is fast because it transmits screen data using UDP packet, which is a datagram type communication method that does not guarantee reliability or packet reception order. Communication between the screen sharing server and the screen sharing client that shares the screen becomes possible.

In addition, by transmitting screen data to multiple clients at once by UDP multicast or UDP broadcast packet, the screen data of the screen sharing server is transmitted to multiple clients at high speed, a network. Communication with high utilization efficiency becomes possible.

Further, by dynamically selecting the packet type manually or automatically, it is possible to improve the efficiency of the communication speed by communicating the screen data with the packet type suitable for the occasion. It is also possible to alleviate the sensory dissatisfaction caused by the time lag with the server, which the client user feels when sharing the screen data of the server by screen sharing. In addition, TCP packets are used by transmitting screen data using UDP packets to less important screen data and clients in a good network client environment that does not need to be reliably transmitted using TCP. It is possible for the user of the client who needs it, or by using a TCP packet, to relatively increase the transfer speed of the screen data that needs to be transmitted reliably.

In addition, when the screen area of the application is updated in advance for each application, the type of the screen data transmission packet is changed by the function of specifying whether to transmit the screen data as a TCP packet or a UDP packet. By selecting and sharing the screen, it is possible to make the shared screen data displayed on the client suitable for the characteristics of the application. In addition, the transfer speed of the screen data of the application registered to be transmitted by the TCP packet can be relatively increased. It is also possible to send screen data of applications such as video data playback applications that prioritize screen update speed over TCP packet reliability by UDP packets, and reduce screen sharing frame dropping on the client screen. It becomes.

Also, by sending the application window in the foreground as a TCP packet and the other rear window or desktop area as a UDP packet, important data at that time is often displayed in the foreground. It is possible to improve communication efficiency by reliably transmitting window data and transmitting less important data in other window areas or desktop areas by giving priority only to speed over certainty. Become. In addition, the transfer speed of the screen data of the window in the foreground registered to be transmitted by the TCP packet can be relatively increased.

Also, by sending the active application window in TCP packets and the other inactive windows or desktop areas in UDP packets, the data in the active window, which often displays important data at that time, is displayed. It is possible to improve the communication efficiency by transmitting less important data in the other inactive window area or desktop area with priority given to speed rather than certainty. In addition, the transfer speed of the screen data of the active window registered to be transmitted by the TCP packet can be relatively increased.

In addition, a function to specify whether to send screen data in TCP packets or UDP packets when the screen area of the application that displays the document is updated for each document displayed by the application on the screen sharing server. As a result, by dynamically selecting the type of the transmission packet of the screen data and sharing the screen, the shared screen data displayed on the client can be made suitable for the characteristics of the application. Further, depending on the importance of the document, it is possible to select whether to prioritize certainty or communication speed, and it is possible to improve communication efficiency. In addition, it is possible to relatively increase the transfer speed of the screen data of the application window that opens the document registered to be transmitted by the TCP packet.

In addition, the screen sharing server has a function to specify whether to send the screen data of a certain screen area by TCP packet or UDP packet in the desktop area, so that the area where you want to surely share the screen can use the screen data. Communication efficiency can be improved by transmitting the screen data in the screen area where the screen data that is transmitted as a TCP packet and gives priority to speed is displayed as a UDP packet. In addition, the transfer speed of screen data in the area registered for transmission in TCP packets can be relatively increased.

In addition, by having a means for switching whether the screen data is transmitted in a TCP packet or a UDP packet according to the user's instruction, the screen data currently shared by the user can be transmitted in a different packet type. By making it possible to switch by packet type when it is determined that communication is better, it is possible to provide a communication method capable of transmitting screen data suitable for the situation.

Also, if the number of clients connected to the screen sharing server is less than a certain number, screen data is transmitted by TCP packet, and if it is more than a certain number, screen data is transmitted by UDP packet. By having a function of automatically switching, it is possible to provide a communication method for screen sharing with high communication efficiency.

In addition, for clients using wireless communication means with many communication errors and poor communication quality, screen data is automatically transmitted in TCP packets to guarantee the reliability of the data, while ensuring the certainty of the data. It is possible to prioritize speed and communication efficiency by selecting to send screen data in UDP packets to clients that communicate with the server only by wired communication means with few communication errors and relatively high communication quality. Become. Also, by sending screen data using UDP packets to clients connected via a wired LAN, screens to clients connected via a wireless LAN registered to send in TCP packets. It is possible to increase the data transfer speed relatively.

In addition, select UDP for clients connected via LAN and TCP for clients connected from a network outside the LAN via WAN, and select a communication method that performs screen sharing suitable for the network environment. It is possible to perform communication that is automatically selected. In addition, by transmitting screen data using UDP packets to clients connected via LAN, screen data is transferred to clients connected via the WAN registered for transmission in TCP packets. It is possible to increase the speed relatively.

In addition, on the screen sharing server, if the remaining amount of screen update data to be transmitted or the transmission rate of screen data is higher than a certain value, it is automatically switched to transmit in UDP packets, and if it is less, it is automatically switched to reliably transmit in TCP packets. This makes it possible to provide a communication method in a screen sharing system that causes less stress on the client user.

In addition, if the communication speed between the server / client is slower than a certain value, it will be sent as a UDP packet, and if it is fast, it will be automatically switched to be surely sent as a TCP packet. It is possible to perform communication in a screen sharing system with less stress on the user. In addition, by transmitting screen data using UDP packets to a client with a high communication speed, the transfer speed of the screen data to a client with a slow communication speed registered to be transmitted as a TCP packet is relatively high. It will be possible to make it faster.

In addition, the screen sharing server measures the MTU of the network with the client, and if the MTU is smaller than a certain value, it sends it as a UDP packet, and if it is large, it automatically switches to send it reliably as a TCP packet. Depending on the network environment, it is possible to perform communication in a screen sharing system with less stress on the client user.

In addition, the screen sharing server measures the QoS with the client, and if the QoS is larger than a certain value, it sends it in a UDP packet, and if it is smaller, it automatically switches to send it in a TCP packet. Depending on the network environment, it is possible to perform a communication method in a screen sharing system that causes less stress on the client user. In addition, by transmitting screen data using UDP packets to a client with good communication quality, the transfer speed of screen data to a client with poor communication quality registered to be transmitted as a TCP packet is relatively high. It will be possible to make it faster.

In addition, by having a function to switch the type of packet used for transmitting screen data between TCP and UDP according to the priority of the client, the screen data can be reliably transmitted to the client with high priority by TCP packet, and others. Communication efficiency can be improved by transmitting screen data in UDP packets to clients with low priority. In addition, by transmitting screen data using UDP packets to low-priority clients, the transfer speed of screen data to high-priority clients registered for transmission in TCP packets is relatively high. It will be possible to make it faster.

In addition, by determining whether the remote operation from the client is in progress and switching the type of packet used for transmitting screen data between TCP and UDP, the client during remote operation can be notified by the TCP packet. Communication efficiency can be improved by reliably transmitting screen data and transmitting screen data in UDP packets to clients that are not performing remote operations. Also, by sending screen data using UDP packets to clients that are not performing remote operations, the transfer speed of screen data to clients that are performing remote operations instructed to send in TCP packets. Can be made relatively fast.

In addition, certainty is required by determining whether the area is where the OS is issuing font drawing commands or not, and switching the packet type used for transmitting screen data between TCP and UDP. By transmitting the screen data in the character area as a TCP packet and the screen data in the other area as a UDP packet, the character part is surely transmitted by the client, the communication speed is improved, and the character part is relatively relative. It is possible to improve the transfer speed of the screen data of. In addition, by transmitting screen data using UDP packets to the area where data other than characters is drawn, the transfer speed of screen data in the character area registered to be transmitted by TCP packets is relatively high. It becomes possible to do.

In addition, by switching the packet type used for transmitting screen data between TCP and UDP according to the CPU load of the screen sharing server, if the processing amount of the server is large, screen update data is generated by that processing. By transmitting the screen data in UDP packets in consideration of the slowness of the transmission process, it is possible to prevent a decrease in the screen data transmission speed when the load on the server is large.

In addition, a relay information device intervenes between the screen sharing server and the client, and the screen data received from the screen sharing server as a TCP packet is converted into a UDP packet and sent to the client, so that many clients can use the server. It is possible to reduce the load on the server that occurs when connecting directly to. Further, by having a function of transmitting by UDP packet, it is possible to improve the transfer speed of screen data.

In addition, when multiple clients connect to the relay information device by broadcast communication using UDP multicast or UDP broadcast for the screen data received by TCP packet from the screen sharing server, the screen data due to the increase in the number of clients It is possible to prevent a decrease in transfer speed.

Further, by recording the received screen sharing data, the state of the server screen can be browsed asynchronously. Further, by transmitting the recorded screen data by UDP packet, it is possible to improve the communication efficiency and realize high-speed screen reproduction that does not make the browsing client feel unnatural. Further, by broadcasting communication using multicast or broadcast, it is possible to prevent a decrease in screen playback speed due to an increase in the number of clients for a plurality of clients.

In addition, it is not necessary to reply to ACK as well as communication by TCP packet, which is a data stream type communication method, in which it is necessary to return an ACK packet that responds to the transmission packet that the packet has been received normally. Therefore, although it is high-speed, it is possible to perform high-speed screen sharing by transmitting screen data using UDP packets, which is a datagram-type communication method that does not guarantee reliability or packet reception order. It becomes possible to provide a conference system.

In addition, a relay device (proxy) intervenes between the screen sharing server and the client, converts the screen data received from the screen sharing server in TCP packets into a UDP packet, and sends it to the client, resulting in a large number of clients. It is possible to provide an electronic conferencing system that can reduce the load on the server that occurs when the user connects directly to the server.

<figref num="1">It is a schematic diagram which showed the network structure of the data communication system which is one Example of this invention.</figref><figref num="2">It is a figure which illustrated the processing flow of the method of performing screen sharing by transmitting only a UDP packet.</figref><figref num="3">It is a figure which showed the algorithm of the monitoring of the connection state in the screen sharing by UDP.</figref><figref num="4">It is the figure which exemplifies the processing flow of the method of using TCP / IP connection for connection of a server and a client, using UDP packet for transmission of screen data and input event, and communicating by unicast.</figref><figref num="5">It is a figure which exemplifies the processing flow of the system which uses TCP / IP connection for connection of a server and a client, uses UDP packet for transmission of screen data and input event, and performs broadcast communication by broadcast or multicast.</figref><figref num="6">It is a figure which illustrated the display screen of a server about the method of transmitting by either TCP or UDP for each application of a server.</figref><figref num="7">It is a figure which showed the algorithm about the method of transmitting by either TCP or UDP for each application of a server.</figref><figref num="8">It is a figure which illustrated the processing flow about the method of transmitting by either TCP or UDP for each application of a server.</figref><figref num="9">It is the figure which illustrated the display screen of the server about the method of switching and transmitting between TCP and UDP depending on whether or not the window of the server is located in the foreground.</figref><figref num="10">It is the figure which showed the algorithm about the method of switching and transmitting between TCP and UDP depending on whether or not the window of the server is located in the foreground.</figref><figref num="11">It is the figure which illustrated the display screen of the server about the method of switching between TCP and UDP depending on whether the window of the server is active or not.</figref><figref num="12">It is the figure which showed the algorithm about the method of switching between TCP and UDP, and transmitting depending on whether the window of the server is active or not.</figref><figref num="13">It is a figure which illustrated the display screen of the server about the method of switching between TCP and UDP depending on whether or not the window of the server is a document specified by the user.</figref><figref num="14">It is a figure which showed the algorithm about the method of switching between TCP and UDP depending on whether or not the window of the server is a document specified by the user.</figref><figref num="15">It is a figure which illustrated the screen display of the server about the method of switching transmission between TCP and UDP depending on whether or not the window of the server is a screen area specified by the user.</figref><figref num="16">It is a figure which showed the algorithm about the method of switching and transmitting between TCP and UDP depending on whether or not the window of the server is the screen area specified by the user.</figref><figref num="17">It is a figure which illustrated the screen which gives the switching instruction of a communication protocol.</figref><figref num="18">It is a figure which showed the flow of the switching operation of a communication protocol.</figref><figref num="19">It is a figure which illustrated the screen which gives the switching instruction of a communication protocol for each user.</figref><figref num="20">It is the figure which showed the algorithm about the method of switching between TCP and UDP depending on the number of connected clients.</figref><figref num="21">It is a figure which showed the algorithm about the method of switching transmission between TCP and UDP by the communication interface of a client.</figref><figref num="22">It is the figure which showed the algorithm about the method of switching transmission between TCP and UDP according to the connection path of a client.</figref><figref num="23">It is a figure which showed the algorithm about the method of switching between TCP and UDP depending on the remaining amount of encoded data to be transmitted.</figref><figref num="24">It is a figure which showed the algorithm about the method of switching between TCP and UDP depending on the transmission data rate.</figref><figref num="25">It is a figure which showed the algorithm about the method of switching transmission between TCP and UDP according to the communication speed between a server and a client.</figref><figref num="26">It is the figure which showed the algorithm about the method of switching and transmitting TCP and UDP by the MTU between the server and the client.</figref><figref num="27">It is a figure which showed the algorithm about the method of switching transmission between TCP and UDP by QoS between a server and a client.</figref><figref num="28">It is a figure which showed the algorithm about the method of switching between TCP and UDP according to the priority of a client and transmitting.</figref><figref num="29">It is a figure which illustrated the screen which sets the control right of a remote control.</figref><figref num="30">It is a figure which showed the algorithm about the setting of a remote operation flag.</figref><figref num="31">It is the figure which showed the algorithm about the method of switching transmission between TCP and UDP by turning on / off the remote operation flag.</figref><figref num="32">It is a figure which showed the algorithm about the method of switching transmission between TCP and UDP depending on whether the shared screen data is font data or image data.</figref><figref num="33">It is the figure which showed the algorithm about the method of switching and transmitting between TCP and UDP according to the load situation of the CPU.</figref><figref num="34">It is a schematic diagram which showed the network structure of the data communication system which is one Example of this invention (with a proxy, Part 1).</figref><figref num="35">It is a figure which illustrated the processing flow of the method of performing screen sharing through a proxy.</figref><figref num="36">It is a schematic diagram which showed the network structure of the data communication system which is one Example of this invention (with a proxy, part 2).</figref><figref num="37">It is the figure which showed the processing flow about the method which applied UDP multicast or UDP broadcast by the proxy.</figref><figref num="38">It is a figure which showed the processing flow about the reproduction function in a proxy.</figref>

Code description

1 Server (screen sharing server) 2 Client 3 Relay device (proxy) 10 IP network 11 TCP shared application window 12 UDP shared application window 13 Front window 14 Rear window 15 Active window 16 Inactive window 17 User-specified document window 18 Non-user Specified document window