Data synchronization method and system and servers
Abstract
The present invention is applicable to the field of data processing technology, and provides a data synchronization method, system and server. The method includes: a second server sends data synchronization request information to a first server; After requesting the information, send the data to be synchronized or the list of data to be synchronized to the second server; the second server receives the data to be synchronized sent by the first server, or obtains it from the first server according to the list of data to be synchronized sent by the first server Corresponding data to be synchronized; the first server synchronizes the write request of the first server to the second server when the amount of remaining data to be synchronized is less than the preset value. The present invention can effectively solve the problems of high cost, excessive data loss, or long service stop time in the prior art when data synchronization is performed.
Term
No projected expiry on record.
- Priority and filed
- Published
- Today
30 claims: 5 independent, 25 dependent
- 1一种数据同步方法,其特征在于,所述方法包括: 第二服务器向第一服务器发送数据同步请求信息; 所述第一服务器在接收到所述数据同步请求信息后,向所述第二服务器发送待同步的 数据或者待同步的数据列表; 所述第二服务器接收所述第一服务器发送的待同步数据,或者根据所述第一服务器发 送的待同步的数据列表从所述第一服务器获取相应的待同步数据; 所述第一服务器判断剩余的待同步数据的量是否小于预设值,在所述剩余的待同步数 据的量小于预设值时,将所述第一服务器的写请求同步到所述第二服务器,以实现第二服 务器与所述第一服务器数据的同步。
- 2如权利要求1所述的方法,其特征在于,所述第一服务器将所述第一服务器的写请 求同步到所述第二服务器包括: 所述第一服务器暂停写请求,并在所述剩余的待同步数据全部发送给所述第二服务器 后或者所述剩余的待同步数据全部被所述第二服务器获取后,接收写请求,同时将所述第 一服务器的写请求同步到所述第二服务器。
- 3如权利要求1所述的方法,其特征在于,所述第一服务器将所述第一服务器的写请 求同步到所述第二服务器包括: 所述第一服务器在发送所述剩余的待同步数据的同时或者所述剩余的待同步数据被 所述第二服务器获取的同时,将所述第一服务器的写请求同步到所述第二服务器。
- 4如权利要求1所述的方法,其特征在于,所述第一服务器在接收到所述数据同步请 求信息后还包括: 所述第一服务器将所述发送数据同步请求信息的第二服务器的信息存储至预设的信 息表; 所述第一服务器周期性的检测所述信息表,在所述请求信息表中存在多个第二服务器 发送的数据同步请求信息时,根据预设的选择条件从所述多个第二服务器中选择一个服 务器用于数据同步。
- 5如权利要求4所述的方法,其特征在于,所述预设的选择条件包括所述发送数据同 步请求信息的第二服务器中待同步数据最少和/或历史数据同步过程中出错次数最少的 第二服务器。
- 6如权利要求1所述的方法,其特征在于,所述方法还包括: 所述第一服务器设定有数据同步时间,在数据同步的时间大于所设定的时间时,结束 当前的数据同步过程。
- 7如权利要求1所述的方法,其特征在于,所述第一服务器和第二服务器中存储有用 于确定各服务器的状态信息,所述状态信息中包括各服务器存储的最新的数据序列号以及 各服务器的唯一标示符,所述数据序列号与所述唯一标示符存在映射关系,以使得各服务 器通过比较所述状态信息中各服务器的数据序列号大小确定自己的状态。 &如权利要求1-7任一项所述的方法,其特征在于,所述数据为元数据; 所述第一服务器向所述第二服务器发送待同步的数据包括: 所述第一服务器向所述第二服务器发送待同步的操作日志和/或待同步的合并后的 元数据。
- 89. 一种数据同步方法,其特征在于,所述方法包括: 第一服务器接收第二服务器发送的数据同步请求信息; 第一服务器向所述第二服务器发送待同步的数据或者待同步的数据列表,以使得所述 第二服务器接收所述第一服务器发送的待同步数据,或者根据所述第一服务器发送的待同 步的数据列表从所述第一服务器获取相应的待同步数据; 第一服务器判断剩余的待同步数据的量是否小于预设值,在所述剩余的待同步数据的 量小于预设值时,将所述第一服务器的写请求同步到所述第二服务器,以实现第二服务器 与所述第一服务器数据的同步。
- 910. 如权利要求9所述的方法,其特征在于,所述将所述第一服务器的写 请求同步到 所述第二服务器包括: 所述第一服务器暂停写请求,并在所述剩余的待同步数据全部发送给所述第二服务器 后或者所述剩余的待同步数据全部被所述第二服务器获取后,接收写请求,同时将所述第 一服务器的写请求同步到所述第二服务器。
- 1011. 如权利要求9所述的方法,其特征在于,所述将所述第一服务器的写请求同步到所 述第二服务器包括: 所述第一服务器在发送所述剩余的待同步数据的同时或者所述剩余的待同步数据被 所述第二服务器获取的同时,将所述第一服务器的写请求同步到所述第二服务器。
- 1112. 如权利要求9所述的方法,其特征在于,所述第一服务器在接收到所述数据同步请 求信息后还包括: 所述第一服务器将所述发送数据同步请求信息的第二服务器的信息存储至预设的信 息表; 所述第一服务器周期性的检测所述信息表,在所述请求信息表中存在多个第二服务器 发送的数据同步请求信息时,根据预设的选择条件从所述多个第一服务器中选择一个服务 器用于数据同步。
- 1213. 如权利要求12所述的方法,其特征在于,所述预设的选择条件包括所述发送数据 同步请求信息的第二服务器中待同步数据最少和/或历史数据同步过程中出错次数最少 的第二服务器。
- 1314. 如权利要求9所述的方法,其特征在于,所述方法还包括: 所述第一服务器设定数据同步时间,在数据同步的时间大于预设时间值时,结束当前 的数据同步过程。
- 1415. 如权利要求9-14任一项所述的方法,其特征在于,所述数据为元数据; 所述第一服务器向所述第二服务器发送待同步的数据包括: 所述第一服务器向所述第二服务器发送待同步的操作日志和/或待同步的合并后的 元数据。
- 1516. 一种数据同步方法,其特征在于,所述方法包括: 第二服务器向第一服务器发送数据同步请求信息,以使得所述第一服务器在接收到所 述数据同步请求信息后,向所述第二服务器发送待同步的数据或者待同步的数据列表; 第二服务器接收所述第一服务器发送的待同步数据,或者根据所述第一服务器发送的 待同步的数据列表从所述第一服务器获取相应的待同步数据,并在剩余的待同步数据的量 小于预设值时,接收所述第一服务器的写请求,以实现第二服务器与所述第一服务器数据 的同步。 17.如权利要求16所述的方法,其特征在于,所述数据为元数据。 1& 一种数据同步系统,其特征在于,所述系统包括: 第一服务器和第二服务器; 所述第二服务器,用于向所述第一服务器发送数据同步请求信息,接收所述第一服务 器发送的待同步数据,或者根据所述第一服务器发送的待同步的数据列表从所述第一服务 器获取相应的待同步数据,并在剩余的待同步数据小于预设值时,接收所述第一服务器的 写请求; 所述第一服务器,用于在接收到所述数据同步请求信息后,向所述第二服务器发送待 同步的数据或者待同步的数据列表,判断剩余的待同步数据的量是否小于预设值,在所述 剩余的待同步数据的量小于预设值时,将所述第一服务器的写请求同步到所述第二服务 器,以实现第二服务器与所述第一服务器数据的同步。
- 1619. 如权利要求18所述的系统,其特征在于,所述第一服务器具体用于,在剩余的待同 步数据的量小于预设值时,暂停写请求,并在剩余的待同步数据全部发送给所述第二服务 器后或者所述剩余的待同步数据全部被所述第二服务器获取后,接收写请求,同时将所述 第一服务器的写请求同步到所述第二服务器。
- 1720. 如权利要求18所述的系统,其特征在于,所述第一服务器具体用于,在剩余的待 同步数据的量小于预设值时,将所述剩余的待同步数据发送给所述第二服务器的同时或者 所述剩余的待同步数据被所述第二服务器获取的同时,将所述第一服务器的写请求同步到 所述第二服务器。
- 1821. 如权利要求18所述的系统,其特征在于,所述第一服务器还用于,将所述发送数据 同步请求信息的第一服务器的信息存储至预设的信息表,周期性的检测所述信息表,在所 述请求信息表中存在多个第二服务器发送的数据同步请求信息时,根据预设的选择条件从 所述多个第二服务器中选择一个服务器用于数据同步。
- 1922. 如权利要求21所述的系统,其特征在于,所述预设的选择条件包括所述发送数据 同步请求信息的第二服务器中待同步数据最少和/或历史数据同步过程中出错次数最少 的第二服务器。
- 2023. 如权利要求18所述的系统,其特征在于,所述第一服务器还设定有数据同步时间, 用于在数据同步的时间大于所设定的时间时,结束当前的数据同步过程。
- 2124. 如权利要求18所述的系统,其特征在于,所述系统还包括: 管理后台,用于获取各服务器的状态信息,并将所获取的状态信息分别发送给各服务 器,以使得各服务器根据所述状态信息确定各服务器的状态后发起数据同步请求,所述服 务器状态包括第一服务器和第二服务器,所述状态信息中包括各服务器存储的最新的数据 序列号以及各服务器的唯一标示符,所述数据序列号与所述唯一标示符存在映射关系,以 使得各服务器通过比较所述状态信息中各服务器的数据序列号大小确定自己的状态。
- 2225. 如权利要求24所述的系统,其特征在于,所述服务器的状态还包括第三服务器; 所述管理后台还用于,在所述第二服务器完成数据同步后,更新所述第二服务器的状 态信息,即将所述第二服务器的状态更新为所述第三服务器。
- 2326. 如权利要求18-25任一项所述的系统,其特征在于,所述数据为元数 据; 所述第一服务器用于向所述第二服务器发送待同步的操作日志和/或待同步的合并 后的元数据。
- 2427. 一种服务器,其特征在于,所述服务器包括: 信息接收单元,用于接收第二服务器发送的数据同步请求信息; 第一信息发送单元,用于向所述第二服务器发送待同步的数据或者待同步的数据列 表,以使得所述第二服务器接收所述信息发送单元发送的待同步数据,或者根据所述信息 发送单元发送的待同步的数据列表从该服务器获取相应的待同步数据; 同步单元,用于判断剩余的待同步数据的量是否小于预设值,在所述剩余的待同步数 据的量小于预设值时,将该服务器的写请求同步到所述第二服务器,以实现第二服务器与 该服务器数据的同步。 2&如权利要求27所述的服务器,其特征在于,所述同步单元具体用于,暂停写请求, 并在剩余的待同步数据全部发送给所述第二服务器后或者所述剩余的待同步数据全部被 所述第二服务器获取后,接收写请求,同时将该服务器的写请求同步到所述第二服务器。
- 2529. 如权利要求27所述的服务器,其特征在于,所述同步单元具体用于,在发送所述剩 余的待同步数据的同时或者所述剩余的待同步数据被所述第二服务器获取的同时,将该服 务器的写请求同步到所述第二服务器。
- 2630. 如权利要求27所述的服务器,其特征在于,所述服务器还包括: 存储单元,用于将所述发送数据同步请求信息的第二服务器的信息存储至预设的信息 表; 选择单元,用于周期性的检测所述存储单元存储的信息表,在所述请求信息表中存在 多个第二服务器发送的数据同步请求信息时,根据预设的选择条件从所述多个第二服务器 中选择一个服务器用于数据同步。
- 2731. 如权利要求30所述的服务器,其特征在于,所述预设的选择条件包括所述发送数 据同步请求信息的第二服务器中待同步数据最少和/或历史数据同步过程中出错次数最 少的第二服务器。
- 2832. 如权利要求27所述的服务器,其特征在于,所述服务器还包括: 控制单元,用于设定数据同步时间,在数据同步的时间大于预设时间值时,结束当前的 数据同步过程。
- 2933. 如权利要求27-32任一项所述的服务器,其特征在于,所述数据为元数据; 所述同步单元具体用于,向所述第二服务器发送待同步的操作日志和/或待同步的合 并后的元数据。
- 3034. 一种服务器,其特征在于,所述服务器包括: 第二信息发送单元,用于向第一服务器发送数据同步请求信息,以使得所述第一服务 器在接收到所述数据同步请求信息后,向该服务器发送待同步的数据或者待同步的数据列 表; 信息接收单元,用于接收所述第一服务器发送的待同步数据,或者根据所述第一服务 器发送的待同步的数据列表从所述第一服务器获取相应的待同步数据,并在剩余的待同步 数据的量小于预设值时,接收所述第一服务器的写请求,以实现该服务器与所述第一服务 器数据的同步。 35.如权利要求34所述的服务器,其特征在于,所述数据为元数据。
Independent claims30
142 paragraphs, as filed
A data synchronization method, system and server technical field
[0001] The present invention belongs to the field of data processing technology, and in particular relates to a data synchronization method, system, or server.
Background technique
[0002] With the continuous development of information technology, people have higher and higher requirements for data stability and security. In many fields, the main and backup servers are adopted to ensure the stability and security of data. When the main server fails, the backup server is switched to the main server in time to ensure the safe and stable operation of the information system.
[0003] To realize the security mechanism of the primary and backup servers, it is necessary to ensure the synchronization of data in the primary and backup servers. The prior art mainly adopts the following methods to realize the synchronization of data in the active and standby servers:
[0004] Manner 1: Based on shared persistent storage. In this way, because the data is shared, synchronization can be achieved directly without additional synchronization process. However, this method needs to rely on third-party expensive storage equipment, and the cost is relatively high.
[0005] Manner 2: Periodically synchronize the data of the previous time period. This method sets the synchronization period in advance. After the set synchronization period time is reached, the standby server obtains the data in the period of time on the primary server, but because the standby server obtains the data in the primary server at the same time, the primary server may still be obtaining the data. The new data causes the backup server to fail to obtain the latest data, and most of the latest data is lost when the primary and backup servers are switched.
[0006] Manner 3: Pause the service of the active and standby servers, and start the service after backing up all the data in the active server to the standby server. Although this method can ensure that the data of the primary and backup servers are completely synchronized, it requires a long service time suspension, such as more than twenty minutes, which affects the update of data.
[0007] In summary, in the prior art, when performing data synchronization, there are problems such as high cost, loss of data, or long time to stop service.
Summary of the invention
[0008] The purpose of the embodiments of the present invention is to provide a data synchronization method to solve the problems of high cost, more data loss, or long service outage time when performing data synchronization in the prior art.
[0009] The embodiment of the present invention is implemented as follows, a data synchronization method, the method includes:
[0010] The second server sends data synchronization request information to the first server;
[0011] After receiving the data synchronization request information, the first server sends the data to be synchronized or the list of data to be synchronized to the second server;
[0012] The second server receives the data to be synchronized sent by the first server, or obtains the corresponding data to be synchronized from the first server according to the list of data to be synchronized sent by the first server;
[0013] The first server determines whether the amount of remaining data to be synchronized is less than a preset value, and when the amount of remaining data to be synchronized is less than the preset value, synchronizes the write request of the first server to all The second server is used to realize data synchronization between the second server and the first server.
[0014] A data synchronization method, the method includes:
[0015] The first server receives the data synchronization request information sent by the second server;
[0016] The first server sends the data to be synchronized or the list of data to be synchronized to the second server, so that
Receiving, by the second server, data to be synchronized sent by the first server, or obtaining corresponding data to be synchronized from the first server according to a list of data to be synchronized sent by the first server;
[0017] The first server determines whether the amount of remaining data to be synchronized is less than a preset value, and when the amount of remaining data to be synchronized is less than the preset value, synchronizes the write request of the first server to the first Two servers to realize data synchronization between the second server and the first server.
[0018] A data synchronization method, the method includes:
[0019] The second server sends data synchronization request information to the first server, so that after receiving the data synchronization request information, the first server sends the data to be synchronized or the data to be synchronized to the second server List
[0020] The second server receives the to-be-synchronized data sent by the first server, or obtains the corresponding to-be-synchronized data from the first server according to the list of to-be-synchronized data sent by the first server, and selects the remaining data to be synchronized. When the amount of synchronization data is less than the preset value, receiving a write request from the first server to achieve data synchronization between the second server and the first server.
[0021] Another object of the embodiments of the present invention is to provide a data synchronization system, the system includes a first server and a second server;
[0022] The second server is configured to send data synchronization request information to the first server, receive the to-be-synchronized data sent by the first server, or read from the list of to-be-synchronized data sent by the first server The first server obtains corresponding data to be synchronized, and when the amount of remaining data to be synchronized is less than a preset value, receives a write request from the first server;
[0023] The first server is configured to send data to be synchronized or a list of data to be synchronized to the second server after receiving the data synchronization request information, and determine whether the amount of remaining data to be synchronized is less than The preset value, when the remaining amount of data to be synchronized is less than the preset value, synchronize the write request of the first server to the second server, so as to realize the data exchange between the first server and the first server Synchronize.
[0024] A server, the server includes:
[0025] The information receiving unit is configured to receive data synchronization request information sent by the second server;
[0026] The first information sending unit is configured to send data to be synchronized or a list of data to be synchronized to the second server, so that the second server receives the data to be synchronized sent by the information sending unit, or according to The data list to be synchronized sent by the information sending unit obtains the corresponding data to be synchronized from the server;
[0027] The synchronization unit is configured to determine whether the amount of remaining data to be synchronized is less than a preset value, and when the amount of remaining data to be synchronized is less than the preset value, synchronize the write request of the server to the second Server to achieve data synchronization between the second server and the server.
[0028] A server, the server includes:
[0029] The second information sending unit is configured to send data synchronization request information to the first server, so that after receiving the data synchronization request information, the first server sends the data to be synchronized or the data to be synchronized to the server List of data;
[0030] The information receiving unit is configured to receive the data to be synchronized sent by the first server, or obtain the corresponding data to be synchronized from the first server according to the list of data to be synchronized sent by the first server, and When the amount of remaining data to be synchronized is less than the preset value, receiving a write request from the first server to realize data synchronization between the server and the first server.
[0031] Compared with the prior art, the embodiment of the present invention has the following beneficial effects: the data synchronization process does not need to rely on a third-party expensive storage device, which saves costs; the critical value of the remaining data to be synchronized is preset. When the remaining data to be synchronized is less than the preset critical value, synchronize the write request of the first server to the second server, that is, the second server and the first server receive the write request at the same time, and obtain the latest data at the same time, which can be effective Avoid the problem that the backup server cannot obtain the latest data in the existing periodic synchronization, which leads to the problem of excessive loss of the latest data. And because the preset critical value is small, data synchronization can be completed with a small pause time (tens of milliseconds) or no pause at all, so that the timely update of data will not be affected. In addition, any one of passive reception and active acquisition can be used to acquire the data to be synchronized, which has strong operability and practicability.
Description of the drawings
[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the embodiments or the accompanying drawings that need to be used in the description of the prior art. Obviously, the accompanying drawings in the following description are merely For some of the embodiments of the present invention, for those of ordinary skill in the art, other drawings may be obtained based on these drawings without creative labor.
[0033] FIG. 1 is a network architecture diagram of a data synchronization system provided by a first embodiment of the present invention;
[0034] FIG. 2 is an interaction flowchart of a data synchronization method provided by a second embodiment of the present invention;
[0035] FIG. 3 is an interaction flowchart of a data synchronization method provided by a third embodiment of the present invention;
[0036] FIG. 4 is an interaction flowchart of a data synchronization method provided by a fourth embodiment of the present invention;
[0037] FIG. 5 is a structural diagram of a server provided by a fifth embodiment of the present invention;
[0038] FIG. 6 is a structural diagram of a server provided by a sixth embodiment of the present invention.
Detailed ways
[0039] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the following further describes the present invention in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, but not to limit the present invention.
[0040] In order to illustrate the technical solutions of the present invention, specific embodiments are described below.
[0041] Embodiment One:
[0042] FIG. 1 shows the network architecture of the data synchronization system provided by the first embodiment of the present invention. The data synchronization method provided by the embodiment of the present invention can be applied to the data synchronization system. For ease of description, only The parts related to this embodiment are described.
[0043] The data synchronization system includes a first server 2 and a second server 3. Among them, the form of the first and second servers may be a single server or a server side composed of several functional servers. The first server 2 is a server that stores the latest and most complete data, and the second server 3 is a server whose stored data is lagging behind the first server 2 or is a newly added server that does not store data.
[0044] Preferably, the data synchronization system further includes a management background 1.
[0045] After the server is started, the management background 1 obtains the status information of each server, and sends the obtained status information to each server. The status information includes, but is not limited to, the latest data serial number stored by each server and the unique identifier of each server, and there is a mapping relationship between the data serial number and the unique identifier. Each operation of the server on the data will be recorded by a monotonically increasing persistent data serial number, the data serial number
It can indicate how old and new the data stored in the server is.
[0046] After receiving the status information, each server determines its own status by comparing the size of the data sequence number of each server in the status information, that is, whether it belongs to the first server 2 (the server with the largest data sequence number) or The second server 3 (the server less than the largest data serial number). After determining its own state, the second server 3 initiates a data synchronization request to the corresponding first server 2 according to the mapping relationship between the data serial number and the unique identifier, and the first server 2 is receiving After the data synchronization request is received, the data to be synchronized or the list of data to be synchronized are sent to the second server 3, and the second server 3 receives the data to be synchronized sent by the first server 2, or according to the The data list to be synchronized sent by the first server 2 obtains the corresponding data to be synchronized from the first server 2, and the first server 2 determines the remaining data to be synchronized (the remaining data to be synchronized refers to the Whether the amount of remaining data in the data to be synchronized is less than the preset value (determined by the data serial number, for example, the serial number of the current data to be synchronized is 7, and the serial number of the latest data is 10, indicating the remaining data The amount is 10-7=3), and when the amount of remaining data to be synchronized is less than the preset value, the write request of the first server 2 is synchronized to the second server 3 to complete the data synchronization. Wherein, the write request includes operation requests such as deletion, modification, and addition of data.
[0047] It should be noted that each server is not limited to determining its own identity through the size of the data serial number, and may also determine its own identity through other methods such as data storage time information, which will not be described here.
[0048] Preferably, the first server 2 may suspend the write request when the amount of remaining data to be synchronized is less than the preset value, and after all the remaining data to be synchronized is sent to the second server 3, or After all the remaining data to be synchronized are acquired by the second server 3, it restarts to receive write requests, and at the same time, the write request of the first server 2 is synchronized to the second server 3. or,
[0049] While directly sending the remaining data to be synchronized to the second server 3 or while the remaining data to be synchronized is acquired by the second server 3, the data of the first server 2 The write request is synchronized to the second server 3.
[0050] Preferably, after receiving the data synchronization request information, the first server 2 further includes:
[0051] The first server 2 stores the information of the second server 3 that sends the data synchronization request information in a preset information table, and the information includes but is not limited to the unique identifier of the server and the corresponding data serial number .
[0052] The first server 2 periodically detects the information table, and when there are multiple data synchronization request information sent by the second server 3 in the request information table, select the information from the multiple One of the second servers 3 is selected for data synchronization.
[0053] Wherein, the preset selection conditions include, but are not limited to, the second server 3 that has the least data to be synchronized and/or the least number of errors in the historical data synchronization process among the second servers 3 that send data synchronization request information. It should be noted that the first server may determine the data to be synchronized according to the latest data serial number in the server. The first server may determine the number of errors in the synchronization process of the second server's historical data according to the pre-stored historical data synchronization record.
[0054] Preferably, the first server 2 is also set with a data synchronization time, and when the data synchronization time is greater than a preset time value, the current data synchronization process ends. Preferably, the server identity further includes a third server 4, which is a backup server of the first server 2, and the third server 4 also stores the latest and most complete data.
[0055] The management background 1 is also used to update the status information of the second server 3 after the second server 3 completes data synchronization, that is, to update the status of the second server 3 to the third Server 4.
[0056] Preferably, the data in this embodiment includes but is not limited to metadata. When the data is metadata,
The first server 2 is configured to send the operation log to be synchronized and/or the merged metadata to be synchronized to the second server 3. Among them, the purpose of merging metadata is to reduce the memory usage of the server and accelerate the speed of sending, receiving and loading data into the memory.
[0057] The second embodiment:
[0058] FIG. 2 shows the interaction flow of the data synchronization method provided by the second embodiment of the present invention, which is applied to multiple interconnected servers, and the server stores status information for determining the status of each server, and the server identity Including the first server and the second server, the process is detailed as follows:
[0059] 1. The second server sends data synchronization request information to the first server.
[0060] In this embodiment, the state information includes, but is not limited to, the latest data serial number stored by each server and the unique identifier of each server. There is a mapping relationship between the data serial number and the unique identifier, and the server Determine your own status by comparing the size of the data serial number, and automatically send data synchronization to the corresponding first server according to the mapping relationship between the data serial number and the unique identifier after determining its status as the second server Request information. The first server is a server that stores the latest and most complete data, and the second server is a server whose stored data lags behind the first server, or is a newly added server that does not store data.
[0061] 2. After receiving the data synchronization request information, the first server sends the data to be synchronized or the list of data to be synchronized to the second server.
[0062] Preferably, in order to ensure the efficiency of data synchronization, the first server further includes after receiving the data synchronization request information:
[0063] The first server stores the information of the second server that sends the data synchronization request information in a preset information table;
[0064] The first server periodically detects the information table, and when there are multiple data synchronization request information sent by the second server in the request information table, select the information from the multiple first servers according to a preset selection condition. Choose one of the two servers for data synchronization.
[0065] Wherein, the preset selection conditions include, but are not limited to, the second server with the least data to be synchronized and/or the second server with the least number of errors during the synchronization of historical data among the second servers that send the data synchronization request information.
[0066] In this embodiment, the first server may determine the data to be synchronized required by the second server according to the latest data serial number of the second server in the status information, and combine the data to be synchronized or the data to be synchronized. The synchronized data list is sent to the second server.
[0067] 3. The second server receives the data to be synchronized sent by the first server, or obtains the corresponding data to be synchronized from the first server according to the list of data to be synchronized sent by the first server.
[0068] In this embodiment, the data to be synchronized can be acquired in either way of passive reception and active acquisition, which enhances operability and practicability.
[0069] 4. Determine whether the amount of remaining data to be synchronized is less than the preset value, if yes, perform step 5, otherwise continue to send the data to be synchronized to the second server or receive the second server to obtain the data to be synchronized.
[0070] 5. When the remaining amount of data to be synchronized is less than the preset value, synchronize the write request of the first server to the second server.
[0071] In order to reduce the server suspension time and ensure that the first server and the second server obtain the latest data at the same time, and avoid the loss of the latest data when the first server and the second server are switched, this embodiment presets the remaining to be synchronized The critical value of the amount of data, when the amount of remaining data to be synchronized is less than the critical value, select any one of the following
Synchronous alignment:
[0072] Manner 1: The first server suspends the write request, and restarts receiving and writing after all the remaining data to be synchronized are sent to the second server or after all the remaining data to be synchronized are acquired by the second server Request, while synchronizing the write request of the first server to the second server. Since the set critical value is small, this method only needs to pause for several tens of milliseconds to achieve data synchronization between the first server and the second server.
[0073] In order not to affect the timely update of data and avoid excessively long pause time, this embodiment may pre-set a pause time threshold. When the time for the first server to pause write requests exceeds the pause time threshold, it is explained that the data synchronization process There may be an error in the process (such as network interruption, etc.). End the current data synchronization process, so that the first server can select other servers that send data synchronization requests to re-synchronize data, or the second server can retransmit to other servers that store the latest and most complete data. Send a data synchronization request.
[0074] Manner 2: While sending the remaining data to be synchronized or the remaining data to be synchronized is acquired by the second server, the first server synchronizes the write request of the first server to all Mentioned second server. This method can complete data synchronization without the need for the server to suspend the service.
[0075] Preferably, in order to avoid that the server does not respond for a long time due to an error during the data synchronization process, this embodiment further includes:
[0076] The first server presets the entire data synchronization time, and when the data synchronization time is greater than the preset time value, the current data synchronization process ends. The data synchronization time includes the time of the entire data synchronization process and/or the time of the synchronization alignment process, and the data synchronization time can be modified according to actual conditions (for example, the performance of the server, the amount of data to be synchronized).
[0077] It should be noted that when the first server adopts the second method to perform synchronization alignment, the second server first temporarily suspends the received new write request data while receiving or acquiring the remaining data to be synchronized. Stored in the memory (the data sequence number of the new write request of the second server is the same as the data sequence number of the new write request of the first server), until the remaining data to be synchronized are all received or acquired, the remaining data to be synchronized The synchronization data and the received new write request data are transferred to the disk together in the order of the data sequence number.
[0078] In order to reduce the loss caused by transmission errors, this embodiment adopts a breakpoint resumable transmission method to transmit data, and records the progress of the transmission.
[0079] Preferably, the server status described in this embodiment further includes a third server. After the second server completes data synchronization, the status information of the second server is updated, that is, the status of the second server is updated to the third server. Wherein, the third server is a backup server of the first server, and the latest and most complete data is also stored in the third server.
[0080] Preferably, the data described in this embodiment includes but is not limited to metadata. When the data is metadata, the first server is used to send the operation log to be synchronized to the second server ( MutationLog file) and/or merged metadata (Checkpoint file) to be synchronized. Among them, the purpose of merging metadata is to reduce the memory usage of the server and accelerate the speed of sending, receiving, and loading data into the memory.
[0081] The embodiment of the present invention does not need to rely on a third-party expensive storage device in the data synchronization process, which saves costs and reduces reliance; each server stores state information for determining the server, so that each server can determine according to the state information After one's own state, automatically initiate a data synchronization request to obtain the latest data. The entire process does not require manual participation, ensuring the efficiency and timeliness of data synchronization; in addition, there are pre-set thresholds for the remaining data to be synchronized. When the data to be synchronized is less than the preset critical value, perform synchronization alignment, and synchronize the write request of the first server
To the second server, that is, the second server and the first server receive the write request at the same time and obtain the latest data at the same time, thereby effectively avoiding the problem that the existing periodic synchronization existing backup server cannot obtain the latest data. In addition, a variety of methods are used to achieve synchronization and alignment, and data synchronization can be completed with a small pause time (tens of milliseconds) or no pause at all, so that the timely update of data will not be affected. Obtain the data to be synchronized by either passive reception or active acquisition, which enhances operability and practicability.
[0082] Embodiment Three:
[0083] FIG. 3 shows the implementation process of the data synchronization method provided by the third embodiment of the present invention. The execution subject of the process in this embodiment is the first server in the data synchronization system described in FIG. 1, and the implementation process is described in detail as follows :
[0084] In step S301, the first server receives data synchronization request information sent by the second server;
[0085] In step S302, the first server sends the data to be synchronized or the list of data to be synchronized to the second server, so that the second server receives the data to be synchronized sent by the first server, or according to The data list to be synchronized sent by the first server obtains the corresponding data to be synchronized from the first server;
[0086] In step S303, the first server determines whether the amount of remaining data to be synchronized is less than the preset value, if yes, execute step S304, otherwise, return to execute S302.
[0087] In step S304, the write request of the first server is synchronized to the second server, so as to achieve data synchronization between the second server and the first server.
[0088] Preferably, in this embodiment, when the remaining data to be synchronized is less than the critical value, any one of the following synchronization alignment modes can be selected:
[0089] Manner 1: The first server suspends the write request, and restarts after all the remaining data to be synchronized are sent to the second server or after the remaining data to be synchronized are all acquired by the second server Start receiving the write request, and at the same time synchronize the write request of the first server to the second server. Since the set critical value is small, this method only needs to pause for tens of milliseconds to achieve complete synchronization of the data of the first server and the second server. The complete synchronization means that the first server and the first server store the same data. .
[0090] Manner 2: While sending the remaining data to be synchronized or the remaining data to be synchronized is acquired by the second server, the first server synchronizes the write request of the first server to all Mentioned second server. This method can complete data synchronization without the need for the server to suspend the service.
[0091] Preferably, in order to ensure the efficiency of data synchronization, this embodiment further includes after receiving the data synchronization request information:
[0092] The first server stores the information of the second server 3 that sends the data synchronization request information in a preset information table;
[0093] The first server periodically detects the information table, and when there are multiple data synchronization request information sent by the second server in the request information table, select the information from the multiple first servers according to a preset selection condition. Choose one of the two servers for data synchronization.
[0094] Wherein, the preset selection conditions include, but are not limited to, the second server with the least data to be synchronized and/or the second server with the least number of errors during the historical data synchronization among the second servers that send the data synchronization request information.
[0095] Preferably, in order to prevent the server from being unresponsive for a long time due to an error during the data synchronization process, this embodiment further includes the following steps:
[0096] The first server presets the entire data synchronization time, and when the data synchronization time is greater than the preset time value, the current data synchronization process ends.
[0097] Preferably, the data in this embodiment includes but is not limited to metadata. When the data is metadata, the first server is used to send the operation log to be synchronized to the second server ( MutationLog file) and/or merged metadata (Checkpoint file) to be synchronized. Among them, the purpose of merging metadata is to reduce the memory usage of the server and accelerate the speed of sending, receiving, and loading data into the memory.
[0098] For the specific implementation process of each step in this embodiment, reference may be made to the description related to the first server in the above second embodiment, which will not be repeated here.
[0099] Embodiment Four:
[0100] FIG. 4 shows the implementation process of the data synchronization method provided by the fourth embodiment of the present invention. The execution subject of the process in this embodiment is the second server in the data synchronization system described in FIG. 1, and the implementation process is described in detail as follows :
[0101] In step S401, the second server sends data synchronization request information to the first server, so that the first server sends the data to be synchronized to the second server after receiving the data synchronization request information. Or the list of data to be synchronized;
[0102] In step S402, the second server receives the data to be synchronized sent by the first server, or obtains the corresponding data to be synchronized from the first server according to the list of data to be synchronized sent by the first server, And when the amount of remaining data to be synchronized is less than a preset value, receiving a write request from the first server, so as to realize data synchronization between the second server and the first server.
[0103] In this embodiment, the data may be metadata.
[0104] For the specific implementation process of each step in this embodiment, reference may be made to the description related to the second server in the foregoing second embodiment, which will not be repeated here.
[0105] Example Stone:
[0106] FIG. 5 shows the composition structure of the server in the fifth embodiment of the present invention. For ease of description, only the parts related to the embodiment of the present invention are shown.
[010] The server 2 can be applied to the data synchronization system as shown in FIG. 1. The server 2 is a server storing the latest and most complete data. The server 2 includes an information receiving unit 21, a first information sending unit 22, and The synchronization unit 23, the specific functions of each unit are as follows:
[0108] The information receiving unit 21 is configured to receive data synchronization request information sent by the second server;
[0109] The first information sending unit 22 is configured to send data to be synchronized or a list of data to be synchronized to the second server, so that the second server receives the data to be synchronized sent by the information sending unit, or Obtaining corresponding data to be synchronized from the server according to the list of data to be synchronized sent by the information sending unit;
[0110] The synchronization unit 23 is configured to determine whether the amount of remaining data to be synchronized is less than a preset value, and when the amount of remaining data to be synchronized is less than the preset value, synchronize the write request of the server to the first Two servers to realize data synchronization between the second server and the server.
[0111] Preferably, the synchronization unit 23 is specifically configured to suspend the write request when the amount of remaining data to be synchronized is less than a preset value, and after all the remaining data to be synchronized are sent to the second server Or, after all the remaining data to be synchronized are acquired by the second server, it restarts to receive write requests, and at the same time synchronizes the write request of the server to the second server. or,
[0112] While sending the remaining data to be synchronized or while the remaining data to be synchronized is acquired by the second server, synchronize the write request of the server to the second server.
[0113] Preferably, the server 2 further includes:
[0114] The storage unit 24 is configured to store the information of the second server that sends the data synchronization request information in a preset information table;
[0115] The selection unit 25 is configured to periodically detect the information table stored in the storage unit 24, and when there are multiple data synchronization request information sent by the second server in the request information table, according to a preset selection condition Select one server from the plurality of second servers for data synchronization.
[0116] Wherein, the preset selection conditions include, but are not limited to, the second server with the least data to be synchronized and/or the second server with the least number of errors during the synchronization of historical data among the second servers that send the data synchronization request information.
[0117] Preferably, the server 2 further includes:
[0118] The control unit 26 is configured to set the data synchronization time, and when the data synchronization time is greater than the preset time value, end the current data synchronization process.
[0119] The data in this embodiment includes but is not limited to metadata. When the data is metadata, the synchronization unit 23 is specifically configured to send the operation log to be synchronized and/or the merged metadata to be synchronized to the second server.
[0120] The server provided in this embodiment can be used in the aforementioned corresponding data synchronization method. For details, please refer to the relevant descriptions of the second and third embodiments of the aforementioned data synchronization method, which will not be repeated here.
[0121] Embodiment 6:
[0122] FIG. 6 shows the composition structure of the server of the sixth embodiment of the present invention. For ease of description, only the parts related to the embodiment of the present invention are shown.
[0123] The server 3 can be applied to the data synchronization system as shown in FIG. 1. The server 3 is a server with lagging data, or a newly added server that does not store data. The server 3 includes an information sending unit 31 and information. The receiving unit 32, the specific functions of each unit are as follows:
[0124] The second information sending unit 31 is configured to send data synchronization request information to the first server, so that after receiving the data synchronization request information, the first server sends the data to be synchronized or the data to be synchronized to the server. Synchronized data list;
[0125] The information receiving unit 32 is configured to receive the data to be synchronized sent by the first server, or obtain corresponding data to be synchronized from the first server according to the list of data to be synchronized sent by the first server, and When the amount of remaining data to be synchronized is less than a preset value, receiving a write request from the first server, so as to realize data synchronization between the server and the first server.
[0126] In this embodiment, the data includes but is not limited to metadata.
[0127] The server provided in this embodiment can use the aforementioned corresponding data synchronization method. For details, refer to the relevant descriptions of the second and fourth embodiments of the aforementioned data synchronization method, which will not be repeated here.
[0128] Those skilled in the art can understand that the units included in the servers described in the fifth and sixth embodiments above are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; The specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the present invention.
[0129] In summary, the embodiment of the present invention does not need to rely on third-party expensive storage devices in the data synchronization process, which saves costs and reduces reliance; each server stores state information that determines the server, so that each server can be based on The state information automatically initiates a data synchronization request after determining its own state, so that the lagging data server can automatically and timely obtain the latest data. The entire process does not require manual participation, ensuring the efficiency and timeliness of data synchronization; in addition, advance A critical value for the remaining data to be synchronized is set, and the remaining data to be synchronized is less than the preset threshold
Synchronize and align the write request of the first server to the second server at the same time, that is, the second server and the first server receive the write request at the same time, and obtain the latest data at the same time, which can effectively avoid the existing backup server that periodically synchronizes Unable to get the latest data. In addition, a variety of methods are used to achieve synchronization and alignment, and data synchronization can be completed with a small pause time (tens of milliseconds) or no pause at all, which will not affect the timely update of data. In addition, it is possible to acquire the data to be synchronized by either passive reception or active acquisition, which is highly operability and practical.
[0130] A person of ordinary skill in the art can also understand that all or part of the steps in the above-mentioned embodiment method can be implemented by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium. In the above, the storage medium includes ROM/RAM, magnetic disk, optical disk, etc.
[0131] The above content is a further detailed description of the present invention in conjunction with specific preferred embodiments, and it cannot be considered that the specific implementation of the present invention is limited to these descriptions. For those of ordinary skill in the technical field to which the present invention belongs, if several equivalent substitutions or obvious modifications are made without departing from the concept of the present invention, and the performance or use is the same, they should all be regarded as belonging to the claims submitted by the present invention. The scope of patent protection determined by the book.
Every citation, both ways
| Document | Relation | Office | Cited during |
|---|---|---|---|
| CN106101167A | Cited by | China | Search report |
| CN107729515A | Cited by | China | Search report |
| CN106850255A | Cited by | China | Search report |
| CN104484136A | Cited by | China | Search report |
| CN109462640A | Cited by | China | Search report |
| CN103973462A | Cited by | China | Search report |
| WO2017101731A1 | Cited by | World Intellectual Property Organization (WIPO) | International search |
| CN109428918A | Cited by | China | Search report |
| CN107861958A | Cited by | China | Search report |
| CN106899648A | Cited by | China | Search report |
| CN107924419A | Cited by | China | Search report |
| US10693965B2 | Cited by | United States of America | Applicant |
| US10798168B2 | Cited by | United States of America | Applicant |
| CN101388759A | Cites | China | Search report |
| CN101719165A | Cites | China | Search report |
| CN101751415A | Cites | China | Search report |
| CN102024040A | Cites | China | Search report |
| US2010299455A1 | Cites | United States of America | Search report |
2 priority claims, no other members on record
Priority claims2
| Document | Office | Kind | Date |
|---|---|---|---|
| 201210264187 | China | A | |
| CN20121264187 | – | – | – |
2 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Rejection of invention patent application after publicationRJ01 | RJ01 | |
| PublicationC06 | C06 |
Numbers
- Publication
- 103580891
- Publication, DOCDB
- 103580891
- Publication, EPODOC
- CN103580891
- Application
- 102641877
- Application, DOCDB
- 201210264187
- Application, EPODOC
- CN20121264187
Titles2
- Chinese
- 一种数据同步的方法、系统及服务器
- English
- Method, system and server for data synchronization
Classification
- IPC, 1
- H04L12 24