Data acquisition and data playback method and system based on message middleware

A method and system for data acquisition and data playback based on message middleware Technical field

[0001] The application relates to the technical field of data processing, in particular to a method and system for data collection and data playback based on message middleware.

Background technique

[0002] In the data transmission process of the distributed system, if the data volume is large and the portability is poor, once an inaccurate result occurs, it will seriously affect the use. Message middleware includes message queue and publish/subscribe two working modes. The message queue mode transmits messages through a message queue, which solves the problem of loose coupling of time and space between the two parties in communication, but consumers cannot reproduce the retransmission process of specific queue or topic messages, including data re-expression display and event processing display reduction.

Contents of the invention

[0003] Based on this, it is necessary to provide a method and system for data collection and data playback based on message middleware for the above-mentioned technical problems.

A kind of method based on the data acquisition of message middleware and data playback, described method comprises: The data that business system produces is sent to message server, and according to business classification, corresponding data information is saved in message server.

[0005] The data information in the message server is set up index information; the index information includes an index with the order of writing as a key.

[0006] Responding to the data playback request of the playback client, according to the data playback request, the index information and the data information stored in the message server, a playback file is generated; the playback file is sent to the playback client.

[0007] In one of the embodiments, indexing information for data information in the message server includes: After the data information is stored in the message server, the message server starts a timing task in the background, A sequential index is established for the data information on the message server, with data information writing time and batch as keywords.

[0008] The sequence index is sorted according to the size of the key value, and each index element saves a queue with the same key value to obtain the corresponding index information of the data information.

[0009] The index information is saved in the index file.

In one of the embodiments, in response to the data playback request of the playback client, according to the data information stored in the data playback request, the index information and the message server, generate a playback file, including: Response to the playback client Data playback request.

When the playback file does not exist, if the historical version is inquired about, then the message server calls the historical query interface, and inquires about the corresponding historical record of the working node; if the historical version is not inquired about, then the directly designated version number is -1, and the latest version is used Create playback files.

[0012] The playback client selects the version number that generates the playback file, and sends a request that generates the corresponding version playback file.

[0013] When the message server receives a playback file request, create a playback index file generation thread.

In playback file generation process, inquire about playback file generation progress, wait for playback file generation success.

After the playback file is generated successfully, if there is no corresponding playback file locally, then pull the whole playback file to the playback client local; if there is a playback file in the local area, then obtain the last offset of the local playback file, and use the last offset to the working node Pull the playback file and update the local playback file.

[0016] After the local playback file is pulled or updated, the playback client loads the playback file, and calls the playback file analysis interface to resolve the playback file.

A kind of system based on the data acquisition of message middleware and data playback, described system comprises: Data acquisition module, for the data that business system produces is sent to message server, and according to business classification corresponding data information is preserved to the message server.

[0018] The data processing module sets up index information for the data information in the message server; the index information includes an index with the order of writing as a key.

Playback data generation module, for responding to the data playback request of playback client; When playback file does not exist, if query history version, then message server calls historical query interface, query the corresponding historical record of work node; If do not query history version, directly specify the version number as -1, and use the latest version to generate the playback file; the playback client selects the version number to generate the playback file, and sends a request to generate the playback file of the corresponding version; when the message server receives the generated playback file When requesting, create a playback index file generation thread; during the playback file generation process, query the playback file generation progress, and wait for the playback file to be successfully generated; when the playback file is successfully generated, if there is no corresponding playback file locally, pull the entire playback file to Play back the client locally; if the playback file already exists locally, get the last offset of the local playback file, use the last offset to pull the playback file to the working node, and update the local playback file; when the local playback file is pulled or updated, the playback client Loading the playback file, calling the playback file parsing interface to parse the playback file; a data module, configured to send the playback file to the playback client.

In one of the embodiments, the data processing module is also used for when the data information is stored in the message server, the message server starts a timing task in the background, and the data on the message server Information establishes a sequence index with the data information writing time and batch as the key; sort the sequence index according to the size of the key value, each index element saves a queue with the same key value, and obtains the data information corresponding to The index information; save the index information to the index file.

In one of the embodiments, the playback data generation module is also used to respond to the data playback request of the playback client; when the playback file does not exist, if the query history version, then the message server calls the history query interface, query work node Corresponding historical records; if you do not query the historical version, directly specify the version number as -1, and use the latest version to generate the playback file; the playback client selects the version number to generate the playback file, and sends a request to generate the playback file of the corresponding version ;When the message server receives a request to generate a playback file, create a playback index file generation thread; during the playback file generation process, query the playback file generation progress, and wait for the playback file to be successfully generated; when the playback file is successfully generated, if there is no corresponding playback locally file, pull the entire playback file to the local playback client; if the playback file already exists locally, get the last offset of the local playback file, use the last offset to pull the playback file to the working node, and update the local playback file; when the local playback file is pulled After fetching or updating is completed, the playback client loads the playback file and calls the playback file parsing interface to parse the playback file.

Above-mentioned a kind of method and system based on the data collection of message middleware and data playback, the method is based on the data collection and data playback method of each link design such as message middleware in data collection, processing, storage, to improve system of

Stability and usability. By designing and implementing the data acquisition and data playback system, the data stored in the distributed message middleware can be processed and stored in a unified manner. Then the data is processed through the derivation of the data model; after the data information is processed, the data is transmitted to the user. This method can help the user to complete the analysis. During the playback process, all the clients that originally participated in the transmission are played back online, and the cooperation process is displayed again, ensuring the correctness of the business processing logic during the playback process. In addition, the playback theme and start and end time are also controllable factors, which are used to flexibly control the progress of playback, so as to analyze the key points of business processing and focus on the execution of key businesses.

Description of drawings

Fig. 1 is the schematic flow chart of the method for data acquisition and data playback based on message middleware in an embodiment; Fig. 2 is the interactive process of playback client and message server in playback file generation process in another embodiment; Fig. 3 is a schematic diagram of playback index in another embodiment.

Detailed ways

[0024] In order to make the application's purpose, technical solutions and advantages clearer, the application will be described in further detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application.

In one embodiment, as shown in Figure 1, a kind of method based on the data collection of message middleware and data playback is provided, and the method comprises the following steps: Step 100: the data that business system produces is sent to message server, and save the corresponding data information in the message server according to the business classification.

[0026] Specifically, the data generated in the business system can be sent to the message system server for storage, and the subsequent business system can pull messages from the message server for business processing. The business system sends the corresponding data information to the message server for classification and storage according to the business classification, so that the business data needed in the business system can be stored in the message server for subsequent use and analysis of the data.

[0027] Timing guarantees the logical correctness of member state changes and business data updates in the playback data collection process, and avoids the confusion of each member's data processing in the playback process.

[0028] Step 102: establish index information for the data information in the message server; the index information includes an index whose key is written order.

The processing of replay data is the basic requirement of replay, replay data must carry out persistent storage, restore at the most accurate point in time when replay, the situation that occurs partial lead or lag all can cause negative influence to the reduction of former process ;In addition, the playback process must be complete. A complete reproduction scheme is required for the dynamic operation of the event execution process. The playback data must also be completely presented during the playback process. Problems such as data loss or repeated data processing will lead to logical occurrences mistake.

[0030] A high-performance index can be established for the collected data to support fast retrieval and positioning of the data. The system allows users to quickly query the messages they want from the message server. A large number of messages will be stored on the message server. If the search and matching of messages are directly performed on the disk, the search efficiency will be seriously reduced. At this time, a corresponding index must be established for the messages stored in the message server.

Step 104: respond to the data playback request of playback client; When playback file does not exist, if query historical version, then message server calls historical query interface, query the corresponding historical record of working node; If not query historical version, then directly Specify the version number as -1, use the latest version to generate the playback file; the playback client selects the version number to generate the playback file, and sends a request to generate the playback file of the corresponding version; when the message server receives the request to generate the playback file, create Playback index file generation thread; during the playback file generation process, query the playback file generation progress and wait for the playback file to be successfully generated; when the playback file is successfully generated, if there is no corresponding playback file locally, pull the entire playback file to the playback client locally ;If the playback file already exists locally, get the last offset of the local playback file, use the last offset to pull the playback file to the worker node, and update the local playback file; when the local playback file is pulled or updated, the playback client loads the playback file, Call the playback file parsing interface to parse the playback file; The interaction process between the playback client and the message server during the generation of the playback file is shown in Figure 2.

Step 106: the playback file is sent to the playback client.

In the above-mentioned data collection based on message middleware and the method for data playback, the method is based on the data collection and data playback method of each link design of message middleware in data collection, processing, storage, etc., to improve the stability of the system and availability. By designing and implementing the data acquisition and data playback system, the data stored in the distributed message middleware can be processed and stored in a unified manner. Then the data is processed through the derivation of the data model; after the data information is processed, the data is transmitted to the user. This method can help the user to complete the analysis. During the playback process, all the clients that originally participated in the transmission are played back online, and the cooperation process is displayed again, ensuring the correctness of the business processing logic during the playback process. In addition, the playback theme and start and end time are also controllable factors, which are used to flexibly control the progress of playback, so as to analyze the key points of business processing and focus on the execution of key businesses.

In one of the embodiments, step 102 includes: after the data information is stored in the message server, the message server starts a timing task in the background, and the data information on the message server is set up to write time and batch of data information on the message server It is the sequential index of the keyword; the sequential index is sorted according to the size of the keyword value, each index element saves a queue with the same keyword value, and the index information corresponding to the data information is obtained; the index information is saved to the index file.

[0035] Specifically, when the client pulls the message, it first matches the corresponding message from the index, and then loads the message content from the disk according to the index and returns it to the client. When the message is saved in the storage server, the message server will start a timing task in the background to periodically save the index information corresponding to the message on the disk in a separate file, and the timing task will create an index for each message. Index information. When a user wants to query a message, he first searches the index file for the index information corresponding to the index according to the index information, and then loads the corresponding message content from the disk according to the index information and returns it to the client.

[0036] The playback index includes an index whose key is written order.

[0037] Sequential indexes with different dimensions as keys. Here, the time and batches in the system are regarded as variables, and a sequential index with these variables as keys is created.

[0038] The keys of the index are different, sorted according to the size, and each index element saves the queue with the same key.

[0039] The playback index schematic diagram is shown in Figure 3.

[0040] It should be understood that although the steps in the flow chart of FIG. 1 are shown in sequence according to the arrows, these steps are not necessarily executed in sequence according to the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in FIG. 1 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution of these sub-steps or stages The order is not necessarily sequential,

Rather, it may be performed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps. In one embodiment, provide a kind of system based on the data acquisition of message middleware and data playback, this system comprises: Data acquisition module, for the data that business system produces is sent to message server, and according to business The classification saves the corresponding data information to the message server.

[0042] The data processing module sets up index information for the data information in the message server; the index information includes an index with the write order as the key.

Playback data generation module, for responding to the data playback request of playback client; When playback file does not exist, if query history version, then message server calls historical query interface, query the corresponding historical record of work node; If do not query history version, directly specify the version number as -1, and use the latest version to generate the playback file; the playback client selects the version number to generate the playback file, and sends a request to generate the playback file of the corresponding version; when the message server receives the generated playback file When requesting, create a playback index file generation thread; during the playback file generation process, query the playback file generation progress, and wait for the playback file to be successfully generated; when the playback file is successfully generated, if there is no corresponding playback file locally, pull the entire playback file to Play back the client locally; if the playback file already exists locally, get the last offset of the local playback file, use the last offset to pull the playback file to the working node, and update the local playback file; when the local playback file is pulled or updated, the playback client Load the playback file, and call the playback file parsing interface to parse the playback file. [0044] The data module is used to send the playback file to the playback client.

In one of the embodiments, the data processing module is also used for when the data information is stored in the message server, the message server starts a timing task in the background, and the data information on the message server is set up to write the time with the data information , The batch is the sequential index of the keyword; the sequential index is sorted according to the size of the keyword value, and each index element saves a queue with the same keyword value to obtain the index information corresponding to the data information; save the index information to the index file .

Each technical characterictic of above-mentioned embodiment can be combined arbitrarily, for making description succinct, all possible combinations of each technical characterictic in above-mentioned embodiment are not all described, yet, as long as there is no contradiction in the combination of these technical characterictics , should be considered within the scope of this record.

[0047] The above-described embodiments have only expressed several embodiments of the present application, and its description is more specific and detailed, but it cannot therefore be interpreted as limiting the scope of the patent for invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be determined by the appended claims.