CN113127217A - Data distribution method, device, equipment and storage medium - Google Patents

Abstract

The invention discloses a data publishing method, device, equipment and storage medium. The method includes: adding the acquired data identifiers of a plurality of data to be published into a first thread pool, so that the data acquisition end can send or receive a message queue MQ; according to the data identifiers of each data to be published, determine each Distributed cache lock for data; generate a first message queue MQ according to the data identifier, and the first MQ includes a data identifier; send the first MQ to the data publishing end, so that the data publishing end can use the multiple data in the first MQ according to the data publishing end Identify multiple data to be published; add multiple data to be published to the second thread pool, and send a second MQ to the data acquisition end to publish the data to be published; the second MQ includes data identification, which can ensure that multiple clients High availability and resource saving.

Description

Data distribution method, apparatus, device and storage medium

technical field

The invention belongs to the technical field of business support, and in particular relates to a data publishing method, device, equipment and storage medium.

Background technique

The method of data publishing is that the data publishing system publishes timing data serially, which is inefficient. And if there is only one machine, a machine failure cannot guarantee high availability. Then, the data acquisition end can choose the cluster system, and use the database row lock to select one of the multiple clients to acquire the lock to acquire the data, which ensures the high availability of the acquired data. However, using database row locks to select a database row lock from multiple clients to obtain data, only one client can publish data, and other clients are in an idle state, resulting in a waste of resources.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a data publishing method, apparatus, device, and storage medium, which can ensure high availability of multiple clients and save resources.

In a first aspect, a data publishing method is provided, the method is applied to a data acquisition end, and the method includes:

adding the acquired data identifiers of the data to be published into the first thread pool, so as to be used by the data acquiring end to send or receive a message queue (Message Queuing, MQ);

Determine the distributed cache lock of each to-be-published data according to the data identifier of each to-be-published data;

Generate a first message queue MQ according to the data identifier, and the first MQ includes the data identifier;

Send the first MQ to the data publishing end, so that the data publishing end can obtain a plurality of data to be published according to the multiple data identifiers in the first MQ; add the plurality of data to be published to the second thread pool, and send the data to the data The acquiring end sends the second MQ to publish the data to be published; the second MQ includes the data identifier.

In a possible implementation, the method further includes:

determine the amount of data to be published;

When the quantity of data to be published is greater than the preset quantity, obtain the preset quantity of data to be published; or,

When the quantity of data to be published is not greater than the preset quantity, the data to be published is acquired.

In a possible implementation, the method further includes:

Determine the time threshold for distributed cache locks;

When the usage time of the distributed cache lock exceeds the time threshold, the distributed cache lock is released.

In a second aspect, a data publishing method is provided, the method is applied to a data publishing end, and the method includes:

Receive the first message queue MQ sent by the data acquisition end; the first MQ carries the data identifier of each data to be published;

Obtain the data to be published corresponding to the data identifiers according to the multiple data identifiers;

Add to-be-published data to the second thread pool for the data publisher to send or receive MQ;

According to each data identifier, the to-be-published data corresponding to each data identifier is released.

In a possible implementation manner, according to each data identifier, the to-be-published data corresponding to each data identifier is published, including:

According to each data identifier, determine the release page of the to-be-released data corresponding to each data identifier;

When the page area occupied by the data of the to-be-published data corresponds to the publishing page, the to-be-published data is published.

In a possible implementation manner, the method further includes: generating a second MQ according to the multiple data identifiers, and sending the second MQ to the data acquisition end, so that the data acquisition end determines a plurality of to-be-published data releases.

In a possible implementation, the method further includes:

Delete the historical data corresponding to the data identifier on the data publisher.

In a possible implementation, the method further includes:

Generate a third MQ according to the data identification, and the third MQ includes the data identification;

The third MQ is sent to the terminal, so that the terminal obtains the data to be published from the database according to the data identifier in the third MQ, and displays the data to be published.

In a possible implementation, the method further includes:

When the release of the data to be released is abnormal, delete the historical data corresponding to the data identifier at the data release end;

Generate a fourth MQ according to multiple data identifiers;

A fourth MQ is sent to the terminal for the terminal to determine that the data to be published is abnormally published.

In a possible implementation, the method further includes:

When the to-be-published data is published abnormally, a fourth MQ is generated according to multiple data identifiers;

The to-be-published data is republished according to the fourth MQ.

In a possible implementation, the method further includes:

When the number of times of republishing the data to be published exceeds the preset number of times, the state of the data to be published in the database is updated to be the data to be published with abnormal publishing.

In a third aspect, a data publishing device is provided, the device is applied to a data acquisition end, and the device includes:

A joining module is used to add the acquired data identifiers of the plurality of data to be published into the first thread pool;

A determination module, used for determining the distributed cache lock of each data to be released according to the data identifier of each data to be released;

a generating module for generating a first message queue MQ according to the data identifier;

The sending module is used to send the first MQ to the data publishing end, so that the data publishing end can obtain a plurality of data to be published according to the plurality of data identifiers in the first MQ; add the plurality of data to be published to the second thread pool , and send the second MQ to the data acquisition end to publish the data to be published.

In a fourth aspect, a data publishing device is provided, the device is applied to a data publishing terminal, and the device includes:

a receiving module, configured to receive the first message queue MQ sent by the data acquisition end; the first MQ carries the data identifier of each data to be published;

an acquisition module, configured to acquire the data to be published corresponding to the data identifiers according to the multiple data identifiers;

A joining module is used to add the data to be published into the second thread pool;

a sending module, configured to generate a second MQ according to multiple data identifiers, and send the second MQ to the data acquisition terminal, so that the data acquisition terminal can determine a plurality of data to be published for publication;

The publishing module is configured to publish the data to be published corresponding to each data identification according to each data identification.

In a fifth aspect, an electronic device is provided, the device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the first aspect or any possible implementation manner of the first aspect is implemented method.

In a sixth aspect, a computer storage medium is provided, where computer program instructions are stored thereon, and when the computer program instructions are executed by a processor, the method in the first aspect or any possible implementation manner of the first aspect is implemented.

Based on the provided data publishing method, device, device and storage medium, the acquired data identifiers of the data to be published are added to the first thread pool, so that the data acquisition end can send or receive the message queue MQ; Publish the data identifier of the data, and determine the distributed cache lock of each data to be published; generate a first message queue MQ according to the data identifier, and the first MQ includes the data identifier; send the first MQ to the data publisher for data The publishing end obtains a plurality of data to be published according to the plurality of data identifiers in the first MQ; adds the plurality of data to be published into the second thread pool, and sends the second MQ to the data obtaining end to publish the data to be published; the second MQ includes data identification, which can ensure high availability of multiple clients and save resources.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings that need to be used in the embodiments of the present invention. For those of ordinary skill in the art, without creative work, the Additional drawings can be obtained from these drawings.

1 is a schematic flowchart of a task publishing method provided by an embodiment of the present invention;

2 is a schematic flowchart of another task publishing method provided by an embodiment of the present invention;

3 is a schematic flowchart of a data publishing method provided by an embodiment of the present invention;

4 is a schematic flowchart of another data publishing method provided by an embodiment of the present invention;

5 is a schematic flowchart of another data publishing method provided by an embodiment of the present invention;

6 is a schematic structural diagram of a data publishing apparatus provided by an embodiment of the present invention;

7 is a schematic structural diagram of another data distribution apparatus provided by an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only configured to explain the present invention, and are not configured to limit the present invention. It will be apparent to those skilled in the art that the present invention may be practiced without some of these specific details. The following description of the embodiments is only intended to provide a better understanding of the present invention by illustrating examples of the invention.

It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element defined by the phrase "comprising" does not preclude the presence of additional identical elements in the process, method, article or device that includes the element.

Currently, there are two ways to release tasks. The first data release method is shown in Figure 1. The system queries all timing tasks, and then traverses the tasks and puts them into the thread pool to execute the tasks serially. The second data publishing method is shown in Figure 2. The timed task acquirer queries all timed tasks, traverses all tasks and puts them in the thread pool; sends MQ to the timed task execution end. The scheduled task execution end receives the MQ, then queries the tasks to be published by channel, executes the operation of publishing the task, and returns.

However, the above two methods still have the following problems: only one system executes timed tasks serially, which is inefficient. If there is only one machine, high availability cannot be guaranteed in the event of a machine failure. The client selects the cluster system to acquire tasks, which ensures the high availability of the acquired tasks. However, the database row lock is used to select one of multiple clients to acquire the lock to acquire the task, and other clients are in an idle state, resulting in a waste of resources.

Therefore, the embodiments of the present invention provide a data publishing method, apparatus, device, and storage medium, which can ensure high availability of multiple clients and save resources.

In order to facilitate understanding of the embodiments of the present invention, the data publishing method in the embodiments of the present invention is first described in detail.

FIG. 3 is a schematic flowchart of a data publishing method provided by an embodiment of the present invention.

As shown in FIG. 3 , the data publishing method provided by the embodiment of the present invention, applied to the data acquisition end, includes:

S301: Add the acquired data identifiers of the data to be published into the first thread pool, so that the data acquiring end sends or receives the message queue MQ.

The data to be published is stored in the database, and each data to be published corresponds to a data identifier. For example, if the data to be released is product data to be released by the merchant, then the identifier of the data to be released is the merchant account.

After acquiring the data identifiers of multiple data to be published, the data to be published is added to the thread pool. The thread pool increases the execution efficiency of multi-threading. In the embodiment of the present invention, not only the execution efficiency of multiple threads is ensured, but also the quick return of the execution result of the MQ is ensured, so as to prevent the MQ from retrying because the result does not return for a long time. The thread pool in the embodiment of the present invention may be implemented based on the base class of thread pool executor (Thread Pool Executor). The core thread is sufficient to execute the task through the core thread. If the core thread is executing the task, the task is placed in the Linked Blocking Queue to wait for execution.

In addition, in order to avoid the waste of memory resources caused by the backlog of data in the data acquisition end, before S301, the following steps may also be included: determining the quantity of data to be published;

When the quantity of data to be published is greater than the preset quantity, obtain the preset quantity of data to be published; or,

When the quantity of data to be published is not greater than the preset quantity, the data to be published is acquired.

Wherein, before acquiring the identifier of the data to be published, it is first necessary to determine the quantity of the data to be published. For example, a maximum of 1500 pieces of data to be published can be published each time. The amount of data to be published needs to be determined first. When the number of data to be published exceeds 1500, only the data identifiers of the 1500 data to be published are acquired. When the data of the data to be published does not reach 1500, the data identifiers of all the data to be published can be obtained.

S302: Determine the distributed cache lock of each data to be published according to the data identifier of each data to be published.

The distributed cache lock of the to-be-published data is obtained through the data identifier of each to-be-published data in the thread pool. If a data identifier acquires the distributed cache lock of a data acquirer, the data acquirer competes with other data acquirers, and other data acquirers cannot acquire the distributed cache lock of the data identifier. Obtaining the distributed cache lock through the data identifier can ensure the high availability of the data acquisition end, and can simultaneously acquire the data identifier of the data to be published.

Among them, in order to prevent deadlock caused by restarting or directly hanging up on the data acquisition end during the execution of the data publishing task, a time threshold is set for the distributed cache lock.

Specifically, determining the time threshold of the distributed cache lock;

When the usage time of the distributed cache lock exceeds the time threshold, the distributed cache lock is released.

When the time that a data identifier uses the distributed cache lock exceeds the time threshold of the distributed cache lock, the distributed cache lock will be released. After the release, other data publishing tasks can acquire the distributed cache lock, thereby ensuring the execution of other data publishing tasks. Get tasks in batches and asynchronously send them to the data publisher cluster through MQ. The data publisher cluster can quickly accept tasks.

S303: Generate a first message queue MQ according to the data identifier, where the first MQ includes the data identifier.

After the distributed cache lock is acquired, a first MQ is generated according to the data identifier, wherein the first MQ carries a plurality of data identifiers, and each data identifier corresponds to one piece of data to be published.

S304: Send the first MQ to the data publishing end, so that the data publishing end obtains a plurality of data to be published according to the plurality of data identifiers in the first MQ; add the plurality of data to be published into the second thread pool, and A second MQ is sent to the data acquisition end to publish the data to be published; the second MQ includes a data identifier.

Send the first MQ to the data publisher. Sending the first MQ to the data publishing end ensures that the data obtaining end can continue to obtain the data publishing task, and also ensures that the data publishing end does not repeatedly execute the same data publishing task.

After the data publishing end receives the first MQ, according to the data identification in the first MQ, the data identification corresponds to the data to be published. After acquiring the data to be published, a second MQ will be generated according to the data identifier, and the second MQ will be sent to the data acquiring end. After the data acquisition end receives the second MQ, the data acquisition end can determine that the data publishing end has obtained the data to be published, and can only perform the data publishing task. Distributed cache locks obtained through data identification are lighter than distributed locks such as databases, and multiple data acquisition ends of the cluster work simultaneously, which will not cause waste of resources and ensure high availability.

In the data publishing method provided by the embodiment of the present invention, the acquired data identifiers of the data to be published are added to the first thread pool, so that the data acquisition end can send or receive the message queue MQ; according to the data of each data to be published identifier, to determine the distributed cache lock of each data to be published; generate a first message queue MQ according to the data identifier, and the first MQ includes the data identifier; send the first MQ to the data publisher for the data publisher to use according to the first MQ Multiple data identifiers in one MQ obtain multiple data to be published; add multiple data to be published to the second thread pool, send the second MQ to the data acquisition end, and publish the data to be published; the second MQ includes data identifiers , which can ensure the high availability of multiple clients and save resources.

FIG. 4 is a schematic flowchart of another data publishing method provided by an embodiment of the present invention.

As shown in FIG. 4 , a data publishing method provided by an embodiment of the present invention, applied to a data publishing end, includes:

S401: Receive a first message queue MQ sent by a data acquisition end; the first MQ carries a data identifier of each data to be published.

After receiving the first MQ, the first MQ is parsed to obtain the data identifier of the data to be published. The first MQ type includes multiple data identifiers. Each data identifier corresponds to a piece of data to be published.

S402: Acquire data to be published corresponding to the data identifiers according to the multiple data identifiers.

The data to be published is stored in the database, and after the data identification is obtained by parsing, the data to be published is obtained from the database according to the data identification.

Here, there may be multiple release channels for the data to be released. For example, taking the data to be released as commodity data as an example, the release channel of the data can be a small program, or a certain e-commerce station, etc. When obtaining the data to be published, it is necessary to obtain the data to be published by channels according to the data identifier.

S403: Add the to-be-published data to the second thread pool for the data publishing end to send or receive MQ.

After the acquired data to be published is put into the second thread pool for generating the second MQ, the second MQ can be immediately returned to the data acquisition end.

Specifically, the second MQ is generated according to the multiple data identifiers, and the second MQ is sent to the data acquisition end, so that the data acquisition end determines to publish a plurality of data to be published.

A second MQ is generated according to the acquired data identifier of the data to be published, and the second MQ is sent to the data acquiring end. The data acquisition end can continue to obtain the data identifier of the data to be published, and perform the next data publishing task without causing a backlog of MQ. The data publishing task of the data to be published can be executed asynchronously without increasing the time.

S404: Publish the to-be-published data corresponding to each data identifier according to each data identifier.

After the data to be published is acquired, the data identifier of each data to be published is traversed, and the data to be published is published.

Specifically, according to each data identifier, determine the release page of the to-be-published data corresponding to each data identifier;

When the page area occupied by the data of the to-be-published data corresponds to the publishing page, the to-be-published data is published.

Each data to be published corresponds to a publishing page. For example, if the data is identified as a merchant account, and the data to be released is commodity data such as commodity prices to be released at a certain time set by the online merchant, then each online merchant will have a set data display page. The release page corresponding to the data to be released is the same as the data display page. Then, by traversing the data identifiers, the publishing page of each data to be published can be determined, and then the page area occupied by the published data is corresponding to the publishing page. If the page area occupied by the data to be published corresponds to the published page, the data to be published is published.

If the page area occupied by the to-be-published data does not correspond to the published page, the to-be-published data is published abnormally. When the to-be-published data is published abnormally, delete the historical data corresponding to the data identifier at the data publishing end; generate a fourth MQ according to the multiple data identifiers; wherein, the fourth MQ includes multiple data identifiers. Send the fourth MQ to the terminal. After receiving the fourth MQ, the terminal can determine that the data to be published is abnormally published, and does not need to obtain the data to be published from the database for display.

After the release of the data to be released is abnormal, the data release end may also retry the data release based on the data identifier. A fourth MQ is generated according to the data identifier of the data to be published that is abnormally published. The publication of the to-be-published data is retried based on the fourth MQ. If the number of retries to publish the to-be-published data exceeds the preset number of times, the status in the database of the to-be-published data that is abnormally published is updated to the published abnormality.

After the data to be published is successfully published, it is also necessary to delete the historical data corresponding to the data identifier on the data publishing side. Avoid resource occupancy.

In addition, after S404, the following steps may also be included:

Generate a third MQ according to the data identification, and the third MQ includes the data identification;

The third MQ is sent to the terminal, so that the terminal obtains the data to be published from the database according to the data identifier in the third MQ, and displays the data to be published.

After the data to be published is successfully published, a third MQ will be generated according to the data identifier, and the third MQ will be sent to the terminal. Wherein, the third MQ includes a data identifier. After receiving the third MQ, the terminal determines the data to be published that has been published according to the data identifier. The terminal acquires the data to be published from the database according to the data identifier, and displays the data to be published.

A data publishing method provided by an embodiment of the present invention includes receiving a first message queue MQ sent by a data acquisition end; the first MQ carries a data identifier of each data to be published; and the corresponding data identifiers are acquired according to multiple data identifiers. Data to be published; the data to be published is added to the second thread pool for the data publisher to send or receive MQ; according to each data ID, the data to be published corresponding to each data ID is published, which can ensure the High availability and resource saving.

Taking the product promotion and the task release time in the early morning, the merchant of the store decoration system may set thousands of timed release tasks, and all the timed tasks need to be completed accurately and quickly. As shown in FIG. 5, the embodiment of the present invention provides another A schematic flowchart of a data publishing method.

As shown in FIG. 5 , the data publishing method provided by the embodiment of the present invention includes:

S501: Timed task triggering, for example, triggering a task release every 30s.

Multiple clients work at the same time to query all scheduled tasks at the current moment.

S502: The client determines whether the number of tasks exceeds 1500. If yes, execute S503, if not, execute S504.

S503: The client obtains 1500 tasks each time.

S504: The client obtains all tasks.

Determine whether the task is greater than the threshold, and if it exceeds the threshold, divide the task into several batches.

S505: The client traversal task is added to the thread pool.

The client adds all tasks to the thread pool, which is convenient for sending MQ messages later.

S506: Obtain a distributed lock with an ID (IDentity Document, ID) as a key (key).

In the thread pool of the client, all tasks will compete for the distributed cache lock through the merchant ID. If a merchant ID obtains the distributed cache lock of a client, other clients will form a competition and can only obtain the distributed cache lock of other merchants. Cache locks, so that multiple clients can not only ensure high availability, but also work to acquire tasks at the same time.

The client obtains the distributed cache lock through the merchant ID, and sets a timeout to prevent the machine from restarting or directly hanging up during the execution of the task, causing deadlock. After this time, other tasks will acquire the lock and then execute other tasks.

S507: The client sends an MQ message to the server.

After the client obtains the distributed lock of the merchant ID, it constructs the parameters required by MQ, and then sends the MQ information to the server, which can obtain tasks concurrently and ensure non-repetitive execution.

S508: The server receives the MQ.

S509: The publishing task is put into the thread pool MQ and returned to the client.

The server obtains the message delivered by the MQ message, and puts the task into the thread pool after obtaining the task through the topic mechanism, so that the MQ message can be returned immediately without causing a backlog of MQ messages. The publishing task will be executed asynchronously and will not cause an increase in time.

S510: Check and update the database before the server publishes it.

S511: The server updates the cache and publishes the MQ. If the data release is abnormal, execute S518; if the data release is successful, execute S513 and S515.

S513: The server publishes snapshot storage.

The server executes the publishing process, updates the publishing status, deletes the cache of the current regular publishing store, sends MQ to the downstream system, and does some work of publishing snapshots. All of these operations are in one transaction, and if an exception occurs, the transaction is rolled back.

S514: The server-side publishing ends.

S515: The application (Application, App) side receives the MQ.

S516: Update the data of the home page interface of the App.

S517: The user checks the latest data.

S518: The server updates the cache and sends an MQ exception.

S519: The server is added to the queue.

S520: The server resends the MQ to the server.

The compensation mechanism of the server will use the retry mechanism of MQ to retry the publishing operation three times. If it still fails, it will update the status of the task to be an unsuccessful scheduled publishing task.

S521: All published exceptions occurred.

S522: The database status is not updated.

S523: Re-acquire the scheduled release task next time.

The next time the client queries the scheduled task release, the task will continue to be queried, but if the task has not been successfully executed for more than five minutes, the task will not be executed, and log analysis will be performed to prevent other scheduled tasks from being affected by retries. .

In the embodiment of the present invention, when the number of distributed cache locks in the cluster is more than 10, the efficiency will be increased by about 50% compared with other distributed locks, because each machine is querying and executing tasks, and the other distributed locks are one of the cluster. The machine is executing. The MQ mechanism and the thread pool technology will increase the efficiency by about 40% when the MQ retry reaches three or more times. The MQ retry is equivalent to increasing the number of tasks published regularly. The entire set of embodiments of the present invention can be used on both the C-end user and the B-end merchant side, thereby improving the overall publishing efficiency.

The technical means adopted by the embodiments of the present invention are described in detail below:

The embodiment of the present invention is realized by the client using thread pool + distributed cache lock + MQ message notification mechanism + exception compensation mechanism, and the server side is realized by using MQ message notification mechanism + thread pool technology + transaction management + exception compensation mechanism. The entire embodiment of the present invention enables the timing release task to be executed quickly and correctly, and the efficiency test is improved by 150% compared with the second set of embodiments of the present invention.

In the embodiment of the present invention, several technical connections for ensuring the improvement of timing release efficiency are as follows:

Distributed cache lock: Obtain the distributed cache lock by ID, obtain the distributed lock and query MQ related information, and then send the MQ message to the server. The client has made a compensation mechanism that may cause deadlock or abnormality. If the client does not perform the function of releasing the lock, it will increase the timeout period. If the set time is exceeded, other IDs can obtain the distributed cache. Lock. In this way, the client can work on multiple machines at the same time, and the ID can be used to ensure that only one client can obtain the scheduled task, and the task will not be executed repeatedly. The task is obtained concurrently, which is efficient and highly available.

MQ asynchronous sending: implemented based on voting, the client and server register the same topic, so that messages can be saved and notifications can be sent. It is that multiple clients send the query task to the decoration system asynchronously by sending MQ to execute the task. It is highly efficient for multiple servers to execute the published task at the same time, and the published task is executed in a separate line pool. In many cases, the number of tasks to be executed can be increased, and other tasks that are waiting can be executed when the central processor is waiting for io, thereby improving the execution efficiency of tasks.

Thread pool technology: The thread pool itself increases the execution efficiency of multiple threads, but in this embodiment of the present invention, it not only ensures the execution efficiency of multiple threads, but also ensures the rapid return of the execution result of MQ messages, preventing MQ from being out of service for a long time. A retry to return the result. Implemented based on the Thread Pool Executor base class. The core thread is sufficient to execute the task through the core thread. If the core thread is executing the task, the task is placed in the Linked Blocking Queue queue for execution.

Compensation mechanism: The server will control various exceptions to ensure the status update of timed tasks. The client uses the MQ mechanism to retry the publishing operation, but if the task has not been successfully executed for more than five minutes, it will not query the task next time, and will perform log analysis through the ID dimension to prevent other scheduled tasks from being affected by too many retries.

FIG. 6 is a schematic structural diagram of a data publishing apparatus according to an embodiment of the present invention.

As shown in FIG. 6 , the data publishing apparatus provided in the embodiment of the present invention may include: an adding module 601 , a determining module 602 , a generating module 603 , and a sending module 604 .

A joining module 601 is used to add the acquired data identifiers of the data to be published into the first thread pool;

A determination module 602, configured to determine the distributed cache lock of each to-be-released data according to the data identifier of each to-be-released data;

a generating module 603, configured to generate a first message queue MQ according to the data identifier;

The sending module 604 is used to send the first MQ to the data publishing end, so that the data publishing end can obtain a plurality of data to be published according to the plurality of data identifiers in the first MQ; add the plurality of data to be published to the second thread In the pool, the second MQ is sent to the data acquisition end to publish the data to be published.

Optionally, in an embodiment of the present invention, the device further includes:

A determination module 602, configured to determine the quantity of data to be published;

an acquisition module, configured to acquire a preset amount of data to be published when the amount of data to be published is greater than a preset amount; or,

The obtaining module is used for obtaining the data to be published when the quantity of the data to be published is not greater than the preset quantity.

Optionally, in an embodiment of the present invention, the device further includes:

A determination module 602, configured to determine the time threshold of the distributed cache lock;

The release module is used to release the distributed cache lock when the usage time of the distributed cache lock exceeds the time threshold.

The data publishing apparatus provided by the embodiment of the present invention executes each step in the method shown in FIG. 3 , and can achieve the technical effects of ensuring high availability of multiple clients and saving resources.

In the data publishing device provided by the embodiment of the present invention, the adding module 601 is used for adding the acquired data identifiers of the data to be published into the first thread pool; the determining module 602 is used for adding the data identifiers of each data to be published according to the data identifiers of each data to be published. , determine the distributed cache lock of each data to be published; the generating module 603 is used to generate the first message queue MQ according to the data identifier; the sending module 604 is used to send the first MQ to the data publisher for data The publishing end obtains multiple data to be published according to the multiple data identifiers in the first MQ; adds the multiple data to be published to the second thread pool, and sends the second MQ to the data obtaining end to publish the data to be published, which can ensure High availability of multiple clients, saving resources.

FIG. 7 is a schematic structural diagram of another data distribution apparatus provided by an embodiment of the present invention.

As shown in FIG. 7 , a data publishing apparatus provided by an embodiment of the present invention is applied to a data publishing end. The apparatus may include: a receiving module 701 , an obtaining module 702 , an adding module 703 , a sending module 704 , and a publishing module 705 .

The receiving module 701 is used for receiving the first message queue MQ sent by the data acquisition terminal; the first MQ carries the data identifier of each data to be published;

an obtaining module 702, configured to obtain data to be published corresponding to the data identifiers according to multiple data identifiers;

Joining module 703, for adding the data to be published into the second thread pool;

A sending module 704, configured to generate a second MQ according to multiple data identifiers, and send the second MQ to the data acquisition terminal, so that the data acquisition terminal can determine a plurality of to-be-published data releases;

The publishing module 705 is configured to publish the data to be published corresponding to each data identification according to each data identification.

Optionally, in an embodiment of the present invention, the publishing module 705 is specifically configured to:

According to each data identifier, determine the release page of the to-be-released data corresponding to each data identifier;

When the page area occupied by the data of the to-be-published data corresponds to the publishing page, the to-be-published data is published.

Optionally, in an embodiment of the present invention, the sending module 704 is configured to generate a second MQ according to multiple data identifiers, and send the second MQ to the data acquisition end, so that the data acquisition end determines a plurality of pending MQs. Publish data release.

Optionally, in an embodiment of the present invention, the device further includes a deletion module;

The deletion module is used to delete the historical data corresponding to the data identifier on the data publishing side.

Optionally, in an embodiment of the present invention, the device further includes:

a generating module for generating a third MQ according to the data identification, and the third MQ includes the data identification;

The sending module 704 is configured to send the third MQ to the terminal, so that the terminal obtains the data to be published from the database according to the data identifier in the third MQ, and displays the data to be published.

Optionally, in an embodiment of the present invention, deleting a module is also used to:

When the release of the data to be released is abnormal, delete the historical data corresponding to the data identifier at the data release end;

Generate a fourth MQ according to multiple data identifiers;

A fourth MQ is sent to the terminal for the terminal to determine that the data to be published is abnormally published.

Optionally, in an embodiment of the present invention, the generating module is further configured to generate a fourth MQ according to multiple data identifiers when the to-be-published data is published abnormally;

The to-be-published data is republished according to the fourth MQ.

Optionally, in an embodiment of the present invention, the device further includes an update module;

The updating module is configured to update the to-be-published data whose status in the database is abnormally published when the number of times of re-publishing the to-be-published data exceeds a preset number of times.

The data publishing apparatus provided by the embodiment of the present invention executes each step in the method shown in FIG. 4, and can achieve the technical effect of ensuring high availability of multiple clients and saving resources.

A data publishing apparatus provided by an embodiment of the present invention is configured to receive, through a receiving module 701, a first message queue MQ sent by a data acquiring end; the first MQ carries a data identifier of each data to be published; the acquiring module 702, for obtaining the data to be published corresponding to the data identifications according to the multiple data identifications; the adding module 703 for adding the data to be published into the second thread pool; the sending module 704 for generating the second MQ according to the multiple data identifications, and send the second MQ to the data acquisition end, so that the data acquisition end determines a plurality of data to be published; the publishing module 705 is used to publish the data to be published corresponding to each data identification according to each data identification, which can ensure that High availability of multiple clients, saving resources.

FIG. 8 shows a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present invention.

The electronic device may include a processor 801 and a memory 802 storing computer program instructions.

Specifically, the above-mentioned processor 801 may include a central processing unit (CPU), or a specific integrated circuit (Application Specific Integrated Circuit, ASIC), or may be configured as one or more integrated circuits implementing the embodiments of the present invention.

Memory 802 may include mass storage for data or instructions. By way of example and not limitation, memory 802 may include a Hard Disk Drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive or two or more A combination of more than one of the above. Memory 802 may include removable or non-removable (or fixed) media, where appropriate. Storage 802 may be internal or external to the integrated gateway disaster recovery device, where appropriate. In certain embodiments, memory 802 is non-volatile solid state memory. In certain embodiments, memory 802 includes read only memory (ROM). Where appropriate, the ROM may be a mask programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically rewritable ROM (EAROM) or flash memory or A combination of two or more of the above.

The processor 801 reads and executes the computer program instructions stored in the memory 802 to implement any one of the data publishing methods in the foregoing embodiments.

In one example, the electronic device may also include a communication interface 803 and a bus 810 . Among them, as shown in FIG. 8 , the processor 801 , the memory 802 , and the communication interface 803 are connected through the bus 810 and complete the mutual communication.

The communication interface 803 is mainly used to implement communication between modules, apparatuses, units and/or devices in the embodiments of the present invention.

The bus 810 includes hardware, software, or both, coupling the components of the electronic device to each other. By way of example and not limitation, the bus may include Accelerated Graphics Port (AGP) or other graphics bus, Enhanced Industry Standard Architecture (EISA) bus, Front Side Bus (FSB), HyperTransport (HT) Interconnect, Industry Standard Architecture (ISA) Bus, Infiniband Interconnect, Low Pin Count (LPC) Bus, Memory Bus, Microchannel Architecture (MCA) Bus, Peripheral Component Interconnect (PCI) Bus, PCI-Express (PCI-X) Bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local (VLB) bus or other suitable bus or a combination of two or more of the above. Bus 810 may include one or more buses, where appropriate. Although embodiments of the present invention describe and illustrate a particular bus, the present invention contemplates any suitable bus or interconnect.

The electronic device may execute the data publishing method in the embodiment of the present invention, thereby implementing the data publishing method described in conjunction with FIG. 3 and FIG. 4 .

In addition, in combination with the data publishing method in the foregoing embodiments, the embodiments of the present invention may provide a computer storage medium for implementation. Computer program instructions are stored on the computer storage medium; when the computer program instructions are executed by the processor, any one of the data publishing methods in the foregoing embodiments is implemented.

It is to be understood that the present invention is not limited to the specific arrangements and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above-described embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the sequence of steps after comprehending the spirit of the present invention.

The functional blocks shown in the above-described structural block diagrams may be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an application specific integrated circuit (ASIC), suitable firmware, a plug-in, a function card, or the like. When implemented in software, the elements of the invention are programs or code segments used to execute the required data. The program or code segments may be stored in a machine-readable medium or transmitted over a transmission medium or communication link by a data signal carried in a carrier wave. A "machine-readable medium" may include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and the like. The code segments may be downloaded via a computer network such as the Internet, an intranet, or the like.

It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps may be performed in the order mentioned in the embodiments, or may be different from the order in the embodiments, or several steps may be performed simultaneously.

The above are only specific implementations of the present invention. Those skilled in the art can clearly understand that, for the convenience and simplicity of the description, the specific working process of the above-described systems, modules and units may refer to the foregoing method embodiments. The corresponding process in , will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should all cover within the protection scope of the present invention.

Claims (13)

1. A data publishing method, wherein the method is applied to a data acquisition terminal, and the method comprises: adding the acquired data identifiers of the data to be published into the first thread pool for the data acquisition end to send or receive the message queue MQ; According to the data identifier of each of the data to be published, determine the distributed cache lock of each of the data to be published; generating a first message queue MQ according to the data identifier, where the first MQ includes the data identifier; The first MQ is sent to the data publishing end, so that the data publishing end can obtain a plurality of data to be published according to the plurality of data identifiers in the first MQ; add a plurality of the data to be published to the first MQ. In the second thread pool, a second MQ is sent to the data acquisition end to publish the data to be published; the second MQ includes the data identifier. 2. The method according to claim 1, wherein the method further comprises: determining the quantity of the data to be published; When the quantity of the data to be published is greater than a preset quantity, acquire the preset quantity of the data to be published; or, When the quantity of the data to be published is not greater than the preset quantity, the data to be published is acquired. 3. The method according to claim 1, wherein the method further comprises: determining the time threshold of the distributed cache lock; When the usage time of the distributed cache lock exceeds the time threshold, the distributed cache lock is released. 4. A data publishing method, wherein the method is applied to a data publishing terminal, and the method comprises: Receive the first message queue MQ sent by the data acquisition end; the first MQ carries the data identifier of each data to be published; Acquiring data to be published corresponding to the data identifiers according to a plurality of the data identifiers; adding the to-be-published data to the second thread pool for the data publishing end to send or receive MQ; According to each of the data identifiers, the to-be-published data corresponding to each of the data identifiers is published. 5. The method according to claim 4, wherein, according to each of the data identifiers, publishing the data to be released corresponding to each of the data identifiers, comprising: According to each of the data identifiers, determine the release page of the data to be released corresponding to each of the data identifiers; When the page area occupied by the data of the data to be published corresponds to the published page, the data to be published is published. 6. The method according to claim 4, characterized in that, the method further comprises: generating a second MQ according to a plurality of the data identifiers, and sending the second MQ to the data acquisition end to use A plurality of the to-be-published data to be published are determined at the data acquisition end. 7. The method according to claim 4, wherein the method further comprises: Generate a third MQ according to the data identification, and the third MQ includes the data identification; The third MQ is sent to the terminal, so that the terminal obtains the data to be published from the database according to the data identifier in the third MQ, and displays the data to be published. 8. The method according to any one of claims 4-7, wherein the method further comprises: When the release of the data to be released is abnormal, delete the historical data corresponding to the data identifier of the data release terminal; generating a fourth MQ according to a plurality of the data identifiers; The fourth MQ is sent to the terminal for the terminal to determine that the to-be-published data is published abnormally. 9. The method according to claim 8, wherein the method further comprises: When the to-be-published data is published abnormally, generating a fourth MQ according to a plurality of the data identifiers; The to-be-published data is republished according to the fourth MQ. 10. A data publishing device, wherein the device is applied to a data acquisition end, and the device comprises: A joining module is used to add the acquired data identifiers of the plurality of data to be published into the first thread pool; A determination module, configured to determine the distributed cache lock of each of the to-be-released data according to the data identifier of each of the to-be-released data; a generating module, configured to generate a first message queue MQ according to the data identifier; A sending module, configured to send the first MQ to a data publishing end, so that the data publishing end can obtain a plurality of data to be published according to a plurality of data identifiers in the first MQ; The published data is added to the second thread pool, and the second MQ is sent to the data acquisition end to publish the to-be-published data. 11. A data publishing device, wherein the device is applied to a data publishing terminal, and the device comprises: a receiving module, configured to receive the first message queue MQ sent by the data acquisition end; the first MQ carries the data identifier of each data to be published; an acquisition module, configured to acquire data to be published corresponding to the data identifiers according to a plurality of the data identifiers; a joining module for adding the to-be-published data to the second thread pool; a sending module, configured to generate a second MQ according to a plurality of the data identifiers, and send the second MQ to the data acquisition end, so that the data acquisition end determines a plurality of the to-be-published data releases; A publishing module, configured to publish the data to be published corresponding to each of the data identifications according to each of the data identifications. 12. An electronic device, characterized in that it comprises: at least one processor, at least one memory and computer program instructions stored in the memory, when the computer program instructions are executed by the processor, the implementation as claimed in the claims The method of any one of 1-3 or implementing the method of any one of claims 4-9. 13. A computer-readable storage medium having computer program instructions stored thereon, wherein the method or implementation of any one of claims 1-3 is implemented when the computer program instructions are executed by a processor The method of any of claims 4-9.

Images