CN113347238A - Message partitioning method, system, device and storage medium based on block chain - Google Patents

Abstract

The invention discloses a block chain-based message partition method, system, equipment and storage medium. After the block chain generates transaction messages and enters the corresponding partition, the method first calculates the resource consumption of the partition. The threshold value means that the partition is full and no new messages can be entered in the future. By calculating the matching degree of other partitions in the topic where the partition is located and the partition, and the current resource consumption of the partition with the largest matching degree value does not exceed At the first threshold, the new messages that are going to enter the partition are allocated to the partition with the largest matching degree value. On the one hand, some messages that exceed the resource carrying capacity of the partition do not need to wait for a long time for the partition to vacate resources, which saves the waiting time. On the other hand, allocating messages to the partition with the largest matching value for consensus processing ensures the reliability of the consensus, strengthens the overall ability of the Kafka cluster to process high concurrent data, and improves the overall consensus efficiency of the blockchain.

Description

Blockchain-based message partitioning method and system, equipment and storage medium

technical field

The present invention relates to the technical field of blockchain, and in particular, to a method and system, device, and computer-readable storage medium for message partitioning based on blockchain.

Background technique

As a distributed ledger technology, blockchain technology is jointly maintained by multiple parties and comprehensively uses a combination of distributed ledger, chain data structure, point-to-point transmission, consensus mechanism, and cryptographic principles. It has important features such as openness and transparency, data immutability, traceability, privacy protection and a high degree of autonomy, and has a wide range of applications in many fields. In a blockchain network, transactions initiated by different participants must be written into the ledger in the order in which they were generated. How to reach consensus on the same transaction proposal for all nodes in a distributed scenario is an important issue that blockchain technology must consider and solve. To achieve this goal, the transaction order must be established correctly, and must include the processing method for tampered transactions or maliciously submitted transactions, and the consensus algorithm is the solution to ensure the consistency of distributed systems. A process for achieving consistency on a single data value between distributed processes or systems, designed to solve the reliability problem of multiple unreliable nodes in a distributed network.

The existing blockchain network adopts Kafka's message consumption-subscription model to realize the consensus process of data. The related roles include: Producer (producer), Kafka cluster, Topic (topic), Partition (partition), Broker (data storage server) , Zookeeper cluster, Consumer (consumer). Producer is a message producer, which is played by some nodes in the blockchain network (such nodes are designated by the Zookeeper cluster, and are mainly responsible for the acquisition and transmission of messages, as well as packaging blocks). Messages (i.e. transaction data) are sent to the Kafka cluster. Among them, the Kafka cluster consists of a set of servers, which are functionally divided into Topic (topic), Partition (partition) and Broker (data storage server). Topic is a message topic. A Kafka cluster supports multiple topics at the same time. After the message enters the cluster, it enters the corresponding topic. A Partition is a partition under a Topic. A Topic can support multiple Partition partitions at the same time. After the message enters the Topic, it will be allocated to the Partition partition. The number of Partitions under Topic is managed by the Zookeeper cluster. Broker is a data storage server, which is a host server for data storage. Each topic has a Broker, which stores the message data of each current Partition. The Zookeeper cluster consists of a set of servers for managing the Kafka cluster. It can specify and register the Producer and Consumer nodes, configure the Topic and Partition of the Kafka cluster, and perform load balancing when system resources are under pressure. Consumer is a message consumer, which is played by some nodes in the blockchain network. The message is obtained locally from the Broker server by means of pull, packaged into blocks, and distributed to others in the blockchain network. node for verification. A message is the most basic data unit in Kafka. In Kafka, a message consists of two parts: key and value. When sending a message, you can specify this key, and the Producer will determine which Partition the current message should be sent and stored in based on the key. middle. There are two main strategies for selecting Partition partitions for messages: hash strategy and polling strategy. When no key is specified for the message, that is, when the key is null, the message will be sent to each partition in a polling manner. When the key is not null, The Producer node will use the hash value of the key (using the Murmur2Hash algorithm) to modulo the number of Partitions to determine which Partition to send the message to.

However, in the blockchain network, in the process of providing consensus services, the Kafka cluster involves operations such as message distribution, storage, consumption and subscription. Especially in high concurrency scenarios, a large number of messages in a short period of time may cause the Kafka cluster For example, in some large-scale commercial application scenarios, a large number of new messages will flood into a topic topic in a short period of time. When these messages are distributed to the Partition partition, the partition not only needs to process the ordering of the messages , storage, and also process the subscription request of the Consumer node. When the number of messages exceeds a certain first threshold of the partition, there will be a problem that the partition cannot continue to serve, resulting in those messages exceeding the first threshold cannot be processed in time, thus Affects the consensus efficiency of the entire cluster.

The solutions to the above situation in the prior art are usually: 1) the message enters the partition after waiting for the remaining resources in the partition; 2) the configuration file of the Partition is manually modified to increase the number of partitions. However, the following problems still exist:

1. When there are too many messages in the partition and the service cannot be continued, the messages cannot enter the partition in time for sorting and processing, and only wait for the partition resources to be vacated before entering, which is inefficient;

2. The increase of the number of partitions is currently mainly based on manual experience judgment, and it is easy to make mistakes in judgment. Such errors will cause too many new partitions and waste of resources; or if the number of new partitions is not enough, there will still be new messages that cannot be entered in time. The case where the partition is sorted;

3. There is a time lag in manually configuring partitions and the efficiency is low.

SUMMARY OF THE INVENTION

The present invention provides a block chain-based message partitioning method, system, equipment, and computer-readable storage medium, so as to solve the problem of message processing existing in high concurrency scenarios when the current block chain adopts Kafka cluster for consensus processing Inefficient technical issues.

According to one aspect of the present invention, a method for partitioning messages based on blockchain is provided, including the following:

Step S1: Deploy the Kafka consensus cluster in the blockchain network, initialize the Partition file, and configure the partition setting parameters;

Step S2: After the blockchain system generates the transaction information, the message enters the partition A through the Producer node, and calculates the resource consumption of the partition A. If the calculated resource consumption of the partition A exceeds the first threshold, the next new message will not be sent. Enter partition A, otherwise the next new message can still enter partition A;

Step S3: When the resource consumption of partition A exceeds the first threshold, calculate the matching degree of all other partitions under the same topic and partition A, form a matching degree sequence, select the partition with the largest matching degree value and calculate its current resource consumption If the current resource consumption of the partition with the largest matching degree value does not exceed the first threshold, when the next new message is ready to enter partition A, it will be allocated to the partition with the largest matching degree value.

Further, the step S3 also includes the following content:

If the current resource consumption of the partition with the largest matching degree value exceeds the first threshold, the current resource consumption of each partition is calculated one by one in descending order of the matching degree value, until it is found that the current resource consumption does not exceed the first threshold. A threshold partition and assign the next new message to partition A to that partition.

Further, it also includes the following:

Step S4: If a partition whose current resource consumption does not exceed the first threshold cannot be found in the matching degree sequence, the Partition configuration file is automatically modified to add a new partition under the topic, and subsequent new messages are allocated. to the new partition.

Further, the following formula is used to calculate the resource consumption of each partition:

Among them, CPC ^t represents the resource consumption of the partition at time t, and Res ^t (x) represents the resource consumption of the message x in the partition at time t;

Res ^t (x)=α ₁ ResNum ^t (x, M ₁ )+α ₂ ResCon ^t (x, M ₂ )+α ₃ ResDat ^t (x, M ₃ )

Among them, α ₁ , α ₂ , α ₃ ∈(0,1) are weighting coefficients, ResNum ^t (x, M ₁ ) represents the total number of subscriptions of message x at time t, and ResCon ^t (x, M ₂ ) represents message x at time t The total number of access connections, ResDat ^t (x, M ₃ ) represents the data size of the message x at time t, and the message data size includes the number of characters corresponding to the value of the message and the number of characters corresponding to the key value.

Further, the following formula is used to calculate the matching degree:

表示分区B与分区A的消息序列的相似匹配度，Meg_i表示消息i在序列中的id编号，sequenceA_i表示分区A的消息序列sequenceA的第i个消息，sequenceB_j表示分区B的消息序列sequenceB的第j个消息，β_i,j∈(0,1)为加权系数，p为指数参数，p≥1。in,

Indicates the similarity matching degree of the message sequence of partition B and partition A, Megi represents the id number of message i in the sequence, sequenceA _i represents the _ith message of the message sequence sequenceA of partition A, and sequenceB _j represents the message sequence sequenceB of partition B The jth message of , β _i,j ∈(0,1) is the weighting coefficient, p is the index parameter, p≥1.

Further, it also includes the following:

Step S5: Monitor the resource consumption of each partition, when the resource consumption of at least one partition is less than the second threshold, delete the new partition after the new partition completes the consensus processing.

In addition, another embodiment of the present invention also provides a block chain-based message partitioning system, including:

The configuration module is used to deploy the Kafka consensus cluster in the blockchain network, initialize the Partition file, and configure the partition setting parameters;

The first calculation module is used to calculate the resource consumption of the partition A when the message enters the partition A;

The second calculation module is configured to calculate the matching degree between all other partitions under the same topic and the partition A when the resource consumption of the partition A calculated by the first calculation module exceeds the first threshold, to form a matching degree sequence;

The allocation module is used to select the partition with the largest matching degree value from the matching degree sequence and calculate its current resource consumption, if the current resource consumption of the partition with the largest matching degree value does not exceed the first threshold, then the next When a new message is ready to enter partition A, it is allocated to the partition with the highest matching degree value.

Further, it also includes:

The update module is used to automatically modify the Partition configuration file to add a new partition under the topic when the partition whose current resource consumption does not exceed the first threshold cannot be found in the matching degree sequence, and the subsequent new message assigned to the new partition.

In addition, another embodiment of the present invention also provides a device, including a processor and a memory, where a computer program is stored in the memory, and the processor is configured to execute the computer program by invoking the computer program stored in the memory The steps of the method as described above.

In addition, another embodiment of the present invention also provides a computer-readable storage medium for storing a computer program for message partitioning based on blockchain, the computer program executing the above method when running on a computer step.

The present invention has the following effects:

In the method for message partitioning based on the blockchain of the present invention, after the blockchain generates a transaction message and enters the corresponding partition, the resource consumption of the partition is calculated first. If it exceeds the first threshold, it means that the partition is full and cannot be further Enter a new message, calculate the matching degree of other partitions in the topic where the partition is located and the partition, and when the current resource consumption of the partition with the largest matching degree value does not exceed the first threshold New messages are allocated to the partition with the largest matching degree value. On the one hand, some messages that exceed the resource carrying capacity of the partition do not need to wait for a long time for the partition to vacate resources, which saves the waiting time. On the other hand, the message is allocated to the largest matching degree value. Consensus processing is performed on the partitions of Kafka, which ensures the reliability of the consensus, strengthens the ability of the Kafka cluster to process high concurrent data as a whole, and improves the overall consensus efficiency of the blockchain.

In addition, the blockchain-based message partitioning system of the present invention also has the above advantages.

In addition to the objects, features and advantages described above, the present invention has other objects, features and advantages. The present invention will be described in further detail below with reference to the drawings.

Description of drawings

The accompanying drawings constituting a part of the present application are used to provide further understanding of the present invention, and the exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the attached image:

Figure 1 is a schematic diagram of the architecture of an existing blockchain network using a Kafka cluster for consensus processing.

FIG. 2 is a schematic flowchart of a block chain-based message partitioning method according to a preferred embodiment of the present invention.

FIG. 3 is a schematic flowchart of another embodiment of the block chain-based message partitioning method of the present invention.

FIG. 4 is a schematic flowchart of another embodiment of the method for message partitioning based on the blockchain of the present invention.

FIG. 5 is a schematic diagram of a module structure of a block chain-based message partitioning system according to another embodiment of the present invention.

Detailed ways

The embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered below.

As shown in Figure 1, in the current blockchain network using Kafka cluster for consensus processing, the specific consensus process includes the following:

The client sends transaction information (ie messages) to the Producer node in the blockchain consensus module;

According to the key value of the message, the Producer node pushes the message to a Partition partition under the Topic in the Kafka cluster according to the hash modulo or polling method;

Messages are sorted in partitions to form an ordered message queue;

The Consumer node obtains the message queue from the Partition partition under the subscribed Topic and packs it into blocks;

Consumer nodes distribute blocks to other nodes in the blockchain network for verification.

Through the above consensus process, the blockchain network can use the Kafka cluster to quickly perform data consensus. However, during this consensus process, if the instantaneous number of messages is too large (such as high concurrency), it is likely that there are not enough Partition partitions. At this time, the redundant messages can only enter the partition for sorting after the partition resources are vacated. Therefore, as shown in FIG. 2, a preferred embodiment of the present invention provides a block chain-based message partitioning method, including the following:

Step S1: Deploy the Kafka consensus cluster in the blockchain network, initialize the Partition file, and configure the partition setting parameters;

Step S2: After the blockchain system generates the transaction information, the message enters the partition A through the Producer node, and calculates the resource consumption of the partition A. If the calculated resource consumption of the partition A exceeds the first threshold, the next new message will not be sent. Enter partition A, otherwise the next new message can still enter partition A;

Step S3: When the resource consumption of partition A exceeds the first threshold, calculate the matching degree of all other partitions under the same topic and partition A, form a matching degree sequence, select the partition with the largest matching degree value and calculate its current resource consumption If the current resource consumption of the partition with the largest matching degree value does not exceed the first threshold, when the next new message is ready to enter partition A, it will be allocated to the partition with the largest matching degree value.

It can be understood that in the step S1, the configured partition setting parameters include the number of partitions, the first threshold of resource consumption of each partition, etc. In addition, the number of topics contained in the Kafka consensus cluster and the consensus algorithm of each topic are both Configured during the initialization phase. After the partition file is configured, a preset number of partitions are generated by default under each topic.

It can be understood that in the step S2, after the message is allocated to the partition A according to the hash strategy or the polling strategy, if the calculated current resource consumption of the partition A exceeds the first threshold, it means that the partition A is full, and the follow-up cannot be performed. Then enter a new message, and the subsequent messages allocated to partition A need to be transferred. If it is calculated that the current resource consumption of partition A does not exceed the first threshold, it means that partition A still has remaining resource processing capacity, and subsequent messages can still be allocated to enter without the risk of overload.

It can be understood that in the step S3, when the resource consumption of the partition A exceeds the first threshold, it means that the partition A is full. At this time, by calculating the matching degree between other partitions under the same topic and partition A, the partition with the largest matching degree value is filtered out, which means that it is the best match with partition A in terms of message attributes, and then the next one that is ready to enter partition A is selected. New messages are assigned to the partition with the highest matching value. On the one hand, it saves the waiting time for new messages and improves the efficiency of message processing. On the other hand, since the partition with the largest matching degree value is the closest to partition A in terms of message attributes, the reliability of consensus processing is ensured.

It can be understood that, in the method for message partitioning based on the blockchain in this embodiment, after the blockchain generates a transaction message and enters the corresponding partition, the resource consumption of the partition is calculated first. If it exceeds the first threshold, it means that the partition is full. , no new messages can be entered in the future, then by calculating the matching degree of other partitions in the topic where the partition is located and the partition, and when the current resource consumption of the partition with the largest matching degree value does not exceed the first threshold, the subsequent preparation New messages entering the partition are allocated to the partition with the largest matching degree value. On the one hand, some messages that exceed the resource carrying capacity of the partition do not need to wait for a long time for the partition to vacate resources, which saves the waiting time. On the other hand, the messages are allocated to Consensus processing is performed on the partition with the largest matching degree value, which ensures the reliability of the consensus, strengthens the ability of the Kafka cluster to process high concurrent data as a whole, and improves the overall consensus efficiency of the blockchain.

It can be understood that the step S3 also includes the following content:

If the current resource consumption of the partition with the largest matching degree value exceeds the first threshold, the current resource consumption of each partition is calculated one by one in descending order of the matching degree value, until it is found that the current resource consumption does not exceed the first threshold. A threshold partition and assign the next new message to partition A to that partition.

When the current resource consumption of the partition with the largest matching degree value exceeds the first threshold, it means that it is also in a fully loaded state and can no longer perform consensus processing on new messages. Screen until a partition whose current resource consumption does not exceed the first threshold is found, and the next new message allocated to partition A is transferred to this partition for consensus processing.

It can be understood that, as shown in Figure 3, the block chain-based message partitioning method further includes the following contents:

Step S4: If a partition whose current resource consumption does not exceed the first threshold cannot be found in the matching degree sequence, the Partition configuration file is automatically modified to add a new partition under the topic, and subsequent new messages are allocated. to the new partition.

If a partition whose current resource consumption does not exceed the first threshold cannot be found under the topic where partition A is located, it means that all partitions under the topic are full and cannot process new messages. In this case, modify the partition configuration file by modifying the partition configuration file. A new partition is added under the topic, and subsequent new messages are allocated to the new partition, and the new partition is sorted by consensus according to the preset consensus algorithm. Therefore, the number of partitions can be automatically increased according to the message processing capability of the entire topic to ensure the message processing efficiency.

It can be understood that, in the present invention, two data indicators of resource consumption and partition matching degree are used to select a message partition. Among them, the resource consumption of the partition is used to describe the resources consumed by a partition at the current moment for processing messages, and the partition matching degree is used to describe the similarity matching degree between different partitions under the same topic at the same time. Specifically, the following formula is used to calculate the resource consumption of each partition:

Among them, CPC ^t represents the resource consumption of the partition at time ^t , and Rest (x) represents the resource consumption of message x in the partition at time t, then the current resource consumption of the partition is equal to the sum of the resource consumption of all messages in the current partition.

Res ^t (x)=α ₁ ResNum ^t (x, M ₁ )+α ₂ ResCon ^t (x, M ₂ )+α ₃ ResDat ^t (x, M ₃ )

Among them, α ₁ , α ₂ , α ₃ ∈(0,1) are weighting coefficients, ResNum ^t (x, M ₁ ) represents the total number of subscriptions of message x at time t, and ResCon ^t (x, M ₂ ) represents message x at time t The total number of access connections, ResDat ^t (x, M ₃ ) represents the data size of the message x at time t, and the message data size includes the number of characters corresponding to the value of the message and the number of characters corresponding to the key value. The present invention comprehensively evaluates the resource consumption of the partition based on the three dimensions of the total number of message subscriptions, the total number of connections and the data size, thereby improving the accuracy of calculation.

In addition, the following formula is used to calculate the matching degree:

表示分区B与分区A的消息序列的相似匹配度，Meg_i表示消息i在序列中的id编号，sequenceA_i表示分区A的消息序列sequenceA的第i个消息，sequenceB_j表示分区B的消息序列sequenceB的第j个消息，β_i,j∈(0,1)为加权系数，p为指数参数，p≥1。本发明通过采用闵可夫斯基距离来计算两个分区消息序列的相似匹配度，并结合了加权计算，可以准确地计算出两个分区的匹配度。Assuming that for any Partition partition, the sequence of messages possessed at time t is represented as Partition ^t = {Meg ₁ , Meg ₂ ,..., Meg _n }, where n is the total number of messages in the sequence at time t, and Meg _i represents message i The id number in the sequence, an id number is generated when the message is inserted into the sequence. Among them, PMD

Indicates the similarity matching degree of the message sequence of partition B and partition A, Megi represents the id number of message i in the sequence, sequenceA _i represents the _ith message of the message sequence sequenceA of partition A, and sequenceB _j represents the message sequence sequenceB of partition B The jth message of , β _i,j ∈(0,1) is the weighting coefficient, p is the index parameter, p≥1. The invention calculates the similarity matching degree of the two partition message sequences by adopting the Minkowski distance, and combining with the weighted calculation, the matching degree of the two partitions can be accurately calculated.

In addition, when p=2, the calculated result is the Manhattan distance, that is, the formula can be expressed as:

It can be understood that, as shown in Figure 4, the block chain-based message partitioning method further includes the following contents:

Step S5: Monitor the resource consumption of each partition, when the resource consumption of at least one partition is less than the second threshold, delete the new partition after the new partition completes the consensus processing.

Real-time monitoring or regular monitoring of the resource consumption of each partition is performed. If the resource consumption of at least one partition is less than the second threshold, it means that at least one partition has sufficient resource carrying capacity. At this time, the consensus is completed in the new partition Delete the new partition after the task to avoid wasting resources after the high concurrency business scenario ends. The second threshold is smaller than the first threshold, and the specific value of the second threshold is also configured during initialization in step S1.

In addition, as shown in FIG. 5 , another embodiment of the present invention also provides a block chain-based message partitioning system, preferably using the message partitioning method described above, and the system includes:

The configuration module is used to deploy the Kafka consensus cluster in the blockchain network, initialize the Partition file, and configure the partition setting parameters;

The first calculation module is used to calculate the resource consumption of the partition A when the message enters the partition A;

The second calculation module is configured to calculate the matching degree between all other partitions under the same topic and the partition A when the resource consumption of the partition A calculated by the first calculation module exceeds the first threshold, to form a matching degree sequence;

The allocation module is used to select the partition with the largest matching degree value from the matching degree sequence and calculate its current resource consumption, if the current resource consumption of the partition with the largest matching degree value does not exceed the first threshold, then the next When a new message is ready to enter partition A, it is allocated to the partition with the highest matching degree value.

It can be understood that in the blockchain-based message partition system of this embodiment, after the blockchain generates a transaction message and enters the corresponding partition, the resource consumption of the partition is calculated first. If it exceeds the first threshold, it means that the partition is full. , no new messages can be entered in the future, then by calculating the matching degree of other partitions in the topic where the partition is located and the partition, and when the current resource consumption of the partition with the largest matching degree value does not exceed the first threshold, the subsequent preparation New messages entering the partition are allocated to the partition with the largest matching degree value. On the one hand, some messages that exceed the resource carrying capacity of the partition do not need to wait for a long time for the partition to vacate resources, which saves the waiting time. On the other hand, the messages are allocated to Consensus processing is performed on the partition with the largest matching degree value, which ensures the reliability of the consensus, strengthens the ability of the Kafka cluster to process high concurrent data as a whole, and improves the overall consensus efficiency of the blockchain.

In addition, the message partition system further includes:

The update module is used to automatically modify the Partition configuration file to add a new partition under the topic when the partition whose current resource consumption does not exceed the first threshold cannot be found in the matching degree sequence, and the subsequent new message assigned to the new partition.

In addition, the message partition system further includes:

The monitoring module is used to monitor the resource consumption of each partition, and when the resource consumption of at least one partition is less than the second threshold, delete the new partition after the new partition completes the consensus processing.

It can be understood that each module in the system of this embodiment corresponds to each step in the foregoing method embodiment, so the specific working process of each module is not repeated here.

In addition, another embodiment of the present invention also provides a device, including a processor and a memory, where a computer program is stored in the memory, and the processor is configured to execute the computer program by invoking the computer program stored in the memory The steps of the method as described above.

In addition, another embodiment of the present invention also provides a computer-readable storage medium for storing a computer program for message partitioning based on blockchain, the computer program executing the above method when running on a computer step.

Typical forms of computer readable media include: floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards ( punchcards), paper tape, any other physical media with a pattern of holes, random access memory (RAM), programmable read only memory (PROM), erasable programmable read only memory (EPROM) , Flash-Erasable Programmable Read-Only Memory (FLASH-EPROM), any other memory chip or cartridge, or any other computer-readable medium. The instructions may further be transmitted or received by a transmission medium. The term transmission medium can include any tangible or intangible medium that can be used to store, encode, or carry instructions for execution by a machine, and includes digital or analog communication signals or intangible media that facilitate communication of such instructions. Transmission media include coaxial cables, copper wire, and fiber optics, which contain the wires of a bus used to transmit a computer data signal.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. A block chain-based message partitioning method is characterized in that, comprising the following content: Step S1: Deploy the Kafka consensus cluster in the blockchain network, initialize the Partition file, and configure the partition setting parameters; Step S2: After the blockchain system generates the transaction information, the message enters the partition A through the Producer node, and calculates the resource consumption of the partition A. If the calculated resource consumption of the partition A exceeds the first threshold, the next new message will not be sent. Enter partition A, otherwise the next new message can still enter partition A; Step S3: When the resource consumption of partition A exceeds the first threshold, calculate the matching degree of all other partitions under the same topic and partition A, form a matching degree sequence, select the partition with the largest matching degree value and calculate its current resource consumption If the current resource consumption of the partition with the largest matching degree value does not exceed the first threshold, when the next new message is ready to enter partition A, it will be allocated to the partition with the largest matching degree value. 2. The block chain-based message partitioning method according to claim 1, wherein the step S3 further comprises the following content: If the current resource consumption of the partition with the largest matching degree value exceeds the first threshold, the current resource consumption of each partition is calculated one by one in descending order of the matching degree value, until it is found that the current resource consumption does not exceed the first threshold. A threshold partition and assign the next new message to partition A to that partition. 3. The block chain-based message partitioning method according to claim 2, further comprising the following content: Step S4: If a partition whose current resource consumption does not exceed the first threshold cannot be found in the matching degree sequence, the Partition configuration file is automatically modified to add a new partition under the topic, and subsequent new messages are allocated. to the new partition. 4. The block chain-based message partitioning method of claim 1, wherein the following formula is used to calculate the resource consumption of each partition:

Among them, CPC ^t represents the resource consumption of the partition at time t, and Res ^t (x) represents the resource consumption of the message x in the partition at time t; Res ^t (x)=α ₁ ResNum ^t (x, M ₁ )+α ₂ ResCon ^t (x, M ₂ )+α ₃ ResDat ^t (x, M ₃ ) Among them, α ₁ , α ₂ , α ₃ ∈(0,1) are weighting coefficients, ResNum ^t (x, M ₁ ) represents the total number of subscriptions of message x at time t, and ResCon ^t (x, M ₂ ) represents message x at time t The total number of access connections, ResDat ^t (x, M ₃ ) represents the data size of the message x at time t, and the message data size includes the number of characters corresponding to the value of the message and the number of characters corresponding to the key value. 5. The block chain-based message partitioning method as claimed in claim 1, wherein the following formula is used to calculate the matching degree:

in,

Indicates the similarity matching degree of the message sequence of partition B and partition A, Megi represents the id number of message i in the sequence, sequenceA _i represents the _ith message of the message sequence sequenceA of partition A, and sequenceB _j represents the message sequence sequenceB of partition B The jth message of , β _i,j ∈(0,1) is the weighting coefficient, p is the index parameter, p≥1. 6. The block chain-based message partitioning method of claim 3, further comprising the following: Step S5: Monitor the resource consumption of each partition, when the resource consumption of at least one partition is less than the second threshold, delete the new partition after the new partition completes the consensus processing. 7. A block chain-based message partitioning system, comprising: The configuration module is used to deploy the Kafka consensus cluster in the blockchain network, initialize the Partition file, and configure the partition setting parameters; The first calculation module is used to calculate the resource consumption of the partition A when the message enters the partition A; The second calculation module is configured to calculate the matching degree between all other partitions under the same topic and the partition A when the resource consumption of the partition A calculated by the first calculation module exceeds the first threshold, to form a matching degree sequence; The allocation module is used to select the partition with the largest matching degree value from the matching degree sequence and calculate its current resource consumption, if the current resource consumption of the partition with the largest matching degree value does not exceed the first threshold, then the next When a new message is ready to enter partition A, it is allocated to the partition with the highest matching degree value. 8. The block chain-based message partitioning system of claim 7, further comprising: The update module is used to automatically modify the Partition configuration file to add a new partition under the topic when the partition whose current resource consumption does not exceed the first threshold cannot be found in the matching degree sequence, and the subsequent new message assigned to the new partition. 9. A device, characterized by comprising a processor and a memory, wherein a computer program is stored in the memory, and the processor is used to execute the computer program according to claim 1 by calling the computer program stored in the memory. 6 the steps of any one of the methods. 10. A computer-readable storage medium for storing a computer program for message partitioning based on a block chain, characterized in that, when the computer program is run on a computer, the execution of any one of claims 1 to 6 is performed. steps of the method.

Images