CN118101651B - Distributed system for realizing low retention of service high-availability data - Google Patents

Abstract

The present application relates to the field of data processing technology, and in particular to a distributed system for realizing high-availability services and low data retention, wherein the system comprises: a first service platform, a plurality of second service platforms, a scheduling platform, a plurality of initial data access platforms, a processor, and a memory storing a computer program, and when the computer program is executed by the processor, the following steps are implemented: when the scheduling platform receives a target query request sent by any second service platform, the target data access platform is obtained from a plurality of initial data access platforms, and according to the target data access platform, the scheduling platform sends the target query request to the first service platform, and according to the target query request, the first service platform sends the target service-related data set to the second service platform, and the second service platform completes the deletion or storage of the target service-related data. It can be seen that the present invention can select whether to retain data according to the data type, thereby improving data security.

Description

A distributed system that achieves high service availability and low data retention

Technical Field

The present invention relates to the field of data processing technology, and in particular to a distributed system that realizes high-availability services and low data retention.

Background Art

At present, for multiple low-level platforms, the personnel data they collect are stored separately without data sharing, and the availability is low. In addition, due to the low security of low-level platforms, or the large amount of personnel information stored, that is, when there is a high data retention in the low-level platforms, there will be problems of information leakage, resulting in low data security. In addition, if the data of multiple low-level platforms are transmitted to the high-level platform for unified storage and management, when each low-level platform needs to query personnel information, problems such as access congestion may occur, affecting the normal operation of the system.

Summary of the invention

In view of the above technical problems, the technical solution adopted by the present invention is:

A distributed system for realizing high-availability service and low-retention data, the system comprising: a first service platform, a plurality of second service platforms, a scheduling platform, a plurality of initial data access platforms, a processor, and a memory storing a computer program, the first service platform is communicatively connected with each of the second service platforms, each of the second service platforms is communicatively connected with the scheduling platform, the scheduling platform is communicatively connected with each initial data access platform, and each initial data access platform is communicatively connected with the first service platform, and when the computer program is executed by the processor, the following steps are implemented:

S100: When the scheduling platform receives a target query request sent by any second service platform, it obtains a target data access platform from a plurality of initial data access platforms.

S200: According to the target data access platform, the scheduling platform sends the target query request to the first service platform.

S300: According to the target query request, the first service platform sends a target service associated data set corresponding to the target query request to the second service platform.

S400: When the second service platform receives the target service associated data set, it determines a data tag corresponding to each target service associated data in the target service associated data set.

S500: When the data tag corresponding to the target service-related data is the first data tag, the second service platform deletes the target service-related data.

S600: When the data tag corresponding to the target service associated data is a second data tag, the second service platform stores the target service associated data.

Compared with the prior art, the present invention has obvious beneficial effects. By means of the above technical solution, the present invention provides a distributed system for realizing high-availability services and low data retention, which can achieve considerable technical advancement and practicality, and has wide industrial utilization value, and has at least the following beneficial effects:

The present invention provides a distributed system for realizing high-availability service and low-retention data. The system comprises: a first service platform, several second service platforms, a scheduling platform, several initial data access platforms, a processor and a memory storing a computer program. The first service platform is communicatively connected with each of the second service platforms, each of the second service platforms is communicatively connected with the scheduling platform, the scheduling platform is communicatively connected with each initial data access platform, and each initial data access platform is communicatively connected with the first service platform. When the computer program is executed by the processor, the following steps are implemented: when the scheduling platform receives a target query request sent by any second service platform, a target data access platform is obtained from several initial data access platforms, and the target query request is sent to the first service platform through the target data access platform. According to the target query request, the first service platform sends a target service-related data set corresponding to the target query request to the second service platform. When the data label corresponding to the target service-related data is a first data label, the second service platform deletes the target service-related data. When the data label corresponding to the target service-related data is a second data label, the second service platform stores the target service-related data. On the one hand, by obtaining the target data access platform, the target query request can be dispatched to the target data access platform rather than other initial data access platforms, preventing congestion on other initial data access platforms, which is beneficial to maintaining the normal operation of the system; on the other hand, the second data platform can prevent the leakage of personnel information in the second data platform by deleting data that cannot be retained according to the type of data tag, thereby enhancing data security.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

FIG1 is a flow chart of a distributed system executing a computer program to implement high-availability service and low-data retention, provided by an embodiment of the present invention.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

The present invention provides a distributed system for realizing high-availability service and low-retention data, the system comprising: a first service platform, a plurality of second service platforms, a scheduling platform, a plurality of initial data access platforms, a processor and a memory storing a computer program, the first service platform is communicatively connected with each of the second service platforms, each of the second service platforms is communicatively connected with the scheduling platform, the scheduling platform is communicatively connected with each initial data access platform, and each initial data access platform is communicatively connected with the first service platform, when the computer program is executed by the processor, the following steps are implemented, as shown in FIG1:

S100: When the scheduling platform receives a target query request sent by any second service platform, it obtains a target data access platform from a plurality of initial data access platforms.

Specifically, the first service platform, several second service platforms, the scheduling platform and several initial data access platforms are connected in the form of a distributed architecture to achieve communication between the various platforms.

Specifically, the first service platform is used to monitor each of the second service platforms.

Specifically, the scheduling platform refers to a platform that sends a query request to an initial data access platform when receiving the query request sent by the second service platform.

Specifically, the initial data access platform refers to a data access platform that sends a query request to the first service platform when receiving a query request sent by the scheduling platform.

Specifically, the target query request refers to a request for querying a target service associated data set corresponding to a target user.

In a specific embodiment, step S100 also includes the following steps:

S101, when the scheduling platform receives a target query request sent by any second service platform, obtains an access priority set E={E ₁ , E ₂ , ..., E _δ , ..., E _d } corresponding to the initial data access platform, where E _δ is the access priority corresponding to the δth initial data access platform, δ=1, 2...d, and d is the number of initial data access platforms;

Among them, E _δ meets the following conditions:

E _δ =1/Q _δ , where Q _δ is the number of visits received by the δth initial data access platform within one second before the current moment.

Specifically, the current time refers to the time when the scheduling platform receives the target query request sent by the second service platform.

S102: The scheduling platform obtains a target data access platform according to the access priority corresponding to each of the initial data access platforms.

Specifically, the target data access platform is an initial data access platform corresponding to the highest access priority.

As mentioned above, by calculating the access priority corresponding to each initial data access platform, the access volume of each initial data access platform can be known, so that the target query request can be scheduled to the initial data access platform with the highest access priority rather than other initial data access platforms. Reasonable allocation can be achieved, and congestion can be prevented from occurring in the initial data access platforms with low access priority, which is conducive to maintaining the normal operation of the system.

S200, according to the target data access platform, the scheduling platform sends the target query request to the first service platform; it can be understood that: the scheduling platform sends the target query request to the target data access platform, and the target data access platform sends the target query request to the first service platform.

S300, according to the target query request, the first service platform sends the target service associated data set corresponding to the target query request to the second service platform; it can be understood that: the target service associated data set is sent to the second service platform and the target service associated data set is also stored in the first service platform.

As described above, when the second service platform needs to use the target service associated data set, it sends a target query request corresponding to the target service associated data set to the first service platform, so as to obtain the target service associated data set and then process subsequent work.

In a specific embodiment, before step S100, the target service associated data set is obtained by the following steps:

S001, when the first service platform receives a target service name of any second service platform, the first service platform sends initial service web page information corresponding to the target service name to the second service platform.

Specifically, the first service platform is used to monitor the second service platform; it can be understood that: the level of the first service platform is greater than the level of the second service platform, that is, the first service platform can monitor the operation status, storage status, etc. of the second service platform.

Specifically, the initial service web page information includes: an initial service name, a website address corresponding to the initial service name, and web page information corresponding to the initial service name; for example, the initial service name is "personnel information collection" and the like.

Furthermore, the URL corresponding to the initial service name is characterized as a URL used to obtain the web page information corresponding to the initial service name; it can be understood that: by operating the URL of the initial service name, you can jump to the initial service web page and obtain the web page information corresponding to the initial service name.

Furthermore, the web page information corresponding to the initial service name includes several web page option names corresponding to the initial service names, labels of each web page option name, and rules of each web page option name, wherein, for example, the web page option name may be name, urine test, alcohol content, etc.

Specifically, the label of the web page option name is used to indicate whether the web page option name needs to be re-collected. For example, the label of blood type is "1", and the label of urine test is "0". "0" indicates that re-collection is required, and "1" indicates that re-collection is not required.

Specifically, the rule of the web page option name refers to the rule of the collection requirement corresponding to the web page option name, for example, the rule of urine test is to collect 5-10 ml.

Specifically, the target service name refers to the service name of the target service, wherein the target service refers to the service that the second service platform needs to request.

Specifically, step S001 also includes the following steps:

S0011, obtain the word vector A corresponding to the target service name and the word vector set A ⁰ ={A ⁰ ₁ , A ⁰ ₂ , ..., A ⁰ _j , ..., A ⁰ _n } corresponding to the initial service name, where A ⁰ _j refers to the word vector corresponding to the j-th initial service name, j=1, 2...n.

S0012, based on A and A ⁰ , obtain the word vector similarity F = {F ₁ , F ₂ , ..., F _j , ..., F _n } corresponding to A, where F _j refers to the similarity between A and A ⁰ _j .

Specifically, F _j meets the following conditions:

F _j =(A·A ⁰ _j )/(||A||×||A ⁰ _j ||).

S0013, when the number of maximum similarities in F is 1, the initial service web page information corresponding to the maximum similarity is sent to the second service platform; it can be understood that: if there is only one maximum value among the n similarities, the initial service web page information corresponding to the maximum similarity is sent to the second service platform.

S0014: When the number of maximum similarities in F is not 1, the first prompt information is fed back to the second service platform.

Specifically, the first prompt information is used to indicate that the target service name sent by the second service platform is incorrect.

In the above, by respectively calculating the similarity between the word vectors corresponding to all initial service names in the first service platform and the word vectors corresponding to the target service name, the initial service name with the greatest similarity to the target service name can be obtained. Even when the target service name is inconsistent with all the initial service names, the initial service web page information corresponding to the target service name can be obtained, thereby improving the usability. When the maximum number of similarities is not 1, the first prompt information can be sent to the second service platform so that the staff of the second service platform can correct the target service name in time.

S002: When the first service platform receives the first service-related data set fed back by the second service platform, the first service platform generates a target service-related data set according to the first service-related data set.

Specifically, in S002, the target service associated data set is generated through the following steps:

S0021, when the first service platform receives the first feedback information from the second service platform, it queries the intermediate service related data set from the preset user database of the first service platform according to the first feedback information, and sends the intermediate service related data set to the second service platform; it can be understood that: when the second service platform receives the initial service web page information corresponding to the target service name sent by the first service platform, it sends the first feedback information to the first service platform.

Specifically, the preset user database is a database storing several collected user-related data information.

In a specific embodiment, the first feedback information is realized by the following steps:

S1, the second service platform traverses the label list of intermediate web page option names, wherein the label list of intermediate web page option names includes several labels of intermediate web page option names, wherein the label of the intermediate web page option name refers to the label of any web page option name corresponding to the initial service name in the initial service web page information corresponding to the target service name.

S2, when the label of the intermediate web page option name is of the first label type, the second service platform obtains the associated data of the target user corresponding to the intermediate web page option name; it can be understood as: collecting data of the target user according to each item in the intermediate web page option name, and transmitting the collected data to the second service platform.

Specifically, the target user refers to a user whose relevant data information is collected when processing the target service corresponding to the target service name.

Furthermore, the first tag type refers to a tag type corresponding to a web page option name that needs to be re-collected for each service request.

Furthermore, the associated data of the target user refers to data corresponding to the target user collected according to the rule of the intermediate web page option name. For example, when the intermediate web page option name is a blood index or a urine test index, the rule of the intermediate web page option name is to collect 5-10ml; when the intermediate web page option name is a face image, the rule of the intermediate web page option name is a bare-headed image.

S3, when the label of the intermediate web page option name is the second label type, the second service platform generates the collection item information of the target user and uses the collection item information of the target user as the first feedback information, which can be understood as: the second service platform feeds back the intermediate web page option name corresponding to the second label type to the first service platform.

Furthermore, the second tag type refers to a tag type corresponding to a web page option name that does not need to be re-collected for each service request.

Furthermore, the intermediate service associated data set includes a number of intermediate service associated data, wherein the intermediate service associated data refers to relevant data of the target user corresponding to the intermediate web page option name queried from a preset user database based on the intermediate web page option name in the first feedback information. It can be understood that: the intermediate service associated data is data that has been collected and stored in the first service platform.

S0022: The first service platform queries a second service-related data set from a preset user database of the first service platform according to the intermediate service-related data set.

Specifically, the second service association data set includes a plurality of second service association data, wherein the second service association data refers to the intermediate service association data corresponding when the intermediate service association data is not null.

S0023: When the first service platform receives the first service-related data set fed back by the second service platform, the first service-related data set and the second service-related data set are merged into a target service-related data set.

Specifically, the first service-related data set includes several first service-related data, and the first service-related data is the first category of related data or the second category of related data, wherein the first category of related data refers to the related data of the target user corresponding to the intermediate web page option name obtained when the label of the intermediate web page option name is the first label type; the second category of related data refers to the relevant data of the target user re-collected according to the intermediate web page option name in the first feedback information when the intermediate service-related data is null; it can be understood that: if the second service platform does not obtain a certain intermediate service-related data in the first service platform, then according to the intermediate web page option name corresponding to the intermediate service-related data, the data of the target user that has been re-collected is obtained.

As described above, the first service platform sends the intermediate service associated data set to the second service platform, so that the second service platform does not need to collect the data already in the intermediate service associated data set again, saving time and human resources. At the same time, the second service associated data set is merged with the first service associated data set sent by the second service platform into a target service associated data set and is uniformly stored in the first service platform, thereby realizing the expansion of the relevant data of the target user, so that each second service platform can call the relevant data of the target user when performing related service processing, thereby achieving the high availability of the service platform.

S400: When the second service platform receives the target service associated data set, it determines a data tag corresponding to each target service associated data in the target service associated data set.

Specifically, the target service associated data is the first service associated data or the second service associated data.

S500, when the data tag corresponding to the target service-related data is the first data tag, the second service platform deletes the target service-related data; it can be understood that: after using the target service-related data, the second service platform deletes the target service-related data and does not retain the target service-related data.

Specifically, the first data tag refers to the type of data that cannot be retained on the second service platform, which can be understood as: data not corresponding to the first data tag refers to data that can be retained on the second service platform. For example, data that cannot be retained may be collected SIM card numbers, user accounts, and other data that represent unique user-related information.

S600: When the data tag corresponding to the target service associated data is a second data tag, the second service platform stores the target service associated data.

Specifically, the second data tag refers to the type of data that can be retained on the second service platform. For example, the data that can be retained may be the primary key data of the target service-related data or the desensitized serial number and other data that will not disclose user personal information.

As described above, by judging the type of data tag corresponding to each target service-related data, the data that can be retained is stored in the second service platform, and the data that cannot be retained is deleted. A large amount of data is deleted from the second service platform, and only a small amount of data that does not involve the user's personal privacy is retained, thereby achieving low data retention. This prevents the operator from obtaining several service-related data of the user from the second service platform during subsequent use, thereby preventing the leakage of personnel information in the second data platform and enhancing data security.

In a specific embodiment, the following steps are also included:

S01, according to a preset time interval, the first service platform detects the storage disk in the second service platform. Those skilled in the art set the time interval according to actual needs, which will not be described in detail here.

Specifically, the storage disk in the second service platform refers to one or more disks used to store all data in the second service platform.

S02: If the target service-related data is stored in the storage disk in the second service platform, the first service platform sends a second prompt message to the second service platform.

Specifically, the second prompt information is used to be characterized as information prompting the second service platform to delete the target service-related data.

As mentioned above, the first service platform detects the storage disk of each second service platform at regular intervals, and when the target service-related data is stored in the second service platform, sends a prompt message to the second service platform to delete the target service-related data, thereby serving as a warning to prevent the second service platform from storing the target service-related data and protecting the security of user data.

In summary, the present invention provides a distributed system for realizing high service availability and low data retention. The system includes: a first service platform, several second service platforms, a scheduling platform, several initial data access platforms, a processor and a memory storing a computer program. The first service platform is communicatively connected to each of the second service platforms, each of the second service platforms is communicatively connected to the scheduling platform, the scheduling platform is communicatively connected to each initial data access platform, and each initial data access platform is communicatively connected to the first service platform. When the computer program is executed by the processor, the following steps are implemented: when the scheduling platform receives a target query request sent by any second service platform, a target data access platform is obtained from several initial data access platforms, and the target query request is sent to the first service platform through the target data access platform. According to the target query request, the first service platform sends a target service associated data set corresponding to the target query request to the second service platform. When the data label corresponding to the target service associated data is the first data label, the second service platform deletes the target service associated data. When the data label corresponding to the target service associated data is the second data label, the second service platform stores the target service associated data. On the one hand, by obtaining the target data access platform, the target query request can be dispatched to the target data access platform rather than other initial data access platforms, preventing congestion on other initial data access platforms, which is beneficial to maintaining the normal operation of the system; on the other hand, the second data platform can prevent the leakage of personnel information in the second data platform by deleting data that cannot be retained according to the type of data tag, thereby enhancing data security.

Although some specific embodiments of the present invention have been described in detail by way of example, it will be appreciated by those skilled in the art that the above examples are provided for illustration only and are not intended to limit the scope of the present invention. It will also be appreciated by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims (9)

1. A distributed system for realizing high-availability service and low-retention data, characterized in that the system comprises: a first service platform, a plurality of second service platforms, a scheduling platform, a plurality of initial data access platforms, a processor and a memory storing a computer program, wherein the first service platform is communicatively connected with each of the second service platforms, each of the second service platforms is communicatively connected with the scheduling platform, the scheduling platform is communicatively connected with each initial data access platform, and each initial data access platform is communicatively connected with the first service platform, and when the computer program is executed by the processor, the following steps are implemented: S100, when the scheduling platform receives a target query request sent by any second service platform, it obtains a target data access platform from a plurality of initial data access platforms; S200, according to the target data access platform, the scheduling platform sends the target query request to the first service platform; S300: According to the target query request, the first service platform sends a target service associated data set corresponding to the target query request to the second service platform; S400, when the second service platform receives the target service associated data set, determining a data label corresponding to each target service associated data in the target service associated data set; S500: When the data tag corresponding to the target service-related data is the first data tag, the second service platform deletes the target service-related data; S600: When the data tag corresponding to the target service associated data is a second data tag, the second service platform stores the target service associated data. 2. According to claim 1, the distributed system for achieving high service availability and low data retention is characterized in that the first service platform is used to monitor each of the second service platforms. 3. The distributed system for realizing high-availability service and low-retention data according to claim 1, characterized in that the step S100 further comprises the following steps: S101, when the scheduling platform receives a target query request sent by any second service platform, obtains an access priority set E={E ₁ , E ₂ , ..., E _δ , ..., E _d } corresponding to the initial data access platform, where E _δ is the access priority corresponding to the δth initial data access platform, δ=1, 2...d, and d is the number of initial data access platforms; Among them, E _δ meets the following conditions: E _δ =1/Q _δ , where Q _δ is the number of visits received by the δth initial data access platform within one second before the current moment; S102: The scheduling platform obtains a target data access platform according to the access priority corresponding to each of the initial data access platforms. 4. According to claim 1, the distributed system for achieving high service availability and low data retention is characterized in that the first data tag refers to a data type that cannot be retained on the second service platform. 5. The distributed system for realizing high-availability service and low data retention according to claim 1 is characterized in that the second data tag refers to the type of data that can be retained on the second service platform. 6. The distributed system for realizing high-availability service and low-data retention according to claim 1, characterized in that it also includes the following steps: S01, according to a preset time interval, the first service platform detects the storage disk in the second service platform; S02: If the target service-related data is stored in the storage disk in the second service platform, the first service platform sends a second prompt message to the second service platform. 7. According to claim 1, the distributed system for achieving high service availability and low data retention is characterized in that the target query request refers to a request for querying a target service associated data set corresponding to a target user. 8. The distributed system for realizing high service availability and low data retention according to claim 1, characterized in that before step S100, the target service associated data set is obtained by the following steps: S001, when the first service platform receives a target service name of any second service platform, the first service platform sends initial service webpage information corresponding to the target service name to the second service platform; S002: When the first service platform receives the first service-related data set fed back by the second service platform, the first service platform generates a target service-related data set according to the first service-related data set. 9. The distributed system for realizing high service availability and low data retention according to claim 8, characterized in that, in S002, the target service associated data set is further generated by the following steps: S0021, when the first service platform receives the first feedback information from the second service platform, it queries the intermediate service associated data set from the preset user database of the first service platform according to the first feedback information, and sends the intermediate service associated data set to the second service platform; S0022, the first service platform queries a second service-related data set from a preset user database of the first service platform according to the intermediate service-related data set; S0023: When the first service platform receives the first service-related data set fed back by the second service platform, the first service-related data set and the second service-related data set are merged into a target service-related data set.