CN120561973B - Department data resource scheduling management method and system - Google Patents

Abstract

This invention relates to the field of data processing and management technology, and discloses a method and system for scheduling and managing departmental data resources. The method includes: data nodes actively broadcasting data reputation beacons containing their intrinsic reputation scores and authoritative update timestamps; data request terminals then perform preliminary sorting of the received beacons and, in conjunction with the results of real-time service health detection of authoritative candidate data sources, ultimately determine a unique authoritative data source and initiate a data request. This invention establishes a dynamic reputation competition and service capability verification mechanism, transforming the traditional administrative approval method in data scheduling into a collaborative method in which each data department actively maintains data quality and service stability to ensure its data is prioritized for adoption, thereby stimulating the intrinsic motivation for data sharing.

Description

Department data resource scheduling management method and system

Technical Field

The invention relates to a department data resource scheduling management method, and belongs to the technical field of data processing and management.

Background

In the current data intensive management and business activities, two data resource scheduling modes are generally adopted for realizing the integration and utilization of the data in the organization, one is to establish a data pipeline, collect the data scattered in each business department to a unified data center through extraction, conversion and loading processes, and the other is to provide the data outwards in the form of an application program interface by a data holding part for other departments to call after being authorized, wherein the two modes provide a basic framework for realizing the centralized management and control of the data and the orderly access.

When the service scenario puts higher demands on timeliness and acquisition flexibility of data, for example, the market department needs to acquire the latest transaction data of the sales department in real time to respond to market dynamics quickly, the scheduling mode generates flow delay in actual operation, because the operation mode is based on a fixed flow of first application, later authorization and then access, which necessarily requires communication coordination between a data demander and a data holder and execution of corresponding internal approval programs, and the interaction processes themselves need time, one direct coping thought is to increase the frequency of data extraction or expand the bandwidth of a data interface, but the mode increases the processing load and maintenance complexity of a central system, does not change the essence of central authorization, and the other thought is to relax the authority limit of data access, but directly conflicts with the data security protection responsibility borne by the data holder, so that under the prior art framework, a realistic operation tension exists between the demands of organizing flexible flow of data and the responsibility of the departments for data security management.

Specifically, the prior art has the following conditions that 1, the core of the existing scheduling mechanism is to verify the identity of a requester and plan an access path, the mechanism does not comprise a component for measuring the quality of data shared by a data provider, and lacks a corresponding link for feeding back the value of sharing behavior to the provider, and 2, under the mechanism, the provided data is constructed as a one-way task of a data holding department, the maintenance cost and the value of data contribution cannot be reflected in a system, and thus the active supply of high-quality and high-timeliness data is affected. Therefore, how to establish a data resource scheduling management method, so that the method is not dependent on a fixed application-approval process any more, but can dynamically judge the authority of data acquisition according to verifiable information issued by each data source, thereby forming a collaborative environment for promoting active data sharing.

Disclosure of Invention

The invention provides a department data resource scheduling management method, which mainly aims to solve the problems that the existing data scheduling mode cannot effectively promote active sharing of data in a mechanism due to the fixed application-approval process, so that the inter-department data cooperation process is long and the timeliness is insufficient.

In order to achieve the above object, the present invention provides a method for managing and scheduling department data resources, comprising the following steps:

Step a, generating and broadcasting a data reputation beacon to a network after each data node holding the same data item copy completes one-time authority updating, wherein the data reputation beacon comprises a unique identifier of the data item, a time stamp of the authority updating and an endogenous reputation point determined according to preset ownership level information;

step b, receiving a data credit beacon by a data demand terminal, and primarily sequencing a plurality of data nodes according to the endogenous credit points and the authoritative updated time stamps so as to determine one or more authoritative candidate data sources;

Step c, the data demand side terminal sends a query intention signaling to one or more authoritative candidate data sources, and receives a service health score returned by the authoritative candidate data sources according to the self operation load state;

step d, the data demand side terminal uniquely determines a final authoritative data source from the authority candidate data sources based on the preliminary sequencing and in combination with the service health degree score;

And e, the data demand terminal only initiates a data request to the final authoritative data source so as to acquire the data item.

Preferably, in the step d, the final authoritative data source is uniquely determined, specifically, the authoritative candidate data sources are sorted in a descending order according to the service health degree score, and the first sorted authoritative candidate data source is determined as the final authoritative data source.

Preferably, before the step b, the method further comprises the step that the data requiring party terminal counts the arrival time interval of the historical data credit beacons from each data node to calculate a dynamic reliability weight representing the broadcasting rhythm stability of the data node beacons, and in the step b, the preliminary ranking is performed based on a weighted credit score obtained by multiplying the internal credit score and the dynamic reliability weight, and the calculation of the weighted credit score is as follows: , wherein, In order to weight the reputation score,For the dynamic reliability weight to be a function of the dynamic reliability weight,Is an endogenous reputation score.

Preferably, the method further comprises the steps of setting a trigger for the service overrule operation in the downstream service application, generating and broadcasting a reputation instruction containing the overrule data source identification when the trigger is activated due to the service overrule operation, and carrying out temporary attenuation processing on the endogenous reputation integral of the overrule data source according to the received reputation instruction when the data requiring party terminal performs the preliminary sequencing in the step b.

Preferably, in step c, the query intention signaling is a pre-request detection signal containing a unique identifier of the data item, and the service health score is calculated by a health agent of the authoritative candidate data source according to at least one performance index of the actual monitored CPU load, the memory utilization rate and the current connection number of the database through a preset weighting formula.

Preferably, the method further comprises the steps that step a, at least one data demand side terminal receives the data reputation beacons and simultaneously carries out classified collection according to the business domain fields contained in the data reputation beacons, step b, the data demand side terminal counts the broadcasting frequency of the data reputation beacons of each business domain by taking a time window as a unit, compares the broadcasting frequency with a historical frequency base line of the business domain, and step c, generates and sends business trend early warning signals when the broadcasting frequency exceeds a preset range of the historical frequency base line.

Preferably, before the step b, the method further comprises an input end purifying step, wherein the data demand side terminal performs frequency domain feature analysis on the authoritative updated timestamp value streams in the received data credit beacons from different data nodes, and when the power spectrum density of the timestamp value stream of a certain data node is identified to be abnormal in a high-frequency region, the data demand side terminal performs normalization processing on the timestamp value of the data node before performing the preliminary sequencing of the step b.

The method comprises the steps of setting an emergency mode trigger for monitoring an emergency broadcast channel at a data demand terminal, suspending execution of steps b to e and switching to an emergency arbitration mode when the trigger receives emergency signaling containing event keywords and geofence information, selecting a data node with the geographic position closest to the geofence center as a final data source according to the geofence information in the emergency signaling and the real-time geographic position of each data node in the emergency arbitration mode, establishing a beacon cache by the data demand terminal for temporarily storing recently received data reputation beacons, and preferentially executing steps b to e in the beacon cache when the same data item is required to be scheduled again, wherein the network broadcast is monitored again only when the data reputation beacons in the beacon cache do not meet timeliness requirements.

The temporary credit score is preferably subjected to temporary attenuation treatment, and specifically comprises the steps that a data demand side terminal locally maintains a temporary credit score for each data source, when a credit instruction for a certain data source is received, the temporary credit score of the data source is reduced under the constraint of a preset attenuation coefficient and an attenuation upper limit, and the temporary credit score automatically rises along with time according to a preset recovery rule.

A department data resource scheduling management system, comprising:

A beacon receiving module configured to receive a data reputation beacon broadcast by a plurality of data nodes holding copies of the same data item, the data reputation beacon comprising a unique identification of the data item, a time stamp of the data node completing an authoritative update and an endogenous reputation score determined from preset ownership level information;

A preliminary arbitration module configured to preliminarily rank the plurality of data nodes according to the endogenous reputation points and the time stamps of the authoritative updates to determine one or more authoritative candidate data sources;

A health detection module configured to send a query intent signaling to one or more authoritative candidate data sources and to receive a service health score returned by the authoritative candidate data sources in accordance with their own operating load status;

A final arbitration module configured to uniquely determine a final authoritative data source from the authority candidate data sources based on the results of the preliminary ranking in combination with the service health score;

A data retrieval module is configured to initiate a data request only to the final authoritative data source to retrieve the data item.

Compared with the prior art, the invention has the beneficial effects that:

1. The invention changes the fixed mode of application-approval in the traditional data scheduling by constructing a data cooperation mode, in the invention, a data node actively broadcasts credit beacons containing ownership level and update time, and a demand party carries out localized authoritative source judgment according to the published beacons, so that data sharing is changed from a passive administrative task to competitive behaviors of actively maintaining data quality and updating timeliness for leading data of each data holder to be adopted preferentially, thereby exciting the internal power of cross-department data cooperation.

2. By combining the static ownership level and the dynamic update time of the data source and introducing detection of the real-time running state of the service node on the basis, a set of multi-level dynamic source searching decision mechanism is established, the problem of how to select the most authoritative party in the management sense from a plurality of plausible data sources is solved, and the risk of initiating an invalid request to an authoritative node with insufficient current service capability is avoided through the detection of the health degree of the pre-request, so that the whole process of data acquisition is ensured in terms of the accuracy of decision and the reliability of execution.

3. The constructed data scheduling management method has the capabilities of self-regulation and continuous optimization, and by introducing the consideration of the stability of the arrival rhythm of the beacons, the arbitration logic can sense and adapt to the change of the network environment and preferentially trust the more reliable data source of the transmission channel, and meanwhile, a set of distributed negative feedback loop is established by capturing the business overrule action of the downstream application and generating the reputation instruction, so that the real use experience of the data quality can reversely correct the reputation evaluation of the data source, thereby continuously inhibiting the diffusion of low-quality data under the condition of no human intervention and maintaining the health of the whole data ecology.

4. The method of the invention shows adaptability and toughness when dealing with unconventional and sudden conditions, and can suspend conventional arbitration logic based on reputation under the extreme scene of damage of a conventional communication network by setting an emergency mode trigger and coupling an emergency broadcast channel, automatically switch to an emergency decision mode taking geographical position proximity as priority, ensure that data with the most field significance can be acquired in time at key moment, and provide data support for emergency command.

Drawings

FIG. 1 is a schematic diagram of an operation flow of a method for managing the scheduling of department data resources according to the present invention;

FIG. 2 is a graph showing the results of a comparative test of the performance of the method of the present invention and the conventional method under different conditions;

FIG. 3 is a logic flow diagram of a scheduling decision and status switching for a data consumer terminal according to the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be described in detail with reference to the examples below, and it is apparent that the described embodiments are some, but not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without making any inventive effort based on the embodiments of the present invention are within the scope of the present invention.

The invention discloses a department data resource scheduling management method and a system, wherein the technical scheme is applied to the inside of a large organization, when a plurality of business departments all hold copies of the same data item, the data demand side needs to identify and acquire a situation of a unique authority version, the core operation flow of the method mainly comprises beacon generation and broadcasting steps executed by data nodes, dynamic arbitration and source searching steps executed by data demand side terminals and data acquisition steps, wherein the dynamic arbitration and source searching steps are configured into a multi-stage confirmation process comprising preliminary sequencing and health final selection, in a specific application scene, such as a large retail group, the market part is used for making marketing strategies, the recent high-value client consumption details generated by the sales part are needed to be acquired in time, the financial part is used for carrying out cost accounting, and also holds a copy data which is delayed by T+1, and therefore, the existing static authorization calling mode is difficult to simultaneously meet the requirements of timeliness of data acquisition, source authority and service availability, the method of the invention provides a multi-stage confirmation process comprising preliminary sequencing and health final selection, in a specific application scene, such as a new high-value client consumption details generated by the sales part is required to be acquired by the latest high-value client consumption details generated by the sales part, the latest high-value client consumption details generated by the sales part is required to carry out cost accounting, and the latest cost accounting is also held by a copy data which is delayed by a part, therefore, the authority data is only required to be acquired by the latest by the data based on the data which is based on the latest on the data of the data which has a new cost, the data with the latest authority item, the data which has a new authority data item, has been generated by the data with the authority data which has a new authority, the record is not random database writing time, but the data owner confirms that the updated content is complete and accurate, the record can be used as a standard specific moment and an endogenous reputation score, the score is preset by a data management committee according to all departments when the system is initialized, for example, the endogenous reputation score of a sales part is 90 for recent high-value customer consumption details, the endogenous reputation score of a financial part is 70, and a first re-judgment basis is provided for subsequent arbitration by quantizing the management hierarchy of data sources into a computable weight.

In the beacon broadcasting step, the data nodes of the sales part and the finance part are configured to broadcast the data reputation beacons generated respectively to a preset subnet by using a user datagram protocol, the broadcasting action can be realized by mounting light weight scripts on the update triggers of the database, so that the respective data states are announced to the network on the premise of not invading the core logic of the existing service system, the interaction mode from point-to-point request to many-to-many broadcasting forms the basis of the whole collaboration flow, furthermore, the preliminary sequencing phase of dynamic arbitration and source searching steps is performed on the data demand part, namely the terminal of the market part, in order to cope with the problem that the authority beacons possibly delay to arrive or lose due to network environment fluctuation and influence arbitration fairness, the method introduces a dynamic reliability weighting mechanism based on the beacon arrival rhythm, and the terminal of the market part is configured to continuously count the arrival time intervals of the historical data reputation beacons of the data nodes from the sales part, the market part and the like, calculates the jitter value of the beacon arrival interval of each data source by adopting an exponential weighted moving average algorithmAnd generates a dynamic reliability weight based thereonIts deterministic procedure is set to an inverse proportional function: Wherein For a sensitivity coefficient determinable by an off-line calibration experiment, the function ensures that the more unstable the beacon arrival rhythm is, the reliability weight isThe lower the market terminal, the more the static endogenous reputation score is used, the more the dynamic reliability weight is multiplied by the endogenous reputation score, the more the static endogenous reputation score is usedTo do this, the calculation of the weighted reputation score follows the deterministic formula: , wherein, Is an endogenous credit score, and the terminal is based on the credit scoreOrdering received beacons in descending order, whenIn the same time, the dynamic weight reflecting channel reliability is combined with static integral reflecting data source authority, so that the arbitration logic can compensate the uncertainty of network transmission layer, so as to select one or more authoritative candidate data sources with highest comprehensive credit under the current network environment, and at the same time, in order to cope with the challenges that a high credit data source can continuously produce legal but incorrect data to result in trust pollution due to system failure or human error, the method also provides a credit integral dynamic attenuation mechanism based on the implicit feedback of downstream application, in the downstream business application of BI tools used by analysts in market portion, the operation of overrule preset triggers such as overrule on the interface of the credit is carried out, when the analysts execute such overrule operation due to the discovery of data quality problem, the trigger is activated, then a credit instruction containing the overrule identification of the overrule data source is generated and broadcast, all data demand terminals in the network including the market portion can continuously produce legal but incorrect data to result in the challenge, after hearing the instruction, the temporary credit source is automatically attenuated according to the overrule of the temporary credit source, and the temporary integral is automatically distributed to the temporary integral, and the temporary integral is reduced according to the temporary integral attenuation coefficient is automatically reduced, enabling a true use experience of data quality to reverse correct reputation evaluations of a data source.

After finishing the preliminary sequencing, determine the authoritative candidate data source beginning with the sales department, the system faces a new problem that the server is in a service sub-health state possibly caused by high load at present, the terminal of the market department is configured to send a query intention signaling as a pre-request detection signal to all authoritative candidate data sources such as the sales department to avoid the risk, correspondingly, a parallel lightweight health agent is deployed on the data node of the sales department, the agent monitors at least one performance index such as the load of a central processor of the self node, the utilization rate of a memory, the current connection number of a database and the like in real time, instantly calculates a service health score according to a preset weighting formula, bypasses business logic after receiving the query intention signaling, and directly returns the score to the terminal of the market department, finally, the terminal of the market department determines the final authoritative data source based on the preliminary sequencing result formed in the steps and combining the received service health score, the final authoritative data source is determined, the certainty logic is that the authority data source is reduced according to the service health score, the current ranking is carried out on the candidate data source, the current ranking is carried out, the current ranking of the candidate data source is not required by the current data source, the final ranking is only can be provided with a unique decision-making mechanism, and the final ranking is carried out, the final ranking is only can be carried out by the final ranking data is a final ranking has a best is required by the final data has a best performance index, and is verified to the final ranking has a best quality has a best performance ranking value is obtained the final ranking end has a final priority, in order to improve the system efficiency, the market terminal is further configured to establish a beacon buffer for temporarily storing recently received data reputation beacons, and when the same data item needs to be scheduled again, the arbitration and source searching logic is preferentially executed in the buffer, and only when the beacons in the buffer do not meet the preset timeliness requirement, the network broadcast is monitored again.

In order to further expand the management supervision capability of the method, the system can be further configured to perceive service trends, an optional service domain field is added in a beacon structure, a data demand side terminal can collect beacons in a classified manner according to the service domains while receiving the beacons, and statistics is carried out on beacon broadcast frequencies of all the service domains by taking a fixed time window as a unit, when the frequency of a certain service domain deviates significantly from the historical baseline thereof, for example, by exceeding 3 standard deviations, the system judges that trend abnormality occurs and automatically generates a transmission service trend early warning signal, an input end purification step is provided for enhancing the robustness of the system in heterogeneous environments, a frequency domain purification gateway is deployed at the forefront end of the demand side terminal for receiving the beacons, the gateway carries out frequency domain feature analysis of accompanying mode on the time stamp numerical value flows which come from different data nodes and are updated in relation to the authority in the same data item beacon, the frequency domain feature analysis is carried out, the recognition logic is that a numerical value flow which is caused by a unit is not uniform as a second, the frequency of a certain service domain deviates significantly from the historical baseline, for example, the frequency of 3 standard deviation is exceeded, the system judges that the trend abnormality occurs, an emergency data has been provided for the emergency data has been set up in a critical mode, a critical mode is normally, a critical data channel has been set up by a priority mode is automatically, a priority is set up in a critical mode is triggered by an emergency data has been used for an emergency data has been normally, and has been set up, and has been normally has been used for a priority mode has been used, and has been used for a priority mode has been provided, has been used to a priority mode has been used to, when the trigger receives the emergency signaling containing the preset event keywords and the geofence information, the conventional arbitration logic based on the data reputation beacons is forcibly suspended and is switched to an emergency arbitration mode, in the mode, the decision logic of the terminal is converted into a real-time geographic position according to the geofence information in the emergency signaling and each data node, and the data node with the physical position closest to the geofence center is forcibly selected as a final data source.

Embodiment 1 in an operating Environment where an urban Emergency Command center handles sudden public health events, a command platform is used as a terminal of a data requiring party and has the task of plotting the geographical position of a newly added diagnosis case in real time to support resource scheduling and regional sealing control, an objective challenge in the scene is that the data item of the newly added diagnosis case position is simultaneously existed in two data nodes, an internal database of the urban health Command, the data of the internal database is reported by a medical institution and checked by an expert, the highest management authority is provided, but the update process has time delay and an on-line inspection system of the urban public security office, the data of the first line inspection system is recorded by a field duty personnel in real time, the timeliness is highest, but the two data nodes broadcast data beacons after authority updating are respectively finished according to regulations, when the emergency command enters a high-strength operating stage, network communication flow is increased, the path of each data node beacon arrives at the command platform to generate unequal jitter and delay, after the command platform hears from the Wei Jian Command the data beacons of the urban public security office are started, the dynamic source is continuously ordered according to a credit value of the initial jitter value of the command platform, and the credit value is continuously counted at the initial value of the command platformThe dynamic reliability weight is calculated for two data sources of the city Wei Jianwei and the city public security bureau respectivelyIn view of the more congested core network in which the city Wei Jian is commissioned, the beacon arrival rhythm stability is lower than that of the city public security bureau, thus obtaining a relatively lower valueSubsequently, command the platform to passWill reflect the channel reliabilityAnd reflecting authority of data sourcesCombining to calculate a weighted reputation score, although the city Wei Jian delegatedThe value is higher, but inAfter correction, itThe score is in the same order of magnitude as the local public office and the local public office's timestamp is updated, both of which are determined to be authoritative candidate data sources.

At this time, the system operation faces conflict of source authority and real-time service availability, although nodes of the city Wei Jian commission are authoritative, the server is in a high-load state because of the need of simultaneously responding to a large amount of internal queries, in order to cope with the situation, the command platform immediately enters a health degree final selection stage, it simultaneously sends a query intention signaling to two authoritative candidate data sources of the city Wei Jianwei and the public security bureau, the health degree agent of the city Wei Jian commission node returns a lower service health degree score because of monitoring that the load of the central processing unit of the agent exceeds an early warning threshold value, the health degree agent of the public security bureau node returns a higher score because of normal system load, and after receiving the two scores, the decision mechanism of the final source searching of the command platform is triggered, and the public security bureau node with the highest service health degree score is uniquely determined as a final authority data source, and a data request is only initiated to the public security bureau node; the electronic map of the command platform receives the position data from the public security bureau node and the latest suspected case, plots the position data in a high-brightness mode, provides immediate data input for the command center to make a routing control route of the next minute, actively avoids the data request of the client node of the city Wei Jian due to the sub-health state of the service, avoids the command decision delay possibly caused by overtime request, converts the data scheduling problem from the traditional application-approval-access fixed flow into a dynamic discovery process based on reputation competition and service capability verification, enables the prior priority ageing or priority authority selection to be processed under a unified arbitration logic, and in tens of minutes after the completion of the scheduling, when the expert review process of the city and defense line committee is completed and updates its database, a new data reputation beacon with a higher authority update timestamp is broadcast, and at the same time, as its server query peak has passed, its service health score also returns to normal level, in the next scheduling period of the command platform, the city Wei Jian committee node is selected as the final authority data source in the reputation competition, and the map data of the command platform is updated as the final position confirmed by medicine accordingly.

Embodiment 2 to quantitatively evaluate the effectiveness of the inventive department data resource scheduling management method, the following comparative experiment was designed and executed, which aims to measure and compare the performance of the inventive method with a conventional data scheduling method in terms of two core performance indexes of end-to-end delay of data acquisition and authority data source selection accuracy, the test platform was built in a virtualized environment consisting of three servers to simulate an organization information system comprising multiple data departments, wherein the servers one, two and three are respectively configured as data node A, data node B and data node C, the three servers are each deployed with the same business database and store copies of the same data item, and to simulate the differences of different departments in terms of data ownership level, the insights of which are integrated according to the rules of the specific embodimentPreset to be of non-equal value, node ASet to 90, representing the highest level of administrative authority, node BSet to 70, node CAnd an independent server is configured as a data demand terminal for initiating data requests and executing arbitration logic, the network connection between the servers is controlled by a network simulator which is programmable to inject preset network delays and data packet jitter, and the running load of the servers is applied by a load generating tool as required.

The test is provided with a test group and a comparison group, and the test group is completely deployed with the scheduling management method of the invention, and the method comprises broadcasting of data reputation beacons and weighting reputation-based componentsThe method comprises the steps of preliminary sequencing with authority update time stamps and final source searching based on service health scores, simulating an API gateway calling mode based on static priority by a comparison group, wherein the API gateway calling mode is configured to be prioritized according to a node A, try a node B again after failure or overtime, and finally try fixed logic of a node C to carry out data request, the test is respectively carried out under four different working conditions, each working condition is repeatedly executed 100 times and takes an average value as a final record, the working condition comprises a working condition I, a reference state, namely that each node server is normal in load and good in network state, a working condition II, a high-load state, namely that only the highest-authority node A is applied with an operation load which enables the CPU utilization rate to reach 95%, a working condition III, namely that 50ms average delay and 20ms jitter are injected into a network between nodes, and a compound pressure state, namely that high load is applied to the node A and high jitter is applied to the whole network simultaneously.

After the test is started, the test group and the control group select the node A as a data source under the working condition, the end-to-end time delay has no significant difference, when the test group is switched to the working condition II, the system response modes of the two groups show difference, the control group still tries to be connected with the node A at first, the time-out of 5000ms is triggered because the service of the node A is not responded, then the node B is turned to and the data is acquired, so that the end-to-end time delay of the node A is increased to 5125ms, the demand side terminal of the test group takes the node A as a candidate data source of the option, but receives the low service health grade returned by the node A in the subsequent health grade detection, so that the final source searching decision logic selects to initiate a request to the node B with higher grade, the access to the high load node is actively avoided, the time delay is only slightly increased to 120ms, and in the working condition III and the working condition IV, the dynamic reliability weight of the test group is increasedThe mechanism intervenes, but because the channel quality of each node has no obvious advantages or disadvantages, the test group can still make correct source searching judgment, the time delay of the test group changes along with the network condition, and the comparison group again causes the request timeout due to the static strategy under the composite pressure, and specific data are shown in the table 1.

Table 1. Comparison tables of the properties of each group under different conditions.

The test data show that under the high-load working condition of the node A, the low-delay performance of the test group is attributed to the health degree detection step executed before the request is initiated, and the node A with insufficient service capability at the time is identified and avoided through the service health degree scoring, so that the request timeout of the control group caused by a fixed priority strategy is avoided.

In this embodiment, in conjunction with fig. 1 to 3, a method and a system for managing a department data resource scheduling are described, as shown in fig. 1, the process starts from a plurality of data nodes holding data copies, such as data node a and data node B, to data node n, after authority updating is completed, data reputation beacons are broadcast respectively, a data demand terminal receives the beacons broadcast by each node through a monitoring network, before preliminary arbitration is performed, the terminal starts dynamic reliability weight calculation in parallel, this step calculates a dynamic reliability weight by counting the stability of the beacon arrival rhythm, so as to correct an endogenous reputation score, thereby compensating uncertainty in the network transmission process, and at the same time, a distributed negative feedback loop derived from a downstream service application also affects the arbitration process, that is, when the service operation is denied due to data quality problem, the system generates and broadcasts instructions, the reputation integral of the denied data source is temporarily attenuated, and then the preliminary authority updating time and dynamic weight of the nodes are synthesized, so as to determine one or more candidate data sources, on this step is further performed by counting the stability of the beacon arrival rhythm, the reliability score is calculated, the final result is obtained, and the final result is only obtained by combining the health score with the health score of the final request of the service, and the final result is obtained.

As shown in fig. 2, the horizontal axis of the graph is a test working condition, including four types of reference state, high load, high network jitter and composite pressure, the vertical axis is an end-to-end time delay represented by a logarithmic scale, the unit is ms, the graph includes two curves, wherein a test group represented by a solid dot represents a system adopting the method of the invention, and a comparison group represented by a dotted triangular dot represents a system adopting a traditional static priority strategy, the test result clearly shows that under the working condition of the reference state and the high network jitter, the time delay of the test group and the comparison group has no significant difference, however, under the working condition of the node high load and the composite pressure, the comparison group retries after initiating a request to the high load node and waiting for timeout, the end-to-end time delay of the comparison group rapidly rises to more than 5000ms, compared with the test group, the test group can actively evade the high load node with insufficient service capability through a health detection mechanism, the time delay of the comparison group only slightly increases, and is always maintained within 310ms, and the result verifies that the invention has the beneficial effect on guaranteeing the reliability and the performance level of data acquisition.

As shown in fig. 3, when a scheduling task is triggered, the system first checks the beacon cache, if the data reputation beacon in the cache is valid, then directly uses the cache data to perform final arbitration and source searching, if the cache is invalid, then enters a listening network broadcast state to wait and collect new data reputation beacons, after receiving the valid beacon, the system will perform preliminary arbitration and sorting, screen candidate sources by calculating weighted reputation, then, the system detects candidate source health, sends query intention signaling to it and obtains service health scores, then enters a final arbitration and source searching stage, combines all scores to uniquely determine final data sources, after determining unique authoritative data sources, the system initiates data acquisition requests to it to successfully acquire data items, in addition, the process also embeds an emergency arbitration mode, when the system receives emergency signaling, conventional logic is suspended, and data sources are determined according to geographical location and other emergency rules, and data acquisition is directly initiated to the emergency data sources.

Embodiment 4 in the application of the department data resource scheduling management method of the present invention deployed for the first time by a large financial institution, a system administrator needs to set a set of operation parameters matching with its specific network environment and business risk policies for three key mechanisms in the system, which are respectively a dynamic reliability weight mechanism for adjusting the influence of channel reliability, a reputation instruction mechanism for suppressing low-quality data diffusion, and an input-end purification mechanism for coping with metadata heterogeneity, and for setting parameters with reproducible data basis, the administrator performs a set of systematic offline calibration and configuration procedures, a calculation formula for dynamic reliability weightSensitivity coefficient of (a)The administrator collects all future beacon data broadcast by servers as data nodes from the production network of the organization over 24 hours of operation and calculates a baseline average of the beacon arrival interval jitter of each node during this periodThe network of the base line is then reproduced in a test environment by a network simulator and additionally injected with a jitter delta known to represent a moderate level of network degradationThe goal of the increment is to weight the dynamic reliability of a nodeTo a preset tolerance lower limit, here set to 0.9, at which time the coefficientBy solving an equation for the value of (2)Rather, it is determined that this approach anchors an abstract sensitivity coefficient on top of the objective physical characteristics of the financial institution's own network environment.

The method comprises the steps of carrying out grading setting on the attenuation coefficient of a temporary credit score according to the key degree of a business scene served by a data item, setting the attenuation coefficient of the data item directly influencing a core transaction decision to be a higher value, setting the attenuation coefficient of the data item to be a lower value for auxiliary analysis data item, uniformly setting the attenuation upper limit to be 50% of an endogenous credit score for preventing the continuous credit attack on a specific node from causing the complete interruption of the service, and determining the recovery rule of the credit score to be an exponential attenuation process, namely, automatically halving the attenuation value applied to the temporary credit score every 24 hours until the attenuation value is zero under the condition that no new credit instruction hits.

To determine the power spectral density threshold for identifying timestamp dimension anomalies in an input-side purification mechanism, an administrator extracts ten thousand consecutive known unit correct timestamp readings of each data node in a reference network state to form a standard sample set, performs fast Fourier transform on each timestamp value stream in the sample set to obtain the power spectral density thereof, calculates the ratio of the energy in a high-frequency region to the total energy, thereby obtaining a statistical distribution about the ratio, and finally, the threshold is determined as the mean value of the distribution plus four times of standard deviation, namelyAfter the calibration and configuration of all the parameters are completed, the department data resource scheduling management system is initialized, and all the key self-adaption and self-adjustment mechanisms in the department data resource scheduling management system obtain quantization rules matched with the running environment and management requirements and having data basis, and the system is immediately put into line to run.

In an embodiment 5, in a scenario where the method of the present invention is applied to a national critical infrastructure, such as a power grid dispatching center, in order to cope with a possible occurrence of an extreme communication interruption condition, a set of emergency mode triggering procedures based on communication heartbeat monitoring are configured in a terminal of a data demand party, the procedures are independent of monitoring an external emergency broadcast channel, the terminal sets a global beacon monitoring watchdog timer for all known data nodes, a time window of the global beacon monitoring watchdog timer is set to be a preset value larger than a longest normal broadcast interval, if the terminal fails to receive a valid data reputation beacon from any data node in the time window, the system determines that global communication interruption occurs, and under the communication interruption determination, the system automatically suspends all beacon monitoring and data requesting activities based on a conventional IP network and performs a data acquisition module for point-to-point communication with a national disaster preparation data center through a satellite link, and initiates a request for random critical dispatching data to the disaster preparation data center according to a preset emergency protocol, and the data security switching mechanism is used for establishing a data failure security switching mechanism under the extreme communication condition independent of external signaling.

To make the core parameters in the system generate credit pointsThe data management committee of the financial institution adopts a scoring rule based on multi-factor quantitative evaluation to determine the value of the data management committee, each data node with a copy of the data item is scored by a preset evaluation matrix aiming at each data item shared by cross departments, the evaluation dimension of the matrix comprises the distance between the data generation source and the business flow step number of the data generation event which is the most initial from the node, legal keeping responsibility, namely whether the department to which the node belongs is designated as the authority keeping party of the data item in law or industry regulation, and the role of the data life cycle, namely the role played by the node in the generation, processing, using and archiving of the data item, the score and the weight of each evaluation dimension are predefined in the data management rule, and the final endogenous credit integral of each nodeI.e., the sum of weighted scores in all evaluation dimensions, the procedure converts the score setting process into a standardized management flow that can be followed and audited with a seal.

Embodiment 6 in an application of the scheduling management method of the present invention for a national supply chain management platform, to ensure the operational robustness of the system and the environmental suitability of the decision model, a standardized set of pre-model building and online fault tolerance mechanism configuration procedures are performed before the system is online, the procedures are aimed at solving three specific engineering problems of quantification of the service health assessment model, optimization of traffic trend early warning threshold values and online filtering of abnormal beacon data, and to construct the service health assessment model, the system administrator firstly collects historical performance index data sets of the system administrator under different operational loads from a plurality of data node servers of the platform, wherein each set comprises a series of multiple groups, each group comprises real-time CPU load, memory usage rate, current connection number of the database and a service quality level corresponding to the data set and calibrated by actual request response time test, and the data sets are then used as a training set for multiple linear regression analysis, and analysis results thereof produce an assessment formula with a deterministic coefficient, such as a service health scoreWherein the constant isWeight coefficient of each performance index,,And all the results of the regression analysis determine that the process converts an abstract evaluation model into a quantization function which can be directly deployed based on the historical data of the target environment.

In order to configure a triggering threshold value of a business trend early warning mechanism, an administrator invokes a key business domain, such as a trunk logistics, beacon broadcast frequency historical data within one year, marks a frequency abnormal period caused by a real supply chain event by a business analyzer to form a time sequence with a true positive mark, traverses and loops the historical data by changing two judging parameters, namely standard deviation multiple and continuous time window number, calculates true positive rate and false positive rate of the historical data for each group of parameter combination, draws and analyzes a working characteristic curve of a subject, selects a parameter combination corresponding to a point closest to the upper left corner on the curve as the optimal configuration of the business domain early warning threshold value, and the procedure enables the balance between sensitivity and specificity of early warning to be established on the basis of carrying out statistical optimization on the historical business data.

In order to cope with the data credit beacon which may generate format damage or content abnormality caused by other system faults or interference in the network, a protocol compliance filter is deployed at the forefront end of a beacon receiving module of a data demand terminal, the filter performs a structured check and a value field check on each received data packet before sending the data packet to a subsequent arbitration logic, the structured check checks whether the field number and the data type of the data packet accord with a preset standard, the value field check judges whether the values of a time stamp, an internal credit score and the like fall within a valid interval meeting the logic, any data packet which fails to pass the two checks is directly discarded and recorded into an abnormal log, and the mechanism provides a pre-arranged compliance check link for the data input end of the system.

It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof.

Finally, it should be noted that the above-mentioned embodiments are merely for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made to the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention.

Claims (9)

1. A department data resource scheduling management method is characterized by comprising the following steps:

Step a, after each data node holding the same data item copy completes one business logic identification update, a data credit beacon is generated and broadcasted to the network, wherein the data credit beacon comprises a unique identifier of the data item, a time stamp of the business logic identification update and an endogenous credit score determined according to preset ownership level information;

Step b, receiving a data credit beacon by a data demand terminal, and primarily sequencing a plurality of data nodes according to the endogenous credit points and the time stamps of business logic identification update to determine one or more business logic identification candidate data sources;

Step c, the data demand side terminal sends a query intention signaling to one or more business logic identification candidate data sources and receives a service health grade returned by the business logic identification candidate data sources according to the self operation load state;

Step d, the data demand side terminal uniquely determines a final business logic identification data source from the business logic identification candidate data sources based on the preliminary sequencing and in combination with the service health degree score;

step e, the data demand terminal only initiates a data request to the final business logic affirming data source to acquire data items;

The method further comprises the steps of setting an emergency mode trigger for monitoring an emergency broadcast channel at the data demand side terminal, suspending execution of the steps b to e when the trigger receives emergency signaling containing event keywords and geofence information, switching to an emergency arbitration mode, selecting a data node with the geographic position closest to the geofence center as a final data source according to the geofence information in the emergency signaling and the real-time geographic position of each data node in the emergency arbitration mode, establishing a beacon cache by the data demand side terminal for temporarily storing recently received data reputation beacons, and preferentially executing the steps b to e in the beacon cache when the same data item is required to be scheduled again, and re-monitoring network broadcast only when the data reputation beacons in the beacon cache do not meet timeliness requirements.

2. The method for managing the scheduling of department data resources according to claim 1, wherein the final service logic identification data source is uniquely determined in the step d, specifically, the service logic identification candidate data sources are sorted in a descending order according to the service health score, and the first sorted service logic identification candidate data source is determined as the final service logic identification data source.

3. The method of claim 1, further comprising, prior to step b, the data consumer terminal counting time intervals of arrival of historical data reputation beacons from each data node to calculate a dynamic reliability weight characterizing the stability of the broadcast rhythms of the data node beacons, and wherein, in step b, the preliminary ranking is based on weighted reputation scores obtained by multiplying the endogenous reputation scores by the dynamic reliability weights, the weighted reputation scores are calculated as: , wherein, In order to weight the reputation score,For the dynamic reliability weight to be a function of the dynamic reliability weight,Is an endogenous reputation score.

4. The method of claim 1, further comprising setting a trigger for a traffic denial in the downstream traffic application, generating and broadcasting a reputation command including a denial data source identifier when the trigger is activated by the traffic denial, and temporarily attenuating an endogenous reputation score of the denial data source based on the received reputation command when the data requiring terminal performs the preliminary ranking of step b.

5. The method of claim 1 wherein in step c, the query intention signaling is a pre-request probe signal containing a unique identifier of the data item, the service health score is determined by the service logic as a health agent of the candidate data source, and the service health score is calculated by a preset weighting formula according to at least one performance index of the central processing unit load, the memory utilization rate and the current connection number of the database monitored by the service logic.

6. The method according to claim 1, further comprising the steps of a receiving the data reputation beacon by at least one data consumer terminal and classifying and collecting according to the service domain fields included in the data reputation beacon, b counting the broadcasting frequency of the data reputation beacon of each service domain by the data consumer terminal in units of time windows, comparing the broadcasting frequency with a historical frequency baseline of the service domain, and c generating and transmitting a service trend early warning signal when the broadcasting frequency exceeds a predetermined range of the historical frequency baseline.

7. The method for managing the scheduling of department data resources according to claim 1, further comprising an input purification step, before the step b, of performing frequency domain feature analysis on the timestamp value stream of the business logic identification update in the data reputation beacons received from different data nodes by the data requiring party terminal, and performing normalization processing on the timestamp value of a certain data node before performing the preliminary sequencing of the step b when the power spectrum density of the timestamp value stream of the certain data node is identified to be abnormal in a high frequency region.

8. The method for managing the scheduling of the department data resources according to claim 4, wherein temporary attenuation processing is performed on the endogenous reputation score, specifically, the data requiring party terminal maintains a temporary reputation score for each data source locally, when a reputation command for a certain data source is received, the temporary reputation score of the data source is reduced under the constraint of a preset attenuation coefficient and an attenuation upper limit, and the temporary reputation score automatically rises with time according to a preset recovery rule.

9. A department data resource scheduling management system for implementing the method of any one of claims 1-8, comprising:

A beacon receiving module configured to receive a data reputation beacon broadcast by a plurality of data nodes holding copies of the same data item, the data reputation beacon comprising a unique identification of the data item, a timestamp of completion of a business logic subscription update by the data node and an endogenous reputation score determined from preset ownership level information;

a preliminary arbitration module configured to perform preliminary ranking on the plurality of data nodes according to the endogenous reputation scores and the timestamp of the business logic identification update to determine one or more business logic identification candidate data sources;

a health detection module configured to send a query intent signaling to one or more business logic identification candidate data sources and to receive a service health score returned by the business logic identification candidate data sources in accordance with its operational load status;

A final arbitration module configured to uniquely determine a final business logic identification data source from the business logic identification candidate data sources based on the results of the preliminary ranking in combination with the service health score;

a data acquisition module configured to initiate a data request only to the final business logic to identify the data source to acquire the data item.