[go: up one dir, main page]

CN119783145B - Secure data transfer methods, apparatus, and equipment for data elements in heterogeneous data sources - Google Patents

Secure data transfer methods, apparatus, and equipment for data elements in heterogeneous data sources

Info

Publication number
CN119783145B
CN119783145B CN202411729538.6A CN202411729538A CN119783145B CN 119783145 B CN119783145 B CN 119783145B CN 202411729538 A CN202411729538 A CN 202411729538A CN 119783145 B CN119783145 B CN 119783145B
Authority
CN
China
Prior art keywords
data
data source
source
management center
sources
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202411729538.6A
Other languages
Chinese (zh)
Other versions
CN119783145A (en
Inventor
陆志鹏
韩光
符兴斌
郑曦
国丽
周崇毅
杨伟伟
郭红刚
董桂君
尚力
刘双
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Cec Digital Innovation Beijing Technology Co ltd
China Electronics Data Industry Group Co ltd
Original Assignee
Cec Digital Innovation Beijing Technology Co ltd
China Electronics Data Industry Group Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Cec Digital Innovation Beijing Technology Co ltd, China Electronics Data Industry Group Co ltd filed Critical Cec Digital Innovation Beijing Technology Co ltd
Priority to CN202411729538.6A priority Critical patent/CN119783145B/en
Publication of CN119783145A publication Critical patent/CN119783145A/en
Application granted granted Critical
Publication of CN119783145B publication Critical patent/CN119783145B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本申请提出了一种数据元件在异构数据源的安全数据转移方法、装置、设备,该方法包括:获取数据源间的连接信息和超管用户信息,并注册到管理中心;通过管理中心对数据源间的连接过程和对数据元件的生成过程进行管理;通过任务调度节点接收客户端的数据查询请求,并向管理中心发送该请求;管理中心查询并解析源数据源的元数据信息及目标数据源对应的相关信息,并将解析的数据传递到任务调度节点;任务调度节点注册节点任务,根据解析的数据执行节点任务进行数据加载;客户端通过数据查询请求获得任务调度节点的数据加载结果,并执行对应的用户任务,执行完成后获得计算结果并写出。本申实现了异构数据源之间的安全又高效的数据转移。

This application proposes a method, apparatus, and device for secure data transfer of data elements between heterogeneous data sources. The method includes: acquiring connection information and super-administrator user information between data sources and registering them with a management center; managing the connection process between data sources and the generation process of data elements through the management center; receiving data query requests from clients through a task scheduling node and sending the requests to the management center; the management center querying and parsing the metadata information of the source data source and the relevant information of the target data source, and transmitting the parsed data to the task scheduling node; the task scheduling node registering node tasks and executing node tasks to load data based on the parsed data; the client obtaining the data loading result from the task scheduling node through a data query request, executing the corresponding user task, obtaining the calculation result after execution, and writing it out. This application achieves secure and efficient data transfer between heterogeneous data sources.

Description

Secure data transfer method, device and equipment for data elements in heterogeneous data sources
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method, an apparatus, a device, and a medium for secure data transfer of a data element in a heterogeneous data source.
Background
With the development of information technology, a data-driven business environment becomes an important component of an enterprise. Due to the heterogeneous nature of the various data sources, the security, reliability and efficiency of data exchange and sharing between different data sources is of paramount importance.
In the related art, data transfer between heterogeneous data sources is generally performed through rights management, and in use, a service side implements rights control according to rights embodied by a user. Not only the management cost is increased, but also the risk of data leakage can be caused by improper authority setting.
Disclosure of Invention
The invention provides a secure data transfer method, device equipment and medium of a data element in a heterogeneous data source, which solve the problem of how to realize secure and efficient data transfer between heterogeneous data sources in the related technology.
In order to achieve the above purpose, the application adopts the following technical scheme:
in a first aspect, a method for secure data transfer of a data element in a heterogeneous data source is provided, including:
Based on the established connection relation between the data sources and the super management account, obtaining connection information between the data sources and super user information, and registering the data sources to a management center through corresponding JDBC;
managing a connection process between data sources through the management center;
managing the generation process of the data element through the management center;
Receiving a data query request initiated by a client through a pre-configured task scheduling node, and sending the data query request to the management center;
The management center queries and analyzes metadata information of a source data source and related information corresponding to a target data source according to the data query request, and transmits the analyzed data to the task scheduling node;
the task scheduling node registers a node task according to the data query request, and executes the node task according to the analyzed data to load the data;
the client obtains a data loading result of the task scheduling node through a data query request, and executes a corresponding user task according to the data loading result, and after the user task is executed, a calculation result is obtained and written out.
In a first possible implementation manner of the first aspect, the step of managing, by the management center, a connection procedure between data sources includes:
determining and acquiring roles of users according to the development, production and circulation processes of the data elements;
according to the role of the user, carrying out automatic authorization or refusal connection of the data source;
when the data source is on line, carrying out on-line recovery notification on an access request of a user during the off line;
The method comprises the steps of detecting the JDBC connection number of a data source, carrying out early warning when the connection number exceeds 80% of a preset threshold value, carrying out liveness sequencing on current connection users, sending a message notification of possible closing to the connection users with inactive TOP20%, and executing closing action on the connection users with inactive TOP20% when the connection number exceeds the threshold value, wherein the standard of the inactive connection users is set to be that no data transmission is detected within 10 minutes.
In a second possible implementation manner of the first aspect, the step of managing, by the management center, a generation process of the data element includes:
periodically scanning and maintaining metadata information of each data source, and recording scanning time;
when the data of the data source is increased, automatically synchronizing and updating metadata information of the data source;
The management center periodically scans heartbeat information of the data source to ensure that the data source is maintained in a usable state;
periodically scanning a registered data source list to ensure that new data sources can be discovered and used at any time;
Based on the virtual JDBC provided by the management center, the virtual JDBC is automatically routed to the JDBC corresponding to the data source according to the data resources required in the development, production and circulation processes of the data element and the metadata information of the data source.
In a third possible implementation manner of the first aspect, the step of transferring the parsed data to the task scheduling node includes:
the data is transferred by reflection or by the cooperation of a public buffer layer with fine grain authority.
In a second aspect, there is provided a secure data transfer device for a data element at a heterogeneous data source, comprising:
The data source registration module is used for acquiring connection information and super user information between the data sources based on the established connection relation between the data sources and the super management account, and registering the data sources to a management center through corresponding JDBC;
The data source connection management module is used for managing the connection process between the data sources through the management center;
the data element generation management module is used for managing the generation process of the data element through the management center;
the task scheduling node module is used for receiving a data query request initiated by a client through a pre-configured task scheduling node and sending the data query request to the management center;
the inquiring and analyzing module is used for inquiring and analyzing the metadata information of the source data source and the related information corresponding to the target data source according to the data inquiring request by the management center and transmitting the analyzed data to the task scheduling node;
The node data loading module is used for the task scheduling node to register a node task according to the data query request and execute the node task according to the analyzed data to load data;
and the user task execution module is used for the client to obtain the data loading result of the task scheduling node through the data query request, and execute the corresponding user task according to the data loading result, and the user task obtains the calculation result after the execution is completed and writes out the calculation result.
In a first possible implementation manner of the second aspect, the data source connection management module is specifically configured to:
determining and acquiring roles of users according to the development, production and circulation processes of the data elements;
according to the role of the user, carrying out automatic authorization or refusal connection of the data source;
when the data source is on line, carrying out on-line recovery notification on an access request of a user during the off line;
The method comprises the steps of detecting the JDBC connection number of a data source, carrying out early warning when the connection number exceeds 80% of a preset threshold value, carrying out liveness sequencing on current connection users, sending a message notification of possible closing to the connection users with inactive TOP20%, and executing closing action on the connection users with inactive TOP20% when the connection number exceeds the threshold value, wherein the standard of the inactive connection users is set to be that no data transmission is detected within 10 minutes.
In a second possible implementation manner of the second aspect, the data element generation management module is specifically configured to:
periodically scanning and maintaining metadata information of each data source, and recording scanning time;
when the data of the data source is increased, automatically synchronizing and updating metadata information of the data source;
The management center periodically scans heartbeat information of the data source to ensure that the data source is maintained in a usable state;
periodically scanning a registered data source list to ensure that new data sources can be discovered and used at any time;
Based on the virtual JDBC provided by the management center, the virtual JDBC is automatically routed to the JDBC corresponding to the data source according to the data resources required in the development, production and circulation processes of the data element and the metadata information of the data source.
In a third possible implementation manner of the second aspect, the query and parse module is specifically configured to:
the data is transferred by reflection or by the cooperation of a public buffer layer with fine grain authority.
In a third aspect, an electronic device is provided, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, which when executed by the processor, implements the steps of the secure data transfer method of data elements in heterogeneous data sources according to the first aspect.
In a fourth aspect, there is provided a readable storage medium having stored thereon a program or instructions which when executed by a processor performs the steps of the secure data transfer method of the data element in a heterogeneous data source according to the first aspect.
The beneficial effects are that:
The application avoids the limitation of complicated configuration information of the database depending on personnel management, realizes centralized automatic management of the configuration information, ensures that the configuration of the database is more consistent and standardized, is convenient for monitoring and maintenance, reduces human errors and improves the efficiency.
The application provides an elastic data abstraction form, data can be invisible, the risk of improper access or leakage is reduced by avoiding generating an intermediate process table, the safety of the data is protected, the requirement of data storage is reduced, and the efficiency of data processing is improved.
Drawings
FIG. 1 is a schematic flow chart of a secure data transfer method of a data element in a heterogeneous data source according to an embodiment of the present application;
FIG. 2 is a schematic flow chart of another method for secure data transfer of data elements in heterogeneous data sources according to an embodiment of the present application;
FIG. 3 is a block diagram of a secure data transfer device for data elements in heterogeneous data sources according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to further describe the technical means and effects adopted by the present application to achieve the predetermined purpose, the technical solutions in the embodiments of the present application are clearly described, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which are obtained by a person skilled in the art based on the embodiments of the present application, fall within the scope of protection of the present application.
The terms first, second and the like in the description and in the claims, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application are capable of operation in sequences other than those illustrated or otherwise described herein, and that the objects identified by "first," "second," etc. are generally of a type not limited to the number of objects, for example, the first object may be one or more. Furthermore, in the description and claims, "and/or" means at least one of the connected objects, and the character "/", generally means that the associated object is an "or" relationship.
The steps of the method flow described in the specification and the flow chart shown in the drawings of the specification are not necessarily strictly executed according to step numbers, and the execution order of the steps of the method may be changed. Moreover, some steps may be omitted, multiple steps may be combined into one step to be performed, and/or one step may be decomposed into multiple steps to be performed.
The following describes in detail the secure data transfer method, apparatus, device and medium of the data element in heterogeneous data source according to the embodiment of the present application with reference to the attached drawings and the preferred embodiments.
Firstly, an application scenario of the secure data transfer method of the data element in the heterogeneous data source according to the embodiment of the present application will be described in detail.
Different database management systems, file formats, network protocols, etc., result in the heterogeneity of different data sources.
In the data transfer process between heterogeneous data sources, an intermediate process table is often generated, which not only increases the complexity of data management, but also may cause the risk of data leakage. The existing data transfer method often ignores the security problem caused by an intermediate process table in the data migration process, and also comprises the form and form of the data elements and the security of data reading and writing of the data elements in heterogeneous data connection. The existence of these problems makes the data vulnerable to attack during transfer, increasing the risk of data leakage and tampering.
Aiming at the technical problems, the application provides a secure data transfer method of a data element in heterogeneous data sources, which can effectively manage connection information and metadata of a plurality of data sources, and can ensure the security and integrity of data in the data transfer process so as to meet the high requirements of modern enterprises on the security and reliability of the data.
Referring to fig. 1-2, an embodiment of the present application provides a secure data transfer method of a data element in a heterogeneous data source, where, as shown in fig. 1-2, the secure data transfer method of the embodiment of the present application includes the following steps:
Step S1, based on the established connection relation between the data sources and the super management account, obtaining connection information between the data sources and super user information, and registering the connection information and the super user information to a management center through the corresponding data sources.
And step S2, managing the connection process between the data sources through a management center.
In some possible embodiments, the managing the connection process includes:
Step S201, determining and acquiring roles of users according to the development, production and circulation processes of the data elements.
Step S202, according to the role of the user, automatic authorization of the data source or connection rejection are carried out.
And step 203, realizing the automatic on-line and off-line of the data source according to the health state of the data source, and carrying out on-line recovery notification on the access request of the user during the off-line when the data source is on-line.
Step S204, detecting the JDBC connection quantity of the data source, early warning when the connection quantity exceeds 80% of a preset threshold value, sorting the liveness of the current connection users, sending a message notice of possible closing to the connection users with the inactive TOP20%, and executing closing action to the connection users with the inactive TOP20% when the connection quantity exceeds the threshold value. Wherein the criteria for an inactive connected user is set such that no data transmission is detected within 10 minutes.
And step S3, managing the generation process of the data element through a management center.
In the generation process of the data elements, unified super account management realizes unified management of multiple data sources, metadata information of the data sources can be automatically scanned and updated through a management center, and virtual jdbc is directly authorized to access source data through the management center. So as to achieve the data timeliness and effectiveness of the element data generation process and the data source data elastic routing authorization.
In some possible embodiments, managing the generation process of the data element includes:
step S301, periodically scans and maintains metadata information of each data source, and records the scanning time.
Step S302, when the data of the data source is increased, the metadata information of the data source is automatically updated synchronously.
In step S303, the management center periodically scans the heartbeat information of the data source to ensure that the data source is maintained in a usable state.
Step S304, periodically scans the registered data source list to ensure that new data sources can be discovered and used at any time.
Step S305, based on the management center providing the virtual JDBC, the virtual JDBC is automatically routed to the JDBC of the corresponding data source according to the data resource and the metadata information of the data source required in the process of developing, producing and circulating the data element.
And S4, receiving a data query request initiated by the client through a pre-configured task scheduling node, and sending the data query request to a management center.
And S5, the management center queries and analyzes the metadata information of the source data source and the related information corresponding to the target data source according to the data query request, and transmits the analyzed data to the task scheduling node.
The data transmission process can be realized by transmitting the data in a reflection mode or by matching a public buffer layer with a fine grain authority.
And S6, the task scheduling node registers the node task according to the data query request, and executes the node task according to the analyzed data to load the data.
And S7, the client obtains a data loading result of the task scheduling node through the data query request, executes a corresponding user task according to the data loading result, and obtains and writes out a calculation result after the user task is executed.
In the above steps, the data element may be understood that the metadata management platform is isolated from the task scheduling and executing platform during the data transfer process of the heterogeneous data source. The data elements are enabled to receive data query requests initiated by the query clients through the set task scheduling nodes in the heterogeneous database system, and the metadata information of the source data sources and the related information of the target data sources are analyzed through the scheduling node request management center, so that the data elements are ensured not to store connection information of the data elements when the data elements are transferred on the scheduling platform, and the data elements are used immediately. The data element is on the whole task scheduling platform, and only the program knows the metadata information.
Thus, the existence of data elements between heterogeneous data sources can be understood as a flexible data abstraction that represents a collection of immutable, partitionable, parallel-computable elements within, when data conversion is performed. The elastomer is as follows:
(1) Fault tolerance elasticity, namely, the data loss can be automatically recovered;
(2) The elasticity of storage, namely automatic switching between the memory and the disk;
(3) Calculating elasticity, namely calculating an error retry mechanism;
(4) The elasticity of the segments can be re-segmented as needed.
In the data processing process of the data element, an intermediate process table is not generated, a user loads the data before using the data, and the user does not contact a data source, so that the reliability and the safety of the data element in the operation process are ensured.
Based on the technical scheme, the method and the device avoid the limitation that the complicated configuration information of the database depends on personnel management, realize centralized automatic management of the configuration information, enable the configuration of the database to be more consistent and standardized, facilitate monitoring and maintenance, reduce human errors and improve efficiency.
The application provides an elastic data abstraction form, data can be invisible, the risk of improper access or leakage is reduced by avoiding generating an intermediate process table, the safety of the data is protected, the requirement of data storage is reduced, and the efficiency of data processing is improved.
Referring to fig. 3, corresponding to the above embodiment of the secure data transfer method of the data element in the heterogeneous data source, the embodiment of the present application provides a secure data transfer device of the data element in the heterogeneous data source, where the secure data transfer device includes:
the data source registration module 1001 is configured to obtain connection information and super user information between data sources based on the established connection relationship between the data sources and the super management account, and register the connection information and the super user information to a management center through corresponding JDBC of the data sources;
a data source connection management module 1002, configured to manage a connection process between data sources through the management center;
a data element generation management module 1003, configured to manage a generation process of a data element by using the management center;
a task scheduling node module 1004, configured to receive, through a task scheduling node configured in advance, a data query request initiated by a client, and send the data query request to the management center;
the query and analysis module 1005 is configured to query and analyze metadata information of a source data source and related information corresponding to a target data source according to the data query request by the management center, and transmit the analyzed data to the task scheduling node;
the node data loading module 1006 is configured to register a node task according to the data query request by the task scheduling node, and execute the node task according to the parsed data to perform data loading;
The user task execution module 1007 is configured to obtain a data loading result of the task scheduling node by using the client through a data query request, execute a corresponding user task according to the data loading result, and obtain a calculation result and write out after the user task is executed.
Further, the data source connection management module is specifically configured to:
determining and acquiring roles of users according to the development, production and circulation processes of the data elements;
according to the role of the user, carrying out automatic authorization or refusal connection of the data source;
when the data source is on line, carrying out on-line recovery notification on an access request of a user during the off line;
The method comprises the steps of detecting the JDBC connection number of a data source, carrying out early warning when the connection number exceeds 80% of a preset threshold value, carrying out liveness sequencing on current connection users, sending a message notification of possible closing to the connection users with inactive TOP20%, and executing closing action on the connection users with inactive TOP20% when the connection number exceeds the threshold value, wherein the standard of the inactive connection users is set to be that no data transmission is detected within 10 minutes.
Further, the data element generation management module is specifically configured to:
periodically scanning and maintaining metadata information of each data source, and recording scanning time;
when the data of the data source is increased, automatically synchronizing and updating metadata information of the data source;
The management center periodically scans heartbeat information of the data source to ensure that the data source is maintained in a usable state;
periodically scanning a registered data source list to ensure that new data sources can be discovered and used at any time;
Based on the virtual JDBC provided by the management center, the virtual JDBC is automatically routed to the JDBC corresponding to the data source according to the data resources required in the development, production and circulation processes of the data element and the metadata information of the data source.
Further, the query and analysis module is specifically configured to:
the data is transferred by reflection or by the cooperation of a public buffer layer with fine grain authority.
The secure data transfer device of the data element in the heterogeneous data source realizes the steps of the secure data transfer method embodiment of the data element in the heterogeneous data source and the processes of the embodiments, and can achieve the same technical effects, so that repetition is avoided and redundant description is omitted.
Referring to fig. 4, corresponding to the embodiment of the secure data transfer method of the data element in the heterogeneous data source, the embodiment of the present application provides an electronic device, where the electronic device includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the computer program when executed by the processor implements the steps of the embodiment of the secure data transfer method of the data element in the heterogeneous data source and the processes of the embodiment, and can achieve the same technical effects, so that repetition is avoided and no further description is given here.
The memory 1009 may be used to store software programs as well as various data. The memory 1009 may mainly include a first memory area storing programs or instructions and a second memory area storing data, wherein the first memory area may store an operating system, application programs or instructions (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like. Further, the memory 1009 may include volatile memory or nonvolatile memory, or the memory 1009 may include both volatile and nonvolatile memory. The nonvolatile memory may be a Read-only memory (ROM), a programmable Read-only memory (ProgrammableROM, PROM), an erasable programmable Read-only memory (ErasablePROM, EPROM), an electrically erasable programmable Read-only memory (ElectricallyEPROM, EEPROM), or a flash memory, among others. The volatile memory may be random access memory (RandomAccessMemory, RAM), static random access memory (STATICRAM, SRAM), dynamic random access memory (DYNAMICRAM, DRAM), synchronous dynamic random access memory (SynchronousDRAM, SDRAM), double data rate synchronous dynamic random access memory (DoubleDataRateSDRAM, DDRSDRAM), enhanced synchronous dynamic random access memory (ENHANCEDSDRAM, ESDRAM), synchronous link dynamic random access memory (SYNCHLINKDRAM, SLDRAM), and direct memory bus random access memory (DirectRambusRAM, DRRAM). Memory 1009 in embodiments of the application includes, but is not limited to, these and any other suitable types of memory.
The processor 1010 may include one or more processing units, and optionally the processor 1010 integrates an application processor that primarily processes operations involving an operating system, user interface, application program, etc., and a modem processor that primarily processes wireless communication signals, such as a baseband processor. It will be appreciated that the modem processor described above may not be integrated into the processor 1010.
Corresponding to the embodiment of the secure data transfer method of the data element in the heterogeneous data source, the embodiment of the present application further provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, and the program or the instruction when executed by a processor implements the steps of the embodiment of the secure data transfer method of the data element in the heterogeneous data source and the processes of the embodiment, and can achieve the same technical effects, so that repetition is avoided and no further description is given here.
The processor is a processor in the electronic device described in the above embodiment of the present application. The readable storage medium includes computer readable storage medium such as computer readable memory ROM, random access memory RAM, magnetic or optical disk, etc.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing the functions in the order shown or discussed, but may also include performing the functions in a substantially simultaneous manner or in an opposite order depending on the functions involved, e.g., the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a computer software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present application.
The embodiments of the present application have been described above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those having ordinary skill in the art without departing from the spirit of the present application and the scope of the claims, which are to be protected by the present application.

Claims (8)

1.数据元件在异构数据源的安全数据转移方法,其特征在于,包括:1. A method for secure data transfer of data elements across heterogeneous data sources, characterized by comprising: 基于建立的数据源间的连接关系和超级管理账户,获取数据源间的连接信息和超管用户信息,并通过对应所述数据源自有的JDBC注册到管理中心;Based on the established connection relationships between data sources and the super administrator account, obtain the connection information between data sources and the super administrator user information, and register them with the management center through the JDBC of the corresponding data source. 通过所述管理中心对数据源之间的连接过程进行管理;The management center manages the connection process between data sources; 通过所述管理中心对数据元件的生成过程进行管理;The data element generation process is managed through the management center; 通过预先配置的任务调度节点接收客户端发起的数据查询请求,并向所述管理中心发送所述数据查询请求;The system receives data query requests initiated by clients through a pre-configured task scheduling node and sends the data query requests to the management center. 所述管理中心根据所述数据查询请求查询并解析源数据源的元数据信息及目标数据源对应的相关信息,并将解析的数据传递到所述任务调度节点;The management center queries and parses the metadata information of the source data source and the relevant information corresponding to the target data source according to the data query request, and transmits the parsed data to the task scheduling node; 所述任务调度节点根据所述数据查询请求注册节点任务,根据所述解析的数据执行所述节点任务进行数据加载;The task scheduling node registers node tasks according to the data query request, and executes the node tasks to load data according to the parsed data. 所述客户端通过数据查询请求获得所述任务调度节点的数据加载结果,并根据所述数据加载结果执行对应的用户任务,所述用户任务执行完成后获得计算结果并写出;The client obtains the data loading result of the task scheduling node through a data query request, and executes the corresponding user task according to the data loading result. After the user task is executed, the calculation result is obtained and written out. 所述的通过所述管理中心对数据源之间的连接过程进行管理步骤,包括:The steps for managing the connection process between data sources through the management center include: 根据所述数据元件开发、生产、流通过程,确定并获取用户的角色;Based on the development, production, and distribution process of the data elements, the user's role is determined and obtained; 根据所述用户的角色,进行所述数据源的自动授权或者拒绝连接;Based on the user's role, automatically authorize or deny the connection to the data source; 根据所述数据源的健康状态,实现数据源的自动上线、下线;Based on the health status of the data source, the data source can be automatically brought online and taken offline. 检测所述数据源的JDBC连接数量;Detect the number of JDBC connections to the data source; 所述的通过所述管理中心对数据元件的生成过程进行管理步骤,包括:The steps for managing the generation process of data elements through the management center include: 定期扫描并维护各个所述数据源的元数据信息,并对扫描的时间进行记录;Regularly scan and maintain the metadata information of each of the data sources, and record the scanning time; 当所述数据源数据增加时,自动同步更新所述数据源的元数据信息;When the data source data increases, the metadata information of the data source is automatically synchronized and updated; 所述管理中心定期扫描所述数据源的心跳信息,确保所述数据源维持在可用状态;The management center periodically scans the heartbeat information of the data source to ensure that the data source remains available. 定期扫描注册的数据源列表,确保新的所述数据源可以随时被发现和使用;Regularly scan the list of registered data sources to ensure that new data sources can be discovered and used at any time; 基于所述管理中心对外提供虚拟的JDBC,根据所述数据元件开发、生产、流通过程中需要的数据资源以及所述数据源的元数据信息,自动路由到对应所述数据源的JDBC。Based on the virtual JDBC provided by the management center, the system automatically routes data to the corresponding JDBC of the data source according to the data resources required during the development, production, and circulation of the data element and the metadata information of the data source. 2.根据权利要求1所述的数据元件在异构数据源的安全数据转移方法,其特征在于,2. The secure data transfer method for data elements in heterogeneous data sources according to claim 1, characterized in that, 当上线时,对下线期间用户的访问请求,进行上线恢复通知;When the system goes live, it will send a notification to users who requested access during the offline period to restore the system. 当所述连接数量超过预先设定的阈值的80%时进行预警;并对当前的连接用户进行活跃度排序,对TOP20%不活跃的所述连接用户发送可能关闭的消息通知;当所述连接数量超过所述阈值时,对当前TOP20%不活跃的所述连接用户执行关闭动作;其中,所述不活跃的连接用户的标准设定为:10分钟内没有检测到数据传输。An alert is issued when the number of connections exceeds 80% of a preset threshold; the current connected users are sorted by activity level, and a message notification that the connection may be closed is sent to the top 20% of inactive connected users; when the number of connections exceeds the threshold, the connection is closed for the top 20% of inactive connected users; wherein, the standard setting for the inactive connected users is: no data transmission has been detected within 10 minutes. 3.根据权利要求1所述的数据元件在异构数据源的安全数据转移方法,其特征在于,3. The secure data transfer method for data elements in heterogeneous data sources according to claim 1, characterized in that, 所述的将解析的数据传递到任务调度节点步骤,包括:The step of transmitting the parsed data to the task scheduling node includes: 通过反射的方式或通过公共缓存层配合使用细粒度权限的方式传递所述数据。The data can be transmitted via reflection or by using a public caching layer in conjunction with fine-grained permissions. 4.数据元件在异构数据源的安全数据转移装置,其特征在于,包括:4. A secure data transfer device for data elements in heterogeneous data sources, characterized in that it comprises: 数据源注册模块,用于基于建立的数据源间的连接关系和超级管理账户,获取数据源间的连接信息和超管用户信息,并通过对应所述数据源自有的JDBC注册到管理中心;The data source registration module is used to obtain connection information between data sources and super administrator user information based on the established connection relationship between data sources and the super administrator account, and register the data source with the management center through the JDBC of the data source. 数据源连接管理模块,用于通过所述管理中心对数据源之间的连接过程进行管理;The data source connection management module is used to manage the connection process between data sources through the management center; 数据元件生成管理模块,用于通过所述管理中心对数据元件的生成过程进行管理;The data element generation management module is used to manage the generation process of data elements through the management center; 任务调度节点模块,用于通过预先配置的任务调度节点接收客户端发起的数据查询请求,并向所述管理中心发送所述数据查询请求;The task scheduling node module is used to receive data query requests initiated by clients through pre-configured task scheduling nodes and send the data query requests to the management center. 查询与解析模块,用于所述管理中心根据所述数据查询请求查询并解析源数据源的元数据信息及目标数据源对应的相关信息,并将解析的数据传递到所述任务调度节点;The query and parsing module is used by the management center to query and parse the metadata information of the source data source and the relevant information corresponding to the target data source according to the data query request, and to transmit the parsed data to the task scheduling node; 节点数据加载模块,用于所述任务调度节点根据所述数据查询请求注册节点任务,根据所述解析的数据执行所述节点任务进行数据加载;A node data loading module is used for the task scheduling node to register a node task according to the data query request, and to execute the node task to load data according to the parsed data. 用户任务执行模块,用于所述客户端通过数据查询请求获得所述任务调度节点的数据加载结果,并根据所述数据加载结果执行对应的用户任务,所述用户任务执行完成后获得计算结果并写出;The user task execution module is used by the client to obtain the data loading result of the task scheduling node through a data query request, and to execute the corresponding user task according to the data loading result. After the user task is executed, the calculation result is obtained and written out. 所述数据源连接管理模块,具体用于:The data source connection management module is specifically used for: 根据所述数据元件开发、生产、流通过程,确定并获取用户的角色;Based on the development, production, and distribution process of the data elements, the user's role is determined and obtained; 根据所述用户的角色,进行所述数据源的自动授权或者拒绝连接;Based on the user's role, automatically authorize or deny the connection to the data source; 根据所述数据源的健康状态,实现数据源的自动上线、下线;Based on the health status of the data source, the data source can be automatically brought online and taken offline. 检测所述数据源的JDBC连接数量;Detect the number of JDBC connections to the data source; 所述数据元件生成管理模块,具体用于:The data element generation and management module is specifically used for: 定期扫描并维护各个所述数据源的元数据信息,并对扫描的时间进行记录;Regularly scan and maintain the metadata information of each of the data sources, and record the scanning time; 当所述数据源数据增加时,自动同步更新所述数据源的元数据信息;When the data source data increases, the metadata information of the data source is automatically synchronized and updated; 所述管理中心定期扫描所述数据源的心跳信息,确保所述数据源维持在可用状态;The management center periodically scans the heartbeat information of the data source to ensure that the data source remains available. 定期扫描注册的数据源列表,确保新的所述数据源可以随时被发现和使用;Regularly scan the list of registered data sources to ensure that new data sources can be discovered and used at any time; 基于所述管理中心对外提供虚拟的JDBC,根据所述数据元件开发、生产、流通过程中需要的数据资源以及所述数据源的元数据信息,自动路由到对应所述数据源的JDBC。Based on the virtual JDBC provided by the management center, the system automatically routes data to the corresponding JDBC of the data source according to the data resources required during the development, production, and circulation of the data element and the metadata information of the data source. 5.根据权利要求4所述的数据元件在异构数据源的安全数据转移装置,其特征在于,5. The secure data transfer device for data elements in heterogeneous data sources according to claim 4, characterized in that, 所述数据源连接管理模块,具体还用于:The data source connection management module is further used for: 当上线时,对下线期间用户的访问请求,进行上线恢复通知;When the system goes live, it will send a notification to users who requested access during the offline period to restore the system. 当所述连接数量超过预先设定的阈值的80%时进行预警;并对当前的连接用户进行活跃度排序,对TOP20%不活跃的所述连接用户发送可能关闭的消息通知;当所述连接数量超过所述阈值时,对当前TOP20%不活跃的所述连接用户执行关闭动作;其中,所述不活跃的连接用户的标准设定为:10分钟内没有检测到数据传输。An alert is issued when the number of connections exceeds 80% of a preset threshold; the current connected users are sorted by activity level, and a message notification that the connection may be closed is sent to the top 20% of inactive connected users; when the number of connections exceeds the threshold, the connection is closed for the top 20% of inactive connected users; wherein, the standard setting for the inactive connected users is: no data transmission has been detected within 10 minutes. 6.根据权利要求4所述的数据元件在异构数据源的安全数据转移装置,其特征在于,6. The secure data transfer device for data elements in heterogeneous data sources according to claim 4, characterized in that, 所述查询与解析模块,具体用于:The query and parsing module is specifically used for: 通过反射的方式或通过公共缓存层配合使用细粒度权限的方式传递所述数据。The data can be transmitted via reflection or by using a public caching layer in conjunction with fine-grained permissions. 7.一种电子设备,其特征在于,所述电子设备包括:存储器、处理器及存储在所述存储器上并可在所述处理器上运行的计算机程序,所述计算机程序被所述处理器执行时实现如权利要求1至3中任一项所述数据元件在异构数据源的安全数据转移方法的步骤。7. An electronic device, characterized in that the electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the secure data transfer method for data elements in heterogeneous data sources as described in any one of claims 1 to 3. 8.一种可读存储介质,其特征在于,所述可读存储介质上存储有程序或指令,所述程序或指令被处理器执行时实现如权利要求1至3中任一项所述数据元件在异构数据源的安全数据转移方法的步骤。8. A readable storage medium, characterized in that the readable storage medium stores a program or instructions, which, when executed by a processor, implement the steps of the secure data transfer method for data elements in a heterogeneous data source as described in any one of claims 1 to 3.
CN202411729538.6A 2024-11-29 2024-11-29 Secure data transfer methods, apparatus, and equipment for data elements in heterogeneous data sources Active CN119783145B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202411729538.6A CN119783145B (en) 2024-11-29 2024-11-29 Secure data transfer methods, apparatus, and equipment for data elements in heterogeneous data sources

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202411729538.6A CN119783145B (en) 2024-11-29 2024-11-29 Secure data transfer methods, apparatus, and equipment for data elements in heterogeneous data sources

Publications (2)

Publication Number Publication Date
CN119783145A CN119783145A (en) 2025-04-08
CN119783145B true CN119783145B (en) 2025-11-21

Family

ID=95231283

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202411729538.6A Active CN119783145B (en) 2024-11-29 2024-11-29 Secure data transfer methods, apparatus, and equipment for data elements in heterogeneous data sources

Country Status (1)

Country Link
CN (1) CN119783145B (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113515499A (en) * 2021-03-25 2021-10-19 中国雄安集团数字城市科技有限公司 Database service method and system
CN113641868A (en) * 2021-08-19 2021-11-12 上海淇玥信息技术有限公司 Data access control method and device for resource guarantee investment and electronic equipment

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030154284A1 (en) * 2000-05-31 2003-08-14 James Bernardin Distributed data propagator
US10516672B2 (en) * 2016-08-05 2019-12-24 Oracle International Corporation Service discovery for a multi-tenant identity and data security management cloud service

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113515499A (en) * 2021-03-25 2021-10-19 中国雄安集团数字城市科技有限公司 Database service method and system
CN113641868A (en) * 2021-08-19 2021-11-12 上海淇玥信息技术有限公司 Data access control method and device for resource guarantee investment and electronic equipment

Also Published As

Publication number Publication date
CN119783145A (en) 2025-04-08

Similar Documents

Publication Publication Date Title
CN107491343B (en) Cross-cluster resource scheduling system based on cloud computing
US8825502B2 (en) System and method for providing patient record synchronization in a healthcare setting
US9973582B2 (en) Method and apparatus for bi-directional communication and data replication between multiple locations during intermittent connectivity
CN112380072B (en) Multi-data center access method and system
CN118631889B (en) Distributed ERP platform portal reconstruction access method and system
CN110691133A (en) Web service mimicry system and method applied to network communication equipment
JP2023541298A (en) Transaction processing methods, systems, devices, equipment, and programs
US10795864B1 (en) Method and apparatus for bi-directional communication and data replication between local and remote databases during intermittent connectivity
US20100043010A1 (en) Data processing method, cluster system, and data processing program
US20120317438A1 (en) Method and system for providing immunity to computers
CN114448686A (en) Cross-network communication device and method based on micro-service
US20070198554A1 (en) Apparatus for business service oriented management infrastructure
CN108965054A (en) A kind of client and service end data rapid interactive method
CN119783145B (en) Secure data transfer methods, apparatus, and equipment for data elements in heterogeneous data sources
EP3416354B1 (en) Method and apparatus for bi-directional communication and data replication between multiple locations during intermittent connectivity
CN103188065A (en) Method and system of data synchronism in business service
CN111813607B (en) Database cluster recovery log processing system based on memory fusion
US20080178182A1 (en) Work state returning apparatus, work state returning method, and computer product
CN114285865A (en) Access control system for shared cloud drives
CN116107801B (en) Transaction processing methods and related products
CN111277499A (en) Gateway-based real-time effective dynamic routing method
CN113778709B (en) Interface calling method, device, server and storage medium
CN112673353B (en) Information processing system and control method of information processing system
CN116579019A (en) A Computer Information Security Supervision System Based on Artificial Intelligence
KR102202645B1 (en) Data Sharing Method for Relational Edge Servers

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant