[go: up one dir, main page]

CN116610737A - Datagin data synchronization method, device, equipment and medium - Google Patents

Datagin data synchronization method, device, equipment and medium Download PDF

Info

Publication number
CN116610737A
CN116610737A CN202310615918.6A CN202310615918A CN116610737A CN 116610737 A CN116610737 A CN 116610737A CN 202310615918 A CN202310615918 A CN 202310615918A CN 116610737 A CN116610737 A CN 116610737A
Authority
CN
China
Prior art keywords
data
file
object storage
storage service
dataphin
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310615918.6A
Other languages
Chinese (zh)
Inventor
陈烨婷
付晓飞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Shuhe Information Technology Co Ltd
Original Assignee
Shanghai Shuhe Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Shuhe Information Technology Co Ltd filed Critical Shanghai Shuhe Information Technology Co Ltd
Priority to CN202310615918.6A priority Critical patent/CN116610737A/en
Publication of CN116610737A publication Critical patent/CN116610737A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • G06F16/254Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • G06F16/2282Tablespace storage structures; Management thereof
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application discloses a data synchronization method, a device, equipment and a medium, and relates to the technical field of data processing. The method comprises the following steps: the client side performs data extraction and data processing on a database corresponding to the cloud server side through a Dataphin channel to generate a Dataphin file; the client maps the Dataphin file to an object storage service file in an outward appearance mode; the cloud server responds to detection that the client completes mapping of the object storage service file, and reads the object storage service file; and the cloud service end maps the data in the object storage service file into Java fields so that Java items can directly use corresponding Datagin data according to the object storage service file mapped into the Java fields. The application can enable the service system to directly access the DP data, thereby reducing the coupling and improving the coding efficiency of the service efficiency.

Description

Datagin data synchronization method, device, equipment and medium
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data synchronization method, apparatus, device, and medium.
Background
The Datagin (intelligent data construction and management) is used as a commercial cloud data development tool, can be used for constructing, managing and applying requirements for big data in various industries, and can be combined with a data middle platform methodology to provide the capabilities of global data integration, visual modeling, standard definition, data asset management and the like in one-stop mode, so that a data system with unified standard, accuracy, reliability, convenience and expense is built for power-assisted enterprises.
The Datagin has full-link data management capability, and can remarkably improve the data management level by the aid of functional modules such as data standard management, data quality management, data security, resource management and the like.
However, the data generated by the DP is processed through several bins, and then is synchronized to the service system through ETL (Extract-Transform-Load) for use and then is processed separately, so that the service system cannot directly access the data generated by the DP at present, and further needs to perform operations such as database I/O. Therefore, a data synchronization method for Dataphin is needed at present, which can enable the service system to directly access DP data, thereby reducing the coupling and improving the coding efficiency of the service efficiency.
Disclosure of Invention
In order to solve at least one of the problems mentioned in the background art, the present application provides a data synchronization method, device, equipment and medium, which can enable a service system to directly access DP data, thereby reducing the coupling and improving the coding efficiency of service efficiency.
The specific technical scheme provided by the embodiment of the application is as follows:
in a first aspect, a data synchronization method is provided, applied to a client, and the method includes:
executing data extraction and data processing on a database corresponding to the cloud server through a Dataphin channel to generate a Dataphin file;
mapping the Dataphin file to an object storage service file in an external mode.
Further, before the mapping the Dataphin file into the object storage service file by way of appearance, the method further includes:
configuring configuration parameters of an object storage service source corresponding to the Datagin file;
the configuration parameters include an access domain name, a storage space, an access key identification number, and an access key user key of the object storage service source.
In a second aspect, a data synchronization method is provided, and is applied to a cloud server, where the method includes:
in response to detecting that the client completes mapping of the object storage service file, performing reading on the object storage service file;
and mapping the data in the object storage service file into a Java field so that the Java item can directly use the corresponding Datagin data according to the object storage service file mapped into the Java field.
Further, the mapping the data in the object storage service file into a Java field includes:
designating a project space, a schema and a table name of the object storage service file corresponding to the Dataphin file;
and designating the mapping relation between the fields in the table and Java attributes in the corresponding Dataphin file of the object storage service file so as to inject the fields in the table into the corresponding Java objects through field reflection according to the mapping relation.
Further, the method further comprises:
designating the mapping relation between the Java attribute and the MySQL attribute;
wherein the MySQL attribute comprises at least one of a list field and a type of MySQL table.
In a third aspect, a data synchronization method is provided, and the method includes:
the client side performs data extraction and data processing on a database corresponding to the cloud server side through a Dataphin channel to generate a Dataphin file;
the client maps the Dataphin file to an object storage service file in an outward appearance mode;
the cloud server responds to detection that the client completes mapping of the object storage service file, and reads the object storage service file;
and the cloud service end maps the data in the object storage service file into Java fields so that Java items can directly use corresponding Datagin data according to the object storage service file mapped into the Java fields.
In a fourth aspect, a datachin data synchronization device is provided, applied to a client, and the device includes:
the data processing module is used for executing data extraction and data processing on the database corresponding to the cloud server through the Datagin channel to generate a Datagin file;
and the data synchronization module is used for mapping the Dataphin file into the object storage service file in an outward appearance mode.
In a fifth aspect, a data synchronization device is provided, and the device is applied to a cloud service end, and includes:
the reading module is used for responding to the fact that the client side completes the mapping of the object storage service file, and reading is carried out on the object storage service file;
and the mapping module is used for mapping the data in the object storage service file into Java fields so that the Java items can directly use the corresponding Datagin data according to the object storage service file mapped into the Java fields.
In a sixth aspect, an electronic device is provided that includes a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the datachin data synchronization method when executing the computer program.
In a seventh aspect, a computer readable storage medium is provided, storing computer executable instructions for performing the Dataphin data synchronization method.
The embodiment of the application has the following beneficial effects:
according to the data synchronization method, device, equipment and medium provided by the embodiment of the application, the client side firstly performs data extraction through the data channel of the Datagin to extract the business data of the cloud service end into the Datagin, and maps the business data into the OSS file in an appearance mode after the data processing is performed through the Datagin, so that a business system can directly use corresponding Datagin data in a memory without storing the table, and I/O operation of a database is not needed, thereby reducing the coupling and improving the business efficiency coding efficiency. In addition, the data output by the Datagin is changed, and the data corresponding to the OSS file is changed, so that the effect of data synchronization is achieved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 shows a general flowchart of a Dataphin data synchronization method applied to a client according to an embodiment of the present application;
fig. 2 shows a general flowchart of a Dataphin data synchronization method applied to a cloud server according to an embodiment of the present application;
FIG. 3 is a general flow chart of a Datagin data synchronization method provided by an embodiment of the present application;
fig. 4 shows a schematic structural diagram of a Dataphin data synchronization device applied to a client according to an embodiment of the present application;
fig. 5 shows a schematic structural diagram of a Dataphin data synchronization device applied to a cloud server according to an embodiment of the present application;
FIG. 6 illustrates an exemplary system that may be used to implement various embodiments described in the present application.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
It should be understood that throughout this specification and the claims, unless the context clearly requires otherwise, the words "comprise", "comprising", and the like, are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is, it is the meaning of "including but not limited to".
It should be noted that the terms "S1", "S2", and the like are used for the purpose of describing the steps only, and are not intended to be construed to be specific as to the order or sequence of steps, nor are they intended to limit the present application, which is merely used to facilitate the description of the method of the present application, and are not to be construed as indicating the sequence of steps. In addition, the technical solutions of the embodiments may be combined with each other, but it is necessary to base that the technical solutions can be realized by those skilled in the art, and when the technical solutions are contradictory or cannot be realized, the combination of the technical solutions should be considered to be absent and not within the scope of protection claimed in the present application.
Example 1
The application provides a data synchronization method, which is applied to a client, and referring to FIG. 1, the method comprises the following steps:
101. executing data extraction and data processing on a database corresponding to the cloud server through a Dataphin channel to generate a Dataphin file;
102. the Dataphin file is mapped to the object storage service file by way of an external table.
Specifically, the Datagin can be used for building, managing and applying requirements on big data in various industries, and combining with a data middle platform methodology, the data management system can provide the capabilities of global data integration, visual modeling, standard definition, data asset management and the like in one-stop mode, and assist enterprises to build a data system which is unified in standard, accurate, reliable, convenient and consumable. The Datagin has full-link data management capability, and the data management level is remarkably improved by a power-assisted enterprise through functional modules such as data standard management, data quality management, data security, resource management and the like. The Dataphin channel is deployed in the client, and can perform data extraction on a database corresponding to the cloud server system, perform data processing and data assembly on the extracted data, so as to generate a Dataphin file, namely a DP file, but the DP file service system cannot directly use the Dataphin data therein, and then needs to map into an Object storage service (Object StorageService, OSS) file in an external manner, so that a subsequent service system can directly use the corresponding Dataphin data through an oss file without performing an I/O (Input/Output) operation of the database. Illustratively, the Dataphin channel has a computing task that executes SQL (structured query language) that cleans the data, and generates Dataphin files by parsing the data output by SQL, which can be used to map into OSS files. It should be noted that the data output by the Datagin through SQL changes, and the data corresponding to the OSS file also changes, so that the effect of data synchronization is achieved. In particular, the Dataphin data synchronization method can be applied to various cloud server systems and corresponding various database systems, including but not limited to relational databases, non-relational databases, key value databases and the like. The data source will be synchronized into the Dataphin system with a D-1 time partition. The application is exemplified by MySQL as the data source.
Specifically, SQL is a database language with multiple functions such as data manipulation and data definition, and the language has the characteristic of interactivity, so that great convenience can be provided for users, and the database management system can fully utilize the SQL language to improve the working quality and efficiency of a computer application system. The SQL language can be independently applied to the terminal, and can also be used as a sub-language to provide effective assistance for other programming.
In some embodiments, prior to step 102, the method further comprises:
100. configuring configuration parameters of an object storage service source corresponding to the Datagin file;
the configuration parameters comprise an access domain name, a storage space, an access key identity number and an access key user key of the object storage service source.
Specifically, mapping the Dataphin file to an OSS file (i.e., the object storage service file described above) requires specifying the corresponding OSS server, key, and file path. Illustratively, OSS Object StorageService, the object storage service described above, is a tool for storing and retrieving unstructured data and metadata objects using HTTP (HyperTe xtTransferProtocol ) API (ApplicationProgrammi ngInterface, application program interface). The data synchronization refers to the file generated by the Datagin to the OSS object storage service file, the data is stored to the hard disk before, and is stored to the object storage service OSS, which is equivalent to pointing to an OSS data source, namely the object storage service source. Parameters need to be configured for the OSS data source. The parameters of the object store service OSS relate to an Endpoint, a socket, and an access key. Wherein a storage space socket is a container for storing objects (objects), all of which have to be affiliated to the storage space. An Object (Object) is a basic unit of OSS storage Data, and is composed of meta information (ObjectMeta), user Data (Data), and a file name (Key). The object is identified by a unique Key within the memory space. Endpoint represents the domain name that the OSS accesses to an external service. OSS provides services externally in the form of httprestful api, requiring different domain names when accessing different regions (regions). The domain names required to access the same region through the intranet and extranet are also different. For details, please refer to the Endpoint corresponding to each Region. Wherein the Region (Region) represents the physical location of the data center of the OSS. The AccessKey refers to an AccessKeyId and an AccessKeySecret used in the access identity authentication, namely the above-mentioned access key identity identification number and the access key user key. The OSS object store service can verify the sender identity of a certain request by using an AccessKeyI d and AccessKeySecret symmetric encryption method. The AccessKeyId is used to identify the user, and the AccessKeySecret is the key that the user uses to encrypt the signature string and the OSS uses to verify the signature string, where the AccessKeySecret must be kept secret.
In this embodiment, the client first performs data extraction through the Dataphin channel to extract the service data of the cloud service end into the Dataphin, and maps the data into the OSS file in an external manner after performing data processing through the Dataphin, so that a subsequent service system can directly use the corresponding Dataphin data through the OSS file without performing an I/O operation of a database, thereby reducing the coupling and improving the service efficiency coding efficiency.
The application also provides a data synchronization method applied to the cloud service end, referring to fig. 2, the method comprises the following steps:
201. in response to detecting that the client completes mapping of the object storage service file, performing reading of the object storage service file;
202. and mapping the data in the object storage service file into a Java field so that the Java item can directly use the corresponding Datagin data according to the object storage service file mapped into the Java field.
Specifically, after the task execution of the Dataphin channel is successfully scheduled, that is, the client has completed mapping of the OSS file of the object storage service, a monitoring event is sent to the cloud server system, and the cloud server system executes task scheduling of reading the OSS file after receiving the notification; the data in the object store service OSS file is then mapped into Java fields so that the corresponding Dataphin data can be used directly in the Java project. By adopting the technical means, the service system can directly take the data in the memory without storing the table, the coupling is low, the service development can be facilitated, and the coding and service efficiency can be improved.
In some embodiments, step 201 comprises:
211. designating a project space, a schema and a table name of the object storage service file corresponding to the Dataphin file;
212. and designating the mapping relation between the fields in the table and Java attributes in the corresponding Dataphin file of the object storage service file so as to inject the fields in the table into the corresponding Java objects through field reflection according to the mapping relation.
Specifically, the mapping relationship between the Dataphin file and the Java attribute can be realized by using the Java reflection and the annotation. Firstly, connecting to a designated OSS server through the OSS object storage service parameters Endpoint, bucket, accessKeyId and the Access Key secret, and matching OSS files through a designated path so as to match the corresponding Datagin files; then, by designating the mapping relation between the Datagin file and the Java attribute, the service system can inject the fields of the table into the corresponding Java objects through field reflection at the cloud server according to the mapping relation, so that the Datagin data can be directly used in Java items.
In some embodiments, the method further comprises:
203. designating a mapping relation between Java attributes and MySQL attributes;
wherein the MySQL attribute includes at least one of a list field and a type of MySQL table.
Specifically, the list field and the type of the MySQL table can be specified through Mybat framework integration, so that the Java class and the MySQL form a corresponding mapping relation, the mapping relation of the Java attribute and the list field and the type of the MySQL table is obtained, and the mapping relation is mapped into the MySQL field, so that any desired processing can be performed on the Datagin data without storing the table and carrying out database IO operation.
In this embodiment, the service system can directly take data in the memory without storing a table, and the coupling is low, which is beneficial to service development and improves coding and service efficiency.
The application also provides a data synchronization method, referring to fig. 3, the method comprises the following steps:
s1, a client performs data extraction and data processing on a database corresponding to a cloud server through a Datagin channel to generate a Datagin file;
s2, the client maps the Datagin file to the object storage service file in an appearance mode;
s3, the cloud server responds to the fact that the client completes mapping of the object storage service file, and reads the object storage service file;
and S4, the cloud service end maps the data in the object storage service file into Java fields so that the Java items can directly use the corresponding Datagin data according to the object storage service file mapped into the Java fields.
Specifically, the Dataphin channel is deployed in the client, and can perform data extraction on a database corresponding to the cloud server system, perform data processing and data assembly on the extracted data, so as to generate a Dataphin file, namely a DP file, but the DP file service system cannot directly use the Dataphin data therein, and then needs to map into an Object storage service (Object StorageService, OSS) file in an external manner, so that the subsequent service system can directly use the corresponding Dataphin data through the oss file without performing I/O operation of the database. In particular, the OSS object storage service parameters may be configured to point to the OSS data source, where the OSS object storage service parameters relate to an Endpoint (access domain name), a socket (storage space), and an access key. The method can be connected to an OSS server by configuring Endpoint, bucket, access Key Id and Access Key secret of an OSS data source so as to download corresponding files through a specified path.
Illustratively, the Dataphin channel has a computing task that performs SQL for cleansing data and generates Dataphin files by parsing the data output by the SQL, which may be used to map into OSS files. The data output by the Datagin through the SQL changes, and the data corresponding to the OSS file also changes, so that the effect of data synchronization is achieved. The data source will be synchronized into the Dataphin system with a D-1 time partition. When the Dataphin channel executes the calculation task and the scheduling is successful, namely, the client side has completed the mapping of the object storage service OSS file, a monitoring event is sent to the cloud server system, and the cloud server system executes the task scheduling of reading the OSS file after receiving the notification; the data in the object store service OSS file is then mapped into Java fields so that the corresponding Dataphin data can be used directly in the Java project. By adopting the technical means, the service system can directly take the data in the memory without storing the table, the coupling is low, the service development can be facilitated, and the coding and service efficiency can be improved.
The data synchronization scheme may be applied to a scheduling system, for example, a lending scenario may be taken as an example, and the cloud service end may store or receive a basic attribute configuration of the sponsor, and store the basic attribute configuration of the sponsor and related data in a corresponding database in an isolated manner. The scheduling system is deployed on the client, performs data extraction on the database system corresponding to the cloud server through the Dataphin channel, performs some data processing on the extracted data, generates scheduling data (wherein the scheduling data comprises attribute configuration and data results of a sponsor) for 30 days, and performs data assembly on the data to generate Dataphin files, wherein the Dataphin files can be used for mapping into OSS files. The schedule data produced by the schedule system can be used as the position of the fund for the fund, for example, if the position of the fund corresponding to the schedule data of a certain fund is 100 ten thousand yuan, the fund of the same fund on the same day cannot exceed 100 ten thousand yuan. If the current user routes to the sponsor during borrowing, judging whether the current user exceeds the current-day position of the sponsor after borrowing, and if the current user does not exceed the current-day position of the sponsor after borrowing, annotating the sponsor to carry out the money release on the current user; if the current user borrows and exceeds the current fund position of the sponsor, the sponsor is not approved to carry out the fund on the current user, and other matched fund of the sponsor can be continuously routed to the current user. It should be noted that the above scheduling scenario is only an example, and the actual Dataphin data synchronization scheme may also be applied in other service scenarios, so as to ensure that the service system can directly take data in the memory without storing a table, thereby facilitating service development and improving coding and service efficiency.
By way of example, in the cloud server system, the project space, schema and table name of Datagin can be specified through the @ Datagin Table annotation, and the column name and annotation of the table can be specified through the @ Datagin Colum annotation, namely, the mapping relation between a certain field in the table and Java attributes is specified, so that the service system can inject the field of the table into the corresponding Java object through Java field reflection according to the mapping relation at the cloud server, and the data of Datagin can be directly used in Java projects. For example, taking the above-described lending scenario as an example, a scheduling month, a scheduling date, a fund plan code, a maximum payable amount, a minimum payable amount, and a scheduling result may be specified, and a mapping relationship between fields of a table describing these dimension attributes and Java attributes may be obtained. Similarly, the Java attribute and MySQL data can be associated, and can be integrated through a Mybat framework, so that the list field and the type of the MySQL table are specified, and the Java attribute and the corresponding MySQL form a mapping relation, so that the corresponding MySQL data is obtained, the data can be processed more flexibly, the table storage is not needed, and the database I/O operation is not needed.
In this embodiment, the client first performs data extraction through the Dataphin channel to extract the service data of the cloud service end into the Dataphin, and maps the data into the OSS file through the way of appearance after the Dataphin performs data processing, so that the service system can directly use the corresponding Dataphin data in the memory without storing a table, and does not need to perform I/O operation of the database, thereby reducing the coupling and improving the service efficiency coding efficiency. In addition, the data output by the Datagin is changed, and the data corresponding to the OSS file is changed, so that the effect of data synchronization is achieved.
Example two
The present application also provides a data synchronization device, corresponding to the above embodiment, applied to a client, referring to fig. 4, where the device includes: the data processing module and the data synchronization module.
The data processing module is used for executing data extraction and data processing on the database corresponding to the cloud service end through the Dataphin channel to generate a Dataphin file;
and the data synchronization module is used for mapping the Dataphin file into the object storage service file in an outward appearance mode.
Further, the device also comprises a parameter configuration module, which is used for configuring the configuration parameters of the object storage service source corresponding to the Datagin file; the configuration parameters comprise an access domain name, a storage space, an access key identity number and an access key user key of the object storage service source.
For specific limitations on the Dataphin data synchronization device applied to the client, reference may be made to the above related limitations on the Dataphin data synchronization method applied to the client, and thus detailed descriptions are omitted herein. The modules in the Dataphin data synchronization device applied to the client can be fully or partially implemented by software, hardware and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
Example III
The application also provides a data synchronization device applied to the cloud service end, corresponding to the embodiment, and referring to fig. 5, the device comprises a reading module and a mapping module.
The reading module is used for responding to the fact that the client side completes the mapping of the object storage service file, and reading is carried out on the object storage service file; and the mapping module is used for mapping the data in the object storage service file into Java fields so that the Java items can directly use the corresponding Datagin data according to the object storage service file mapped into the Java fields.
Further, the mapping module is further configured to specify a project space, a schema, and a table name of the object storage service file corresponding to the Dataphin file; and the mapping relation between the fields in the table and the Java attributes in the corresponding Dataphin file of the object storage service file is specified, so that the fields in the table are conveniently injected into the corresponding Java objects through field reflection according to the mapping relation.
Further, the mapping module is further configured to specify a mapping relationship between the Java attribute and MySQL attribute; wherein the MySQL attribute comprises at least one of a list field and a type of MySQL table.
For specific limitation of the Dataphin data synchronization device applied to the cloud service end, reference may be made to the above related limitation of the Dataphin data synchronization method applied to the cloud service end, so that the description thereof will not be repeated here. All or part of the modules in the Dataphin data synchronization device applied to the cloud service end can be realized by software, hardware and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
Example IV
The application also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the data synchronization method can be realized when the processor executes the program.
As shown in fig. 6, in some embodiments, the system can be used as the above-described electronic device for the Dataphin data synchronization method in any of the described embodiments. In some embodiments, a system may include one or more computer-readable media (e.g., system memory or NVM/storage) having instructions and one or more processors (e.g., processor (s)) coupled with the one or more computer-readable media and configured to execute the instructions to implement the modules to perform the actions described in this disclosure.
For one embodiment, the system control module may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) and/or any suitable device or component in communication with the system control module.
The system control module may include a memory controller module to provide an interface to the system memory. The memory controller modules may be hardware modules, software modules, and/or firmware modules.
The system memory may be used, for example, to load and store data and/or instructions for the system. For one embodiment, the system memory may include any suitable volatile memory, such as, for example, a suitable DRAM. In some embodiments, the system memory may comprise double data rate type four synchronous dynamic random access memory (DDR 4 SDRAM).
For one embodiment, the system control module may include one or more input/output (I/O) controllers to provide an interface to the NVM/storage device and the communication interface(s).
For example, NVM/storage may be used to store data and/or instructions. The NVM/storage may include any suitable nonvolatile memory (e.g., flash memory) and/or may include any suitable nonvolatile storage device(s) (e.g., one or more Hard Disk Drives (HDDs), one or more Compact Disc (CD) drives, and/or one or more Digital Versatile Disc (DVD) drives).
The NVM/storage may include a storage resource that is physically part of the device on which the system is installed or it may be accessed by the device without being part of the device. For example, the NVM/storage may be accessed over a network via the communication interface(s).
The communication interface(s) may provide an interface for the system to communicate over one or more networks and/or with any other suitable device. The system may wirelessly communicate with one or more components of a wireless network in accordance with any of one or more wireless network standards and/or protocols.
For one embodiment, at least one of the processor(s) may be packaged together with logic of one or more controllers (e.g., memory controller modules) of the system control module. For one embodiment, at least one of the processor(s) may be packaged together with logic of one or more controllers of the system control module to form a System In Package (SiP). For one embodiment, at least one of the processor(s) may be integrated on the same die as logic of one or more controllers of the system control module. For one embodiment, at least one of the processor(s) may be integrated on the same die with logic of one or more controllers of the system control module to form a system on chip (SoC).
In various embodiments, the system may be, but is not limited to being: a server, workstation, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet, netbook, etc.). In various embodiments, the system may have more or fewer components and/or different architectures. For example, in some embodiments, a system includes one or more cameras, a keyboard, a Liquid Crystal Display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an Application Specific Integrated Circuit (ASIC), and a speaker.
It should be noted that the present application may be implemented in software and/or a combination of software and hardware, e.g., using Application Specific Integrated Circuits (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the present application may be executed by a processor to perform the steps or functions described above. Likewise, the software programs of the present application (including associated data structures) may be stored on a computer readable recording medium, such as RAM memory, magnetic or optical drive or diskette and the like. In addition, some steps or functions of the present application may be implemented in hardware, for example, as circuitry that cooperates with the processor to perform various steps or functions.
Furthermore, portions of the present application may be implemented as a computer program product, such as computer program instructions, which when executed by a computer, may invoke or provide methods and/or techniques in accordance with the present application by way of operation of the computer. Those skilled in the art will appreciate that the form of computer program instructions present in a computer readable medium includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the manner in which the computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Herein, a computer-readable medium may be any available computer-readable storage medium or communication medium that can be accessed by a computer.
Communication media includes media whereby a communication signal containing, for example, computer readable instructions, data structures, program modules, or other data, is transferred from one system to another. Communication media may include conductive transmission media such as electrical cables and wires (e.g., optical fibers, coaxial, etc.) and wireless (non-conductive transmission) media capable of transmitting energy waves, such as acoustic, electromagnetic, RF, microwave, and infrared. Computer readable instructions, data structures, program modules, or other data may be embodied as a modulated data signal, for example, in a wireless medium, such as a carrier wave or similar mechanism, such as that embodied as part of spread spectrum technology. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The modulation may be analog, digital or hybrid modulation techniques.
An embodiment according to the application comprises an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein the computer program instructions, when executed by the processor, trigger the apparatus to operate a method and/or a solution according to the embodiments of the application as described above.
Example five
Corresponding to the above embodiment, the present application further provides a computer readable storage medium storing computer executable instructions for performing the Dataphin data synchronization method.
In this embodiment, computer-readable storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media include, but are not limited to, volatile memory, such as random access memory (RAM, DRAM, SRAM); and nonvolatile memory such as flash memory, various read only memory (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic/ferroelectric memory (MRAM, feRAM); and magnetic and optical storage devices (hard disk, tape, CD, DVD); or other now known media or later developed computer-readable information/data that can be stored for use by a computer system.
While preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiment and all such alterations and modifications as fall within the scope of the embodiments of the application.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims (10)

1. A data synchronization method applied to a client, the method comprising:
executing data extraction and data processing on a database corresponding to the cloud server through a Dataphin channel to generate a Dataphin file;
mapping the Dataphin file to an object storage service file in an external mode.
2. The datachin data synchronization method of claim 1, wherein prior to said mapping said datachin file into an object storage service file by way of an appearance, said method further comprises:
configuring configuration parameters of an object storage service source corresponding to the Datagin file;
the configuration parameters include an access domain name, a storage space, an access key identification number, and an access key user key of the object storage service source.
3. The data synchronization method is applied to a cloud server and is characterized by comprising the following steps:
in response to detecting that the client completes mapping of the object storage service file, performing reading on the object storage service file;
and mapping the data in the object storage service file into a Java field so that the Java item can directly use the corresponding Datagin data according to the object storage service file mapped into the Java field.
4. The Dataphin data synchronization method of claim 3, wherein the mapping the data in the object store service file into Java fields comprises:
designating a project space, a schema and a table name of the object storage service file corresponding to the Dataphin file;
and designating the mapping relation between the fields in the table and Java attributes in the corresponding Dataphin file of the object storage service file so as to inject the fields in the table into the corresponding Java objects through field reflection according to the mapping relation.
5. The datachin data synchronization method of claim 4, further comprising:
designating the mapping relation between the Java attribute and the MySQL attribute;
wherein the MySQL attribute comprises at least one of a list field and a type of MySQL table.
6. A method for synchronizing data of a Dataphin, the method comprising:
the client side performs data extraction and data processing on a database corresponding to the cloud server side through a Dataphin channel to generate a Dataphin file;
the client maps the Dataphin file to an object storage service file in an outward appearance mode;
the cloud server responds to detection that the client completes mapping of the object storage service file, and reads the object storage service file;
and the cloud service end maps the data in the object storage service file into Java fields so that Java items can directly use corresponding Datagin data according to the object storage service file mapped into the Java fields.
7. A Dataphin data synchronization device for use in a client, the device comprising:
the data processing module is used for executing data extraction and data processing on the database corresponding to the cloud server through the Datagin channel to generate a Datagin file;
and the data synchronization module is used for mapping the Dataphin file into the object storage service file in an outward appearance mode.
8. The utility model provides a data synchronization device, is applied to cloud service end, its characterized in that, the device includes:
the reading module is used for responding to the fact that the client side completes the mapping of the object storage service file, and reading is carried out on the object storage service file;
and the mapping module is used for mapping the data in the object storage service file into Java fields so that the Java items can directly use the corresponding Datagin data according to the object storage service file mapped into the Java fields.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the Dataphin data synchronization method of any one of claims 1 to 6 when the computer program is executed by the processor.
10. A computer readable storage medium storing computer executable instructions for performing the Dataphin data synchronization method of any one of claims 1 to 6.
CN202310615918.6A 2023-05-29 2023-05-29 Datagin data synchronization method, device, equipment and medium Pending CN116610737A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310615918.6A CN116610737A (en) 2023-05-29 2023-05-29 Datagin data synchronization method, device, equipment and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310615918.6A CN116610737A (en) 2023-05-29 2023-05-29 Datagin data synchronization method, device, equipment and medium

Publications (1)

Publication Number Publication Date
CN116610737A true CN116610737A (en) 2023-08-18

Family

ID=87674378

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310615918.6A Pending CN116610737A (en) 2023-05-29 2023-05-29 Datagin data synchronization method, device, equipment and medium

Country Status (1)

Country Link
CN (1) CN116610737A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118260453A (en) * 2024-04-24 2024-06-28 上海数禾信息科技有限公司 Data processing method, device, equipment, medium and product for graph database

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118260453A (en) * 2024-04-24 2024-06-28 上海数禾信息科技有限公司 Data processing method, device, equipment, medium and product for graph database

Similar Documents

Publication Publication Date Title
WO2018137564A1 (en) Service processing method and apparatus
CN110032599B (en) Data structure reading and updating method and device, and electronic equipment
EP3811232A1 (en) Data processing method, apparatus, and device
CN109359237B (en) Method and equipment for searching hosted program
CN114925084B (en) Distributed transaction processing method, system, equipment and readable storage medium
US8826248B2 (en) Enabling computational process as a dynamic data source for bi reporting systems
US20190042288A1 (en) Pl/sql language parsing at a virtual machine
CN109951553B (en) Data processing method, system, electronic device and computer readable storage medium
US20150280786A1 (en) Near field communication based data transfer
US10855750B2 (en) Centralized management of webservice resources in an enterprise
CN110750482A (en) Method and equipment for providing novel reading information
CN102591710A (en) Sharing object representations
CN116610737A (en) Datagin data synchronization method, device, equipment and medium
US8700676B2 (en) System and method for appending metadata to objects
CN110430253A (en) It is a kind of that the method and apparatus of novel update notification information is provided
CN112241461B (en) A method and device for generating a character relationship diagram of a book
US12112180B2 (en) System and method for automatic generation and management of feature level application directory
CN104537035B (en) A kind of data fusion application system and method
US11842176B2 (en) System and method for performing parallel and distributed analysis of program code to generate dependency graphs for executing extract transform load transformations
CN110765390A (en) Method and equipment for publishing shared information in social space
CN114296651A (en) Method and equipment for storing user-defined data information
CN110784596B (en) Method and equipment for processing event information
CN110751003A (en) A method and device for acquiring target data information of a two-dimensional code
CN110784392A (en) Method and equipment for acquiring reading reminding information
CN111125565A (en) A method and device for inputting information in an application

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination