CN110928681A - Data processing method and device, storage medium and electronic device - Google Patents
Data processing method and device, storage medium and electronic device Download PDFInfo
- Publication number
- CN110928681A CN110928681A CN201911096187.9A CN201911096187A CN110928681A CN 110928681 A CN110928681 A CN 110928681A CN 201911096187 A CN201911096187 A CN 201911096187A CN 110928681 A CN110928681 A CN 110928681A
- Authority
- CN
- China
- Prior art keywords
- data
- path
- paths
- identification information
- source data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title abstract description 17
- 238000012545 processing Methods 0.000 claims abstract description 59
- 238000000034 method Methods 0.000 claims abstract description 33
- 230000015654 memory Effects 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 12
- 239000002699 waste material Substances 0.000 abstract description 8
- 230000000694 effects Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 10
- 238000012544 monitoring process Methods 0.000 description 8
- 238000004422 calculation algorithm Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 238000005111 flow chemistry technique Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000005034 decoration Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/503—Resource availability
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method and device, a storage medium and an electronic device. Wherein, the method comprises the following steps: the method comprises the steps of obtaining N paths of source data and identification information, wherein the N paths of source data correspond to different types of data, the identification information is used for respectively representing the attribute of each path of source data in the N paths of source data, and N is an integer greater than 1; configuring a uniform rule for each path of data; each path of configured data is sent to a target interface corresponding to the identification information, and the aim of changing a multi-path data processing task into one task for management is achieved, so that the technical effects of reducing computer resource waste and preventing resource contention among the multi-path data processing tasks are achieved, and the technical problem of computer resource waste caused by processing of multi-path data in the prior art is solved.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a method and an apparatus for processing data, a storage medium, and an electronic apparatus.
Background
With the popularization and spread of big data technologies, more enterprises are looking more and more at analyzing data services. The analysis of data is premised on the collection of such data.
In the prior art, the data collection is usually real-time streaming processing collection, and if a plurality of data sources exist, a plurality of streaming tasks are started to collect the data. For example, for collection of the imsi data, a part of the resources is requested from the computer, and then a streaming task is started, which is always performed, that is, the computer resources are occupied until the task is manually stopped. Then there is also data to collect for mac at this time, which is also the process described above. The two tasks may now take up half of the computer resources. If another data source buckle needs to be collected, the resources of the computer are possibly insufficient, and the resource contention situation occurs among a plurality of tasks, so that the computing efficiency is influenced. The initiation of multiple tasks also places a burden on the management of the tasks at this time, and the code implementation and duplication of multiple tasks increases the workload of a front-end programmer.
In view of the above problems, no effective solution has been proposed.
Disclosure of Invention
The embodiment of the invention provides a data processing method and device, a storage medium and an electronic device, which are used for at least solving the technical problem of computer resource waste caused by processing of multiple paths of data in the prior art.
According to an aspect of an embodiment of the present invention, there is provided a data processing method, including: acquiring N paths of source data and identification information, wherein the N paths of source data correspond to different types of data, the identification information is used for respectively representing the attribute of each path of source data in the N paths of source data, and N is an integer greater than 1; configuring a uniform rule for each path of data; and sending the configured data of each path to a target interface corresponding to the identification information.
Further, the configuring a unified rule for each path of data includes: and configuring a uniform rule for each path of data through a regular expression.
Further, before the acquiring N paths of source data and the identification information, the method further includes: each path of source data in the N paths is respectively stored in different corresponding databases; and acquiring the N paths of source data and the identification information from the database.
Further, after the sending the configured each path of data to the target interface corresponding to the identification information, the method further includes: rewriting the target interface according to each path of data; and processing the data through the rewritten target interface, and storing the processed data into a target database.
According to another aspect of the embodiments of the present invention, there is also provided a data processing apparatus, including: the device comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first acquisition unit is used for acquiring N paths of source data and identification information, the N paths of source data correspond to different types of data, the identification information is used for respectively representing the attribute of each path of source data in the N paths of source data, and N is an integer greater than 1; the configuration unit is used for configuring a uniform rule for each path of data; and the sending unit is used for sending the configured data of each path to a target interface corresponding to the identification information.
Further, the configuration unit includes: and the configuration module is used for configuring a uniform rule for each path of data through a regular expression.
Further, the above apparatus further comprises: the storage unit is used for respectively storing each path of source data in the N paths in different corresponding databases before the N paths of source data and the identification information are obtained; and the second acquisition unit is used for acquiring the N paths of source data and the identification information from the database.
Further, the above apparatus further comprises: the rewriting unit is used for rewriting the target interface according to each path of data after sending each path of configured data to the target interface corresponding to the identification information; and the storage unit is used for processing the data through the rewritten target interface and storing the processed data into a target database.
According to another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the above data processing method when running.
According to another aspect of the embodiments of the present invention, there is also provided an electronic apparatus, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method for processing data through the computer program.
In the embodiment of the invention, N paths of source data and identification information are obtained, wherein the N paths of source data correspond to different types of data, the identification information is used for respectively representing the attribute of each path of source data in the N paths of source data, and N is an integer greater than 1; configuring a uniform rule for each path of data; each path of configured data is sent to a target interface corresponding to the identification information, and the aim of changing a multi-path data processing task into one task for management is achieved, so that the technical effects of reducing computer resource waste and preventing resource contention among the multi-path data processing tasks are achieved, and the technical problem of computer resource waste caused by processing of multi-path data in the prior art is solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
FIG. 1 is a schematic diagram of an application environment of an alternative data processing method according to an embodiment of the invention;
FIG. 2 is a flow chart of an alternative method of processing data according to an embodiment of the invention;
FIG. 3 is a block diagram of an alternative engine for multi-source streaming data processing in accordance with a preferred embodiment of the present invention;
FIG. 4 is a block diagram of an alternative stream processing module in accordance with the preferred embodiment of the present invention;
FIG. 5 is a schematic diagram of an alternative interface according to the preferred embodiment of the present invention;
FIG. 6 is a schematic diagram of an alternative data processing apparatus according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of an electronic device according to an alternative data processing method in an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For a better understanding of the present application, the following is explained for the relevant names:
entity: an entity is the subject of an event occurrence, but it is not limited to a certain category. For example, in the same place, two devices are simultaneously collecting information data, and the first device mainly collects mobile phone number information, and certainly, other accessory information is also provided. The second device mainly collects WIFI signal information, so that the main bodies of the two events are different, namely a mobile phone number (IMSI) and WIFI (MAC), and similarly, the entity types include a bayonet data source, a face recognition data source and the like.
Entity information: detailed information of the entity, such as collecting WIFI (MAC) signal event, including information of MAC address, location code, occurrence time, area code, flag bit, etc
Event data source: the method comprises the steps of collecting mobile phone number (IMSI) information data about a base station in the public security field, collecting WIFI (media access control) (MAC) information data in important places, and license plate number information data after a vehicle passes through a traffic buckle.
Computer resources: refers to computer physical resources such as memory, hard disk, CPU, core count, etc. The physical resources of each computer are certain, certain resources are consumed when one task is started, and the resources are consumed when a plurality of tasks are started.
Flow processing: after the task is started, the data at the data source is appointed to be processed, the data is always received and processed according to the predefined processing logic, and the data is not stopped until an error is met or the data is manually stopped.
Multi-source flow processing: the reality is that there are many data sources, such as data source of collecting IMSI, data source of collecting MAC, data source of collecting vehicle buckle. Because these data sources are very diverse, if the data sources are processed, multiple tasks are started to process, for example, a data source for IMSI will start a task for processing IMSI, and a data source for MAC will start a task for processing MAC. Therefore, a plurality of tasks are started, the plurality of tasks consume computer resources, the computer resources are seriously wasted, and if a newly added data source exists, one task is started, so that resource redundancy is caused. Therefore, the multi-source streaming processing engine aims to solve the problem of multi-source data processing, save computer consumption resources and improve the calculation efficiency.
Topic: a collection of a type of data. Such as IMSI data, stores all data related to IMSI, which are identical in format, number, and column order, but differ in data content.
According to an aspect of the embodiments of the present invention, a method for processing data is provided, and optionally, as an optional implementation, the method for processing data may be applied to, but is not limited to, an environment as shown in fig. 1.
Optionally, in this embodiment, the data processing method may be, but is not limited to, applied to the server 104, and is configured to obtain N paths of source data and identification information, where the N paths of source data correspond to different types of data, the identification information is used to respectively represent an attribute of each path of source data in the N paths of source data, and N is an integer greater than 1; configuring a uniform rule for each path of data; and sending each path of configured data to a target interface corresponding to the identification information. The application client may be but not limited to run in the user equipment 102, and the user equipment 102 may be but not limited to a mobile phone, a tablet computer, a notebook computer, a PC, and other terminal equipment supporting running of the application client. The server 104 and the user device 102 may, but are not limited to, enable data interaction via a network, which may include, but is not limited to, a wireless network or a wired network. Wherein, this wireless network includes: bluetooth, WIFI, and other networks that enable wireless communication. Such wired networks may include, but are not limited to: wide area networks, metropolitan area networks, and local area networks. The above is merely an example, and this is not limited in this embodiment.
Optionally, in this embodiment, the data processing method may be, but is not limited to, applied to the user equipment 102, and is configured to obtain N paths of source data and identification information, where the N paths of source data correspond to different types of data, the identification information is used to respectively represent an attribute of each path of source data in the N paths of source data, and N is an integer greater than 1; configuring a uniform rule for each path of data; and sending each path of configured data to a target interface corresponding to the identification information. The above is merely an example, and this is not limited in this embodiment.
Optionally, in this embodiment, the data processing method may be, but is not limited to, applied to interaction with the user equipment 102 in the server 104, where the user equipment 102 is configured to obtain N paths of source data and identification information, where the N paths of source data correspond to different types of data, the identification information is used to respectively represent an attribute of each path of source data in the N paths of source data, and N is an integer greater than 1. The server 104 is used for configuring a uniform rule for each path of data; and sending each path of configured data to a target interface corresponding to the identification information. The above is merely an example, and this is not limited in this embodiment.
Optionally, as an optional implementation manner, as shown in fig. 2, the data processing method includes:
step S202, N paths of source data and identification information are obtained, wherein the N paths of source data correspond to different types of data, the identification information is used for respectively representing the attribute of each path of source data in the N paths of source data, and N is an integer greater than 1.
Step S204, a uniform rule is configured for each path of data.
And step S206, sending each path of configured data to a target interface corresponding to the identification information.
Alternatively, the scheme in this implementation may be applied to multi-source streaming processing.
Specifically, N paths of source data may be obtained, where the data type of each path of data is different. For example, 3 paths of source data can be acquired, wherein 1 path of data is mobile phone number information, 2 paths of data is WIFI signal information, and 3 paths of data are bayonet data. And uniformly configuring the N paths of data, namely performing character matching on the 3 paths of data through a regular expression, and further sending each path of configured data to a target interface corresponding to the identification information. Then, the algorithm of the target interface is rewritten according to the difference of the data source, and the business logic processing of the target interface is realized in the rewritten algorithm. And finally, receiving the corresponding database, sending the corresponding database to the corresponding interface to complete the processing of the service logic, and storing the service logic into a final result.
Optionally, configuring a unified rule for each path of data may include: and configuring a uniform rule for each path of data through a regular expression.
It should be noted that, before acquiring the N paths of source data and the identification information, each path of source data in the N paths is respectively stored in different corresponding databases; and acquiring N paths of source data and identification information from the database. For example, if there are three data sources whose data needs to be processed, respectively A, B, C, then three topics (corresponding to databases) are created to store the received three types of data, respectively. The command formats of Topic at this time are: streaming _ A, streaming _ B, streaming _ C. And then, monitoring the Topic through a regular expression to acquire data in real time.
It should be noted that after each path of configured data is sent to the target interface corresponding to the identification information, the target interface is rewritten according to each path of data; and processing the data through the rewritten target interface, and storing the processed data into a target database.
According to the embodiment provided by the application, N paths of source data and identification information are obtained, wherein the N paths of source data correspond to different types of data, the identification information is used for respectively representing the attribute of each path of source data in the N paths of source data, and N is an integer greater than 1; configuring a uniform rule for each path of data; each path of configured data is sent to a target interface corresponding to the identification information, and the aim of changing a multi-path data processing task into one task for management is achieved, so that the technical effects of reducing computer resource waste and preventing resource contention among the multi-path data processing tasks are achieved, and the technical problem of computer resource waste caused by processing of multi-path data in the prior art is solved.
The invention also provides a preferred embodiment, which provides a multi-source streaming data processing method. Fig. 3 is a block diagram of an engine for multi-source streaming data processing according to a preferred embodiment of the present invention.
The structure shown in fig. 3 includes: the event data source comprises a mobile phone signal data, WIFI data and buckle data, a monitoring module of a multi-source flow engine frame is used for monitoring each path of data in real time, performing flow processing operation, sending the flow processing module and the processed data to corresponding interfaces, performing business logic operation through the corresponding interfaces, and storing operation results.
The specific implementation may be that, in the process of monitoring the multi-data source incremental data, a uniform command rule is specified, and matching is performed through a regular expression. Collections of the same type of data are stored in the database, Topic, that is, the original data is continuously stored in Topic. Further, by monitoring the Topic in real time, a piece of real-time data can be continuously obtained. For example, if there are three data sources whose data needs to be processed, respectively A, B, C, then three topics are created to store the received three types of data, respectively. The command formats of Topic at this time are: streaming _ A, streaming _ B, streaming _ C. And then, monitoring the Topic through a regular expression to acquire data in real time. It should be noted that the regular expression may be a matching character, and as long as the rule of the Topic command meets the command specification, the regular expression can match the data in the corresponding Topic and can continuously receive new and added data. This may facilitate dynamically increasing data sources.
After the data is taken, the data is distributed to different interfaces according to different names of the data Topic. A uniform interface (either a specification or a method) is assigned, then the algorithms in the interface are rewritten according to the data source, and the business logic processing of the respective data is realized in the rewritten algorithms. And finally, after receiving the corresponding Topic data, sending the received Topic data to a corresponding interface to complete the processing of the business logic and storing the business logic into a final result, thereby realizing a one-stop streaming data processing engine capable of processing a plurality of data sources.
As shown in fig. 4, it is a schematic structural diagram of a stream processing module according to a preferred embodiment of the present invention, and a specific data processing flow thereof is as follows: there are three data sources whose data needs to be processed, A, B, C respectively, then three topics are created to store the received three types of data respectively. The command formats of Topic at this time are: streaming _ A, streaming _ B, streaming _ C. And then, monitoring the Topic through a regular expression to acquire data in real time. It should be noted that the regular expression may be a matching character, and as long as the rule of the Topic command meets the command specification, the regular expression can match the data in the corresponding Topic and can continuously receive new and added data. This may facilitate dynamically increasing data sources.
After the data is taken, the data is distributed to different interfaces according to different names of the data Topic. A uniform interface (either a specification or a method) is assigned, then the algorithms in the interface are rewritten according to the data source, and the business logic processing of the respective data is realized in the rewritten algorithms. And finally, after receiving the corresponding Topic data, sending the received Topic data to a corresponding interface to complete the processing of the business logic and storing the business logic into a final result, thereby realizing a one-stop streaming data processing engine capable of processing a plurality of data sources. Fig. 5 is a schematic diagram showing a specific interface. Each interface corresponds to a different business logic.
Through the preferred embodiment, the data of a plurality of data sources can be simultaneously acquired by uniformly monitoring the plurality of data sources, and then a uniform interface specification is defined. And distributing the received data to different interfaces according to the Topic names of the received data, wherein the interfaces have already been specified in a uniform specification, and at the moment, only the business logic of the data needs to be realized under the specification, and the result is finally stored. Compared with the traditional implementation mode, the method only needs to start one streaming task, and sets uniform attribute configuration and resource configuration. Therefore, only one streaming task is provided, and the situations of task management burden and multi-task resource earning do not exist. The computer resources are applied only once, so that the computer resources are saved, and the resource waste is avoided. A plurality of business logics share a set of streaming processing framework, and the burden of repeated development of code amount by a first-line programmer is reduced.
Compared with the prior art, the preferred embodiment has the following advantages: the original requirement for starting a plurality of streaming tasks is changed into the requirement for directly starting one streaming task, and the streaming task is used as a computing engine base, so that even if a data source is dynamically increased, no code modification is required to be carried out on the streaming engine. The code processing efficiency is improved by uniformly managing tasks, managing resources, distributing resources and dynamically increasing data sources to perform service processing, the processing flow is simplified, and the code development amount is reduced.
It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.
According to another aspect of the embodiment of the present invention, there is also provided a data processing apparatus for implementing the above data processing method. As shown in fig. 6, the data processing apparatus includes: a first acquisition unit 61, a configuration unit 63 and a transmission unit 65.
The first obtaining unit 61 is configured to obtain N paths of source data and identification information, where the N paths of source data correspond to different types of data, the identification information is used to respectively represent an attribute of each path of source data in the N paths of source data, and N is an integer greater than 1.
And the configuration unit 63 is used for configuring a uniform rule for each path of data.
Wherein, the configuration unit 63 may include: and the configuration module is used for configuring a uniform rule for each path of data through a regular expression.
And a sending unit 65, configured to send each configured path of data to the target interface corresponding to the identification information.
By the above apparatus, the first obtaining unit 61 obtains N paths of source data and identification information, where the N paths of source data correspond to different types of data, the identification information is used to respectively represent an attribute of each path of source data in the N paths of source data, and N is an integer greater than 1; the configuration unit 63 configures a uniform rule for each path of data; the transmitting unit 65 transmits each configured path of data to the target interface corresponding to the identification information.
As an alternative embodiment, the apparatus may further include: the storage unit is used for respectively storing each path of source data in the N paths in different corresponding databases before acquiring the source data and the identification information of the N paths; and the second acquisition unit is used for acquiring the N paths of source data and the identification information from the database.
As an alternative embodiment, the apparatus may further include: the rewriting unit is used for rewriting the target interface according to each path of data after each path of configured data is sent to the target interface corresponding to the identification information; and the storage unit is used for processing the data through the rewritten target interface and storing the processed data into the target database.
According to a further aspect of the embodiments of the present invention, there is also provided an electronic device for implementing the above data processing method, as shown in fig. 7, the electronic device includes a memory 702 and a processor 704, the memory 702 stores a computer program therein, and the processor 704 is configured to execute the steps in any one of the above method embodiments through the computer program.
Optionally, in this embodiment, the electronic apparatus may be located in at least one network device of a plurality of network devices of a computer network.
Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:
s1, acquiring N paths of source data and identification information, wherein the N paths of source data correspond to different types of data, the identification information is used for respectively representing the attribute of each path of source data in the N paths of source data, and N is an integer greater than 1;
s2, configuring a uniform rule for each path of data;
and S3, sending each path of configured data to a target interface corresponding to the identification information.
Alternatively, it can be understood by those skilled in the art that the structure shown in fig. 7 is only an illustration, and the electronic device may also be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, and the like. Fig. 7 is a diagram illustrating a structure of the electronic device. For example, the electronic device may also include more or fewer components (e.g., network interfaces, etc.) than shown in FIG. 7, or have a different configuration than shown in FIG. 7.
The memory 702 may be used to store software programs and modules, such as program instructions/modules corresponding to the data processing method and apparatus in the embodiments of the present invention, and the processor 704 executes various functional applications and data processing by running the software programs and modules stored in the memory 702, that is, implements the data processing method described above. The memory 702 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 702 can further include memory located remotely from the processor 704, which can be connected to the terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. The memory 702 may be specifically, but not limited to, used for N-way source data, identification information, and other information. As an example, as shown in fig. 7, the memory 702 may include, but is not limited to, the first acquiring unit 61, the configuring unit 63, and the transmitting unit 65 in the processing device that includes the data. In addition, the data processing device may further include, but is not limited to, other module units in the data processing device, which is not described in this example again.
Optionally, the transmitting device 706 is used for receiving or sending data via a network. Examples of the network may include a wired network and a wireless network. In one example, the transmission device 706 includes a Network adapter (NIC) that can be connected to a router via a Network cable and other Network devices to communicate with the internet or a local area Network. In one example, the transmission device 706 is a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.
In addition, the electronic device further includes: a display 708 for displaying the identification information of the N-way metadata; and a connection bus 710 for connecting the respective module parts in the above-described electronic apparatus.
According to a further aspect of an embodiment of the present invention, there is also provided a computer-readable storage medium having a computer program stored thereon, wherein the computer program is arranged to perform the steps of any of the above method embodiments when executed.
Alternatively, in the present embodiment, the above-mentioned computer-readable storage medium may be configured to store a computer program for executing the steps of:
s1, acquiring N paths of source data and identification information, wherein the N paths of source data correspond to different types of data, the identification information is used for respectively representing the attribute of each path of source data in the N paths of source data, and N is an integer greater than 1;
s2, configuring a uniform rule for each path of data;
and S3, sending each path of configured data to a target interface corresponding to the identification information.
Alternatively, in this embodiment, a person skilled in the art may understand that all or part of the steps in the methods of the foregoing embodiments may be implemented by a program instructing hardware associated with the terminal device, where the program may be stored in a computer-readable storage medium, and the storage medium may include: flash disks, Read-Only memories (ROMs), Random Access Memories (RAMs), magnetic or optical disks, and the like.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
The integrated unit in the above embodiments, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing one or more computer devices (which may be personal computers, servers, network devices, etc.) to execute all or part of the steps of the method according to the embodiments of the present invention.
In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the several embodiments provided in the present application, it should be understood that the disclosed client may be implemented in other manners. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one type of division of logical functions, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (10)
1. A method for processing data, comprising:
acquiring N paths of source data and identification information, wherein the N paths of source data correspond to different types of data, the identification information is used for respectively representing the attribute of each path of source data in the N paths of source data, and N is an integer greater than 1;
configuring a uniform rule for each path of data;
and sending the configured data of each path to a target interface corresponding to the identification information.
2. The method of claim 1, wherein the configuring the unified rule for each path of data comprises:
and configuring a uniform rule for each path of data through a regular expression.
3. The method of claim 1, wherein before the obtaining the N-way source data and the identification information, the method further comprises:
each path of source data in the N paths is respectively stored in different corresponding databases;
and acquiring the N paths of source data and the identification information from the database.
4. The method according to claim 1, wherein after the sending the configured each path of data to the target interface corresponding to the identification information, the method further comprises:
rewriting the target interface according to each path of data;
and processing the data through the rewritten target interface, and storing the processed data into a target database.
5. An apparatus for processing data, comprising:
the device comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first acquisition unit is used for acquiring N paths of source data and identification information, the N paths of source data correspond to different types of data, the identification information is used for respectively representing the attribute of each path of source data in the N paths of source data, and N is an integer greater than 1;
the configuration unit is used for configuring a uniform rule for each path of data;
and the sending unit is used for sending the configured data of each path to a target interface corresponding to the identification information.
6. The apparatus of claim 5, wherein the configuration unit comprises:
and the configuration module is used for configuring a uniform rule for each path of data through a regular expression.
7. The apparatus of claim 5, further comprising:
the storage unit is used for respectively storing each path of source data in the N paths in different corresponding databases before the N paths of source data and the identification information are obtained;
and the second acquisition unit is used for acquiring the N paths of source data and the identification information from the database.
8. The apparatus of claim 5, further comprising:
the rewriting unit is used for rewriting the target interface according to each path of data after sending each path of configured data to the target interface corresponding to the identification information;
and the storage unit is used for processing the data through the rewritten target interface and storing the processed data into a target database.
9. A computer-readable storage medium comprising a stored program, wherein the program when executed performs the method of any of claims 1 to 4.
10. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method of any of claims 1 to 4 by means of the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911096187.9A CN110928681A (en) | 2019-11-11 | 2019-11-11 | Data processing method and device, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911096187.9A CN110928681A (en) | 2019-11-11 | 2019-11-11 | Data processing method and device, storage medium and electronic device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110928681A true CN110928681A (en) | 2020-03-27 |
Family
ID=69853759
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911096187.9A Pending CN110928681A (en) | 2019-11-11 | 2019-11-11 | Data processing method and device, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110928681A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112215606A (en) * | 2020-10-19 | 2021-01-12 | 支付宝(杭州)信息技术有限公司 | Data processing method and device |
| CN112884303A (en) * | 2021-02-02 | 2021-06-01 | 深圳市欢太科技有限公司 | Data annotation method and device, electronic equipment and computer readable storage medium |
| CN115297183A (en) * | 2022-07-29 | 2022-11-04 | 天翼云科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN115827526A (en) * | 2022-11-07 | 2023-03-21 | 北京百度网讯科技有限公司 | Data processing method, device, equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105159951A (en) * | 2015-08-17 | 2015-12-16 | 成都中科大旗软件有限公司 | Open tourism multi-source heterogeneous data fusion method and system |
| CN106557307A (en) * | 2015-09-29 | 2017-04-05 | 腾讯科技（深圳）有限公司 | The processing method and processing system of business datum |
| CN108182233A (en) * | 2017-12-27 | 2018-06-19 | 苏州麦迪斯顿医疗科技股份有限公司 | A kind of distributed data abstracting method, device, computer equipment and storage medium |
| CN109379432A (en) * | 2018-10-31 | 2019-02-22 | 腾讯科技（深圳）有限公司 | Data processing method, device, server and computer readable storage medium |
| CN110532493A (en) * | 2019-08-29 | 2019-12-03 | 北京明略软件系统有限公司 | Processing method and processing device, storage medium and the electronic device of data |
-
2019
- 2019-11-11 CN CN201911096187.9A patent/CN110928681A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105159951A (en) * | 2015-08-17 | 2015-12-16 | 成都中科大旗软件有限公司 | Open tourism multi-source heterogeneous data fusion method and system |
| CN106557307A (en) * | 2015-09-29 | 2017-04-05 | 腾讯科技（深圳）有限公司 | The processing method and processing system of business datum |
| CN108182233A (en) * | 2017-12-27 | 2018-06-19 | 苏州麦迪斯顿医疗科技股份有限公司 | A kind of distributed data abstracting method, device, computer equipment and storage medium |
| CN109379432A (en) * | 2018-10-31 | 2019-02-22 | 腾讯科技（深圳）有限公司 | Data processing method, device, server and computer readable storage medium |
| CN110532493A (en) * | 2019-08-29 | 2019-12-03 | 北京明略软件系统有限公司 | Processing method and processing device, storage medium and the electronic device of data |
Non-Patent Citations (1)
| Title |
|---|
| 陈明: "《大数据技术概论》", 31 January 2019, 中国铁道出版社 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112215606A (en) * | 2020-10-19 | 2021-01-12 | 支付宝(杭州)信息技术有限公司 | Data processing method and device |
| CN112884303A (en) * | 2021-02-02 | 2021-06-01 | 深圳市欢太科技有限公司 | Data annotation method and device, electronic equipment and computer readable storage medium |
| CN115297183A (en) * | 2022-07-29 | 2022-11-04 | 天翼云科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN115297183B (en) * | 2022-07-29 | 2023-11-03 | 天翼云科技有限公司 | A data processing method, device, electronic equipment and storage medium |
| CN115827526A (en) * | 2022-11-07 | 2023-03-21 | 北京百度网讯科技有限公司 | Data processing method, device, equipment and storage medium |
| CN115827526B (en) * | 2022-11-07 | 2023-10-27 | 北京百度网讯科技有限公司 | Data processing method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110908879B (en) | Reporting method, reporting device, reporting terminal and recording medium of buried point data | |
| WO2020001108A1 (en) | Block chain-based data processing method and device | |
| CN112017007B (en) | User behavior data processing method and device, computer equipment, and storage medium | |
| CN110928681A (en) | Data processing method and device, storage medium and electronic device | |
| CN107220142B (en) | Method and device for executing data recovery operation | |
| CN113553310B (en) | Data acquisition method and device, storage medium and electronic equipment | |
| CN111258978B (en) | Data storage method | |
| WO2019109518A1 (en) | Data list uploading method and terminal thereof | |
| CN109831334B (en) | Network topology construction method and device and terminal equipment | |
| CN103312544A (en) | Method, equipment and system for controlling terminals during log file reporting | |
| CN112671878B (en) | Block chain information subscription method, device, server and storage medium | |
| CN112328592B (en) | Data storage method, electronic device, and computer-readable storage medium | |
| CN112181678A (en) | Service data processing method, device and system, storage medium and electronic device | |
| US20190050435A1 (en) | Object data association index system and methods for the construction and applications thereof | |
| CN108563776B (en) | Offline data acquisition method and system, server and storage medium | |
| CN104216698A (en) | Webpage registration method and relative device | |
| CN112445861A (en) | Information processing method, device, system and storage medium | |
| CN107277095B (en) | Session segmentation method and device | |
| CN110188258B (en) | Method and device for acquiring external data by using crawler | |
| CN108520401B (en) | User list management method, device, platform and storage medium | |
| WO2018188607A1 (en) | Stream processing method and device | |
| CN112162731A (en) | Data expansion method, device, storage medium and electronic device | |
| CN113434612B (en) | Data statistics methods and devices, storage media and electronic devices | |
| CN115509640A (en) | Service processing method, device, storage medium and electronic equipment | |
| CN114675931A (en) | A resource monitoring method and monitoring device of an integrated platform instance |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20200327 |
|
| WD01 | Invention patent application deemed withdrawn after publication |