CN115145585A - Data acquisition method, system, electronic equipment and storage medium - Google Patents
Data acquisition method, system, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN115145585A CN115145585A CN202210852215.0A CN202210852215A CN115145585A CN 115145585 A CN115145585 A CN 115145585A CN 202210852215 A CN202210852215 A CN 202210852215A CN 115145585 A CN115145585 A CN 115145585A
- Authority
- CN
- China
- Prior art keywords
- data
- data acquisition
- tool
- file
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 claims abstract description 63
- 238000013480 data collection Methods 0.000 claims description 30
- 230000008569 process Effects 0.000 claims description 22
- 238000004590 computer program Methods 0.000 claims description 15
- 230000000694 effects Effects 0.000 abstract description 12
- 230000006870 function Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 12
- 238000003780 insertion Methods 0.000 description 12
- 230000037431 insertion Effects 0.000 description 12
- 238000004891 communication Methods 0.000 description 8
- 239000007943 implant Substances 0.000 description 8
- 238000011160 research Methods 0.000 description 5
- 238000006243 chemical reaction Methods 0.000 description 4
- 238000002513 implantation Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000012827 research and development Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000003826 tablet Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/41—Compilation
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The application provides a data acquisition method, a system, electronic equipment and a storage medium, firstly acquiring a plurality of application programs which are determined in advance or in real time, independently acquiring service data of each application program according to data acquisition middleware services, and transmitting the acquired service data to a database for summarizing; and uploading the service data summarized in the database to a server side based on a data unification rule determined in advance or in real time. Wherein the service of each application program is independent from the service of other application programs. Therefore, the data can be directly inserted in the code compiling stage, the purpose of data acquisition is achieved, the complexity of data acquisition is greatly simplified, the workload of data acquisition is reduced, the data acquisition efficiency is improved, and the once-for-all data acquisition effect is basically achieved.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data acquisition method, a data acquisition system, an electronic device, and a storage medium.
Background
For a vehicle-mounted terminal, data acquisition is mostly implemented by accessing an App (Application) to an SDK (Software Development Kit, SDK for short) and then performing Http (hypertext Transfer Protocol, http for short) uploading. During data acquisition, the number of the related apps is few, the number of the related apps is dozens, the related functions of each App are multiplied, new requirements are continuously met, and new apps are added. And new data acquisition is added on the basis of the addition of a new App every time when new requirements are met, and research personnel can access the SDK again to realize the function.
On the basis of data embedding points, part of documents can issue data by using a remote instruction to realize required data acquisition, but the part of documents does not solve the problem of complex embedding points, and therefore, a brand-new data acquisition technology is needed to be provided.
Disclosure of Invention
In view of the above shortcomings in the prior art, the present application provides a data acquisition method, system, electronic device and storage medium to solve the above technical problems.
The application provides a data acquisition method, which comprises the following steps:
acquiring a plurality of application programs determined in advance or in real time, wherein the service of each application program is independent of the service of other application programs;
independently collecting the service data of each application program according to the data collection middleware service, and transmitting the collected service data to a database for summarizing;
and uploading the service data summarized in the database to a server side based on a data unification rule determined in advance or in real time.
In an embodiment of the present application, the process of independently collecting the service data of each application according to the data collection middleware service includes:
presenting the source code of each application program by taking a preset computer code as a suffix to serve as a data acquisition middleware;
converting the presented source code into a class file containing a constant pool and a logic class by using a compiling tool;
selecting a linking tool associated with the compiling tool, converting the class file into an executable file based on the selected linking tool, and merging part of repeated data on the basis of the class file, so that the collected business data comprises constant pool, logic and header data.
In an embodiment of the present application, the process of independently collecting the service data of each application according to the data collection middleware service includes:
presenting the source code of each application program by taking a preset computer code as a suffix to serve as a data acquisition middleware;
converting the presented source code into a first class file containing a constant pool and a logic class by using a compiling tool, and directly implanting data into the first class file by using a data acquisition tool to form a second class file;
selecting a linking tool associated with the compiling tool, converting the second type of file into an executable file based on the selected linking tool, and merging part of repeated data on the basis of the second type of file so as to enable the collected business data to comprise constant pool, logic and header data.
In an embodiment of the application, the preset computer code includes java code, the class file includes a class file, the executable file includes a dex file, the compiling tool includes an Android compiling tool, and the linking tool includes an Android linking tool.
In an embodiment of the application, the preset computer code includes java code, the first class file and the second class file include class files, the executable file includes dex files, the compiling tool includes an Android compiling tool, the linking tool includes an Android linking tool, and the data collecting tool includes ASM instrumentation.
The application also provides a data acquisition system, the system including:
the system comprises an application program module, a data acquisition module and a data acquisition module, wherein the application program module is used for determining a plurality of application programs needing data acquisition in advance or in real time, and the service of each application program is independent of the service of other application programs;
the data acquisition module is used for independently acquiring the service data of each application program according to the data acquisition middleware service and transmitting the acquired service data to the database for summarizing;
and the data uploading module is used for uploading the service data summarized in the database to a server side according to a data unification rule determined in advance or in real time.
In an embodiment of the present application, the process of independently collecting the service data of each application according to the data collection middleware service includes:
presenting the source code of each application program by taking a preset computer code as a suffix to serve as a data acquisition middleware;
converting the presented source code into a class file containing a constant pool and a logic class by using a compiling tool;
selecting a linking tool associated with the compiling tool, converting the class file into an executable file based on the selected linking tool, and merging part of repeated data on the basis of the class file, so that the collected business data comprises constant pool, logic and header data.
In an embodiment of the present application, the process of independently collecting the service data of each application according to the data collection middleware service includes:
presenting the source code of each application program by taking a preset computer code as a suffix to serve as a data acquisition middleware;
converting the presented source code into a first class file containing a constant pool and a logic class by using a compiling tool, and directly implanting data into the first class file by using a data acquisition tool to form a second class file;
selecting a linking tool associated with the compiling tool, converting the second type of file into an executable file based on the selected linking tool, and merging part of repeated data on the basis of the second type of file so as to enable the collected business data to comprise constant pool, logic and header data.
The present application further provides an electronic device, the electronic device including:
one or more processors;
a storage device for storing one or more programs that, when executed by the one or more processors, cause the electronic device to implement the data acquisition method as in any one of the above.
The present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor of a computer, causes the computer to perform a data acquisition method as defined in any one of the above.
As described above, the present application provides a data acquisition method, a system, an electronic device, and a storage medium, which have the following advantages:
the method comprises the steps of firstly, obtaining a plurality of application programs which are determined in advance or in real time, independently collecting service data of each application program according to data collection middleware service, and transmitting the collected service data to a database for summarizing; and uploading the service data summarized in the database to a server side based on a data unification rule determined in advance or in real time. Therefore, the data can be directly inserted in the code compiling stage, the purpose of data acquisition is achieved, the complexity of data acquisition is greatly simplified, the workload of data acquisition is reduced, the data acquisition efficiency is improved, and the once-for-all data acquisition effect is basically achieved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application. It is obvious that the drawings in the following description are only some embodiments of the application, and that for a person skilled in the art, other drawings can be derived from them without inventive effort. In the drawings:
FIG. 1 is a schematic diagram of an exemplary system architecture to which aspects of one or more embodiments of the present application may be applied;
fig. 2 is a schematic flowchart of a data acquisition method according to an embodiment of the present application;
fig. 3 is a schematic diagram illustrating a byte code conversion process according to an embodiment of the present application;
fig. 4 is a schematic view of a byte code conversion process according to another embodiment of the present application;
fig. 5 is a schematic flow chart of a data acquisition method according to another embodiment of the present application;
fig. 6 is a schematic hardware structure diagram of a data acquisition system according to an embodiment of the present application;
fig. 7 is a hardware configuration diagram of an electronic device suitable for implementing one or more embodiments of the present application.
Detailed Description
Other advantages and effects of the present application will become apparent to those skilled in the art from the disclosure herein, wherein the embodiments of the present application will be described in detail with reference to the accompanying drawings and preferred embodiments. The present application is capable of other and different embodiments and its several details are capable of modifications and/or changes in various respects, all without departing from the spirit of the present application. It should be understood that the preferred embodiments are for purposes of illustration only and are not intended to limit the scope of the present disclosure.
It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present application, and the drawings only show the components related to the present application and are not drawn according to the number, shape and size of the components in actual implementation, and the type, number and proportion of the components in actual implementation may be changed freely, and the layout of the components may be more complicated.
In the following description, numerous details are set forth to provide a more thorough explanation of the embodiments of the present application, however, it will be apparent to one skilled in the art that the embodiments of the present application may be practiced without these specific details, and in other embodiments, well-known structures and devices are shown in block diagram form rather than in detail in order to avoid obscuring the embodiments of the present application.
Java, is a computer programming language that faces objects.
Header is a word string sent by a server before transmitting HTML (Hyper Text Markup Language, HTML) data to a browser by using HTTP (Hyper Text Transfer Protocol, HTTP for short), and a space is needed between the Header and the HTML file.
ASM, a Java bytecode manipulation framework. It can be used to dynamically generate classes or enhance the functionality of existing classes. The ASM can either directly generate the binary class file or dynamically change the class behavior before the class is loaded into the Java virtual machine.
Fig. 1 shows a schematic diagram of an exemplary system architecture to which technical solutions in one or more embodiments of the present application may be applied. As shown in fig. 1, system architecture 100 may include a terminal device 110, a network 120, and a server 130. The terminal device 110 may include various electronic devices such as a smart phone, a tablet computer, a notebook computer, and a desktop computer. The server 130 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud computing services. Network 120 may be a communication medium of various connection types capable of providing a communication link between terminal device 110 and server 130, such as a wired communication link or a wireless communication link.
The system architecture in the embodiments of the present application may have any number of terminal devices, networks, and servers, according to implementation needs. For example, the server 130 may be a server group composed of a plurality of server devices. In addition, the technical solution provided in the embodiment of the present application may be applied to the terminal device 110, or may be applied to the server 130, or may be implemented by both the terminal device 110 and the server 130, which is not particularly limited in this application.
In an embodiment of the present application, the terminal device 110 or the server 130 of the present application may first obtain a plurality of application programs determined in advance or in real time, independently collect service data of each application program according to a data collection middleware service, and transmit the collected service data to a database for summarizing; and uploading the service data summarized in the database to a server side based on a data unification rule determined in advance or in real time. The data acquisition method is executed by using the terminal device 110 or the server 130, data can be directly inserted in the code compiling stage, the purpose of data acquisition is realized, the complexity of data acquisition is greatly simplified, the workload of data acquisition is reduced, the data acquisition efficiency is improved, and the once-for-all data acquisition effect is basically realized.
The above section introduces the content of an exemplary system architecture to which the technical solution of the present application is applied, and then the data acquisition method of the present application is continuously introduced.
Fig. 2 shows a schematic flow chart of a data acquisition method according to an embodiment of the present application. Specifically, in an exemplary embodiment, as shown in fig. 2, the present embodiment provides a data acquisition method, including the following steps:
s210, acquiring a plurality of application programs determined in advance or in real time, wherein the service of each application program is independent from the service of other application programs;
s220, independently collecting the service data of each application program according to the data collection middleware service, and transmitting the collected service data to a database for summarizing;
and S230, uploading the service data summarized in the database to a server side based on a data unification rule determined in advance or in real time.
Therefore, the data can be directly inserted in the code compiling stage, the purpose of data acquisition is achieved, the complexity of data acquisition is greatly simplified, the workload of data acquisition is reduced, the data acquisition efficiency is improved, and the effect of data acquisition once and for all is basically achieved.
According to the above description, in an exemplary embodiment, the process of independently collecting the business data of each application program according to the data collection middleware service in step S220 includes:
presenting the source code of each application program by taking a preset computer code as a suffix to serve as a data acquisition middleware;
converting the presented source code into a class file containing a constant pool and a logic class by using a compiling tool;
selecting a linking tool associated with the compiling tool, converting the class file into an executable file based on the selected linking tool, and merging part of repeated data on the basis of the class file, so that the collected business data comprises constant pool, logic and header data.
In this embodiment, the preset computer code includes java code, the class file includes a class file, the executable file includes a dex file, the compiling tool includes an Android compiling tool, and the linking tool includes an Android linking tool. Specifically, as shown in fig. 3, in this embodiment, java Source Code (or Java Source Code) is first presented with Java as a suffix, that is, java Source Code may be presented with Java file as a suffix, which is used as data acquisition middleware service. And then, converting the source code presented by using the ". Java file" as a suffix into a class file by using an Android compiling tool or other compiling tools, namely converting the source code presented by using the ". Java file" as a suffix into a class file. Wherein, the converted class file comprises a constant pool, logic and the like. As an example, as shown in fig. 3, the converted Class file may include Constant Pool, class file information Class info, and Data. Meanwhile, the Android linking tool of the Android system can convert the class file into an executable file, namely the class file can be converted into a dex file by using the Android linking tool associated with the Android compiling tool; and merging part of the duplicated data on the basis of the class file. The dex file also includes various constant pools, data, logic, and Header Data or Header Data. As an example, as shown in FIG. 3, the dex file or executable file after merging partial duplicated Data may include Header, constant Pool, class definition Class1 definition, class definition Class2definition, and Data.
According to the above description, in another exemplary embodiment, the process of independently collecting the business data of each application program according to the data collection middleware service in step S220 may further include:
presenting the source code of each application program by taking a preset computer code as a suffix to serve as a data acquisition middleware;
converting the presented source code into a first class file containing a constant pool and a logic class by using a compiling tool, and directly implanting data into the first class file by using a data acquisition tool to form a second class file;
selecting a linking tool associated with the compiling tool, converting the second type of file into an executable file based on the selected linking tool, and merging part of repeated data on the basis of the second type of file so as to enable the collected business data to comprise constant pool, logic and header data.
In this embodiment, the preset computer code includes java code, the first class file and the second class file include class files, the executable file includes dex files, the compiling tool includes an Android compiling tool, the linking tool includes an Android linking tool, and the data collecting tool includes ASM instrumentation. Specifically, as shown in fig. 4, in this embodiment, java Source Code (or Java Source Code) is first presented with Java as a suffix, that is, java Source Code may be presented with Java file as a suffix, and the presented Java Source Code serves as data acquisition middleware service. And then, converting the source code presented by using the ". Java file" as a suffix into a class file by using an Android compiling tool or other compiling tools, namely converting the source code presented by using the ". Java file" as a suffix into a class file. And at this stage, the data acquisition tool can directly implant data in the class file, and the difference with the java source code is the data acquisition tool logic inserted automatically. In other words, at this stage, ASM instrumentation and the like can be used as a data acquisition tool to directly implant data into the class file to obtain a new class file. In this embodiment, the class file with no data embedded therein may be recorded as a first class file, and the class file with data embedded therein may be recorded as a second class file. Wherein, the converted class file comprises a constant pool, logic and the like. As an example, as shown in fig. 4, the converted Class file may include Constant Pool, class file information Class info, and Data. Meanwhile, the Android linking tool of the Android system can convert the class file into an executable file, namely the class file can be converted into a dex file by using the Android linking tool associated with the Android compiling tool; and merging part of the duplicated data on the basis of the class file. The dex file also includes various constant pools, data, logic, and Header Data or Header Data. As an example, as shown in FIG. 4, the executable file or dex file after merging partial duplicate Data may include a Header, a Constant Pool, a Class file definition Class1 definition, a Class file definition Class2definition, and Data.
As shown in fig. 5, the present application further provides an embodiment, where the embodiment provides a method for implementing data acquisition based on a Java bytecode instrumentation manner, and the method includes the following steps:
step 1: firstly, acquiring an Application program APP (Application, application program, APP for short) with independent Application services;
step 2: independently collecting service data of each application program APP by using data collection middleware;
and step 3: summarizing all collected business data into a database DB;
and 4, step 4: and uploading the service data summarized in the database DB to a Server (Server) end according to a data unification rule set in advance or in real time.
As an example, as shown in FIG. 3, the embodiment first presents each App source code with java as a suffix (developers can stand alone without paying attention to the data collection logic); then, when the Android compiling tool or other compiling tools are used for execution, the Android compiling tool or other compiling tools are converted into class files, wherein the class files comprise constant pools, logics and the like; and converting the class file into a dex file based on an Android link tool of the Android, and merging part of repeated Data on the basis of the class file, so that the dex file after merging the repeated Data also comprises various constant pools, data, logic, newly added Header Data and the like. As another example, as shown in fig. 4, the present embodiment first presents each App source code with java as a suffix; and then the file is converted into a class file when being executed by using an Android compiling tool or other compiling tools, wherein the class file comprises a constant pool, logic and the like. Meanwhile, at this stage, data acquisition tools and the like formed based on ASM instrumentation can be directly implanted into class files, and the difference between the data acquisition tools and java source codes is the logic of the data acquisition tools inserted automatically. And converting the class file into a dex file based on an Android link tool of the Android, and merging part of repeated Data on the basis of the class file, so that the dex file after merging the repeated Data also comprises various constant pools, data, logics, newly added Header Data and the like, and the logic and the byte code conversion process have no difference and only need to add a certain amount of Data acquisition Data.
Therefore, the present embodiment provides a technical method for realizing data acquisition based on the Java bytecode instrumentation method, which can use data acquisition with all software realized by Java so as to realize data acquisition with high efficiency and high performance. The data acquisition method based on Java bytecode instrumentation provided by this embodiment relates to Java source code, class file, dex file, and the like. Research personnel normally develop business functions (logic layer, app business service), java byte code insertion can be independent of current business codes (data acquisition middleware service) and interacts with App business service single items, and a byte code insertion tool is based on the data acquisition middleware and the App business service, can realize seamless code implantation and is an actual execution party of byte code insertion. After the research and development personnel develop and complete the service codes, the Android tool compiles, and the data acquisition tool can implant any required codes including acquisition format data and the like into the class codes in the compiling process. The collected data can be uniformly stored in a database or a file, and the service specified time is uploaded to a server through Http, so that the data can be uniformly collected. Therefore, the embodiment can directly insert data in the code compiling stage, realize the purpose of data acquisition, greatly simplify the complexity of data acquisition, reduce the workload of data acquisition, improve the data acquisition efficiency and basically realize the once-for-all data acquisition effect.
In summary, the present application provides a data acquisition method, which includes acquiring a plurality of application programs determined in advance or in real time, independently acquiring service data of each application program according to a data acquisition middleware service, and transmitting the acquired service data to a database for summarization; and uploading the service data summarized in the database to a server side based on a data unification rule determined in advance or in real time. Therefore, the method can directly insert data in the code compiling stage to achieve the purpose of data acquisition, greatly simplifies the complexity of data acquisition, not only reduces the workload of data acquisition, but also improves the data acquisition efficiency, and basically achieves the once-for-all data acquisition effect. Research personnel normally develop business functions (logic layer, app business service), java byte code insertion can be independent of current business codes (data acquisition middleware service) and interacts with App business service single items, and a byte code insertion tool is based on the data acquisition middleware and the App business service, can realize seamless code implantation and is an actual execution party of byte code insertion. After the research and development personnel develop and complete the service codes, the Android tool compiles, and the data acquisition tool can implant any required codes including acquisition format data and the like into the class codes in the compiling process. The collected data can be uniformly stored in a database or a file, and the service specified time is uploaded to a server through Http, so that the data can be uniformly collected. Therefore, the method can directly insert data in the code compiling stage to achieve the purpose of data acquisition, greatly simplifies the complexity of data acquisition, reduces the workload of data acquisition, improves the data acquisition efficiency, and basically achieves the once-for-all data acquisition effect.
As shown in fig. 6, the present application further provides a data acquisition system, which includes:
an application module 610, configured to determine, in advance or in real time, a plurality of applications that need to perform data acquisition, where a service of each application is independent of services of other applications;
the data acquisition module 620 is used for independently acquiring the service data of each application program according to the data acquisition middleware service and transmitting the acquired service data to the database for summarizing;
and a data uploading module 630, configured to upload the service data summarized in the database to a server according to a predetermined or real-time determined data unification rule.
Therefore, the data can be directly inserted in the code compiling stage, the purpose of data acquisition is achieved, the complexity of data acquisition is greatly simplified, the workload of data acquisition is reduced, the data acquisition efficiency is improved, and the once-for-all data acquisition effect is basically achieved.
According to the above description, in an exemplary embodiment, the process of the data collection module 620 collecting the business data of each application program independently according to the data collection middleware service includes:
presenting the source code of each application program by taking a preset computer code as a suffix to serve as a data acquisition middleware;
converting the presented source code into a class file containing a constant pool and a logic class by using a compiling tool;
selecting a linking tool associated with the compiling tool, converting the class file into an executable file based on the selected linking tool, and merging part of repeated data on the basis of the class file so as to enable the collected service data to comprise a constant pool, logic and header data.
In this embodiment, the preset computer code includes java code, the class file includes a class file, the executable file includes a dex file, the compiling tool includes an Android compiling tool, and the linking tool includes an Android linking tool. Specifically, as shown in fig. 3, in this embodiment, java Source Code (or Java Source Code) is first presented with Java as a suffix, that is, java Source Code may be presented with Java file as a suffix, which is used as data acquisition middleware service. And then, converting the source code presented by using the ". Java file" as a suffix into a class file by using an Android compiling tool or other compiling tools, namely converting the source code presented by using the ". Java file" as a suffix into a class file. Wherein, the converted class file comprises a constant pool, logic and the like. As an example, as shown in fig. 3, the converted Class file may include Constant Pool, class file information Class info, and Data. Meanwhile, the Android linking tool of the Android system can convert the class file into an executable file, namely the class file can be converted into a dex file by using the Android linking tool associated with the Android compiling tool; and merging part of the duplicated data on the basis of the class file. The dex file also includes various constant pools, data, logic, and Header Data or Header Data. As an example, as shown in FIG. 3, the executable file or dex file after merging partial duplicate Data may include a Header, a Constant Pool, a Class file definition Class1 definition, a Class file definition Class2definition, and Data.
According to the above description, in another exemplary embodiment, the process of the data collection module 620 independently collecting the business data of each application program according to the data collection middleware service includes:
presenting the source code of each application program by taking a preset computer code as a suffix to serve as a data acquisition middleware;
converting the presented source code into a first class file containing a constant pool and a logic class by using a compiling tool, and directly implanting data into the first class file by using a data acquisition tool to form a second class file;
selecting a linking tool associated with the compiling tool, converting the second type of file into an executable file based on the selected linking tool, and merging part of repeated data on the basis of the second type of file so as to enable the collected business data to comprise constant pool, logic and header data.
In this embodiment, the preset computer code includes java code, the first class file and the second class file include class files, the executable file includes dex files, the compiling tool includes an Android compiling tool, the linking tool includes an Android linking tool, and the data collecting tool includes ASM instrumentation. Specifically, as shown in fig. 4, in this embodiment, java Source Code (or Java Source Code) is first presented with Java as a suffix, that is, java Source Code may be presented with Java file as a suffix, and the presented Java Source Code serves as data acquisition middleware service. And then, converting the source code presented by using the ". Java file" as a suffix into a class file by using an Android compiling tool or other compiling tools, namely converting the source code presented by using the ". Java file" as a suffix into a class file. And at this stage, the data acquisition tool can directly implant data in the class file, and the difference with the java source code is the data acquisition tool logic inserted automatically. In other words, at this stage, ASM instrumentation and the like can be used as a data acquisition tool to directly implant data into the class file, so as to obtain a new class file. In this embodiment, the class file in which data is not embedded may be recorded as a first class file, and the class file in which data is embedded may be recorded as a second class file. Wherein, the converted class file comprises a constant pool, logic and the like. As an example, as shown in fig. 4, the converted Class file may include Constant Pool, class file information Class info, and Data. Meanwhile, the Android linking tool of the Android system can convert the class file into an executable file, namely the class file can be converted into a dex file by using the Android linking tool associated with the Android compiling tool; and merging part of the duplicated data on the basis of the class file. The dex file also includes various constant pools, data, logic, and Header Data or Header Data. As an example, as shown in FIG. 4, the executable file or dex file after merging partial duplicate Data may include a Header, a Constant Pool, a Class file definition Class1 definition, a Class file definition Class2definition, and Data.
The present application further provides an embodiment, which provides a system for implementing data acquisition based on a Java bytecode instrumentation manner, and the system is configured to perform the following steps:
step 1: firstly, acquiring an Application program APP (Application, application program, APP for short) with independent Application services;
step 2: independently collecting service data of each application program APP by using data collection middleware;
and step 3: summarizing all collected business data into a database DB;
and 4, step 4: and uploading the service data summarized in the database DB to a Server (Server) side according to a data unification rule set in advance or in real time.
As an example, as shown in FIG. 3, the embodiment first presents each App source code with java as a suffix (developers can stand alone without paying attention to the data collection logic); then, when the Android compiling tool or other compiling tools are used for execution, the Android compiling tool or other compiling tools are converted into class files, wherein the class files comprise constant pools, logics and the like; and converting the class file into a dex file based on an Android link tool of the Android, and merging part of repeated Data on the basis of the class file, so that the dex file after merging the repeated Data also comprises various constant pools, data, logic, newly added Header Data and the like. As another example, as shown in fig. 4, the present embodiment first presents each App source code with java as a suffix; and then the file is converted into a class file when being executed by using an Android compiling tool or other compiling tools, wherein the class file comprises a constant pool, logic and the like. Meanwhile, at this stage, data acquisition tools and the like formed based on ASM instrumentation can be directly implanted into class files, and the difference between the data acquisition tools and java source codes is the logic of the data acquisition tools inserted automatically. And converting the class file into a dex file based on an Android link tool of the Android, and merging part of repeated Data on the basis of the class file, so that the dex file after merging the repeated Data also comprises various constant pools, data, logic, newly added Header Data and the like, and the logic is not different from a byte code conversion process and has the advantage of adding some Data acquisition Data.
Therefore, the present embodiment provides a technical method for implementing data acquisition based on Java bytecode instrumentation, which can implement data acquisition with high efficiency and high performance by using data acquisition with all software implemented in Java. The data acquisition method based on Java bytecode instrumentation provided by this embodiment relates to Java source code, class file, dex file, and the like. Research personnel normally develop business functions (logic layer, app business service), java byte code insertion can be independent of current business codes (data acquisition middleware service) and interacts with App business service single items, and a byte code insertion tool is based on the data acquisition middleware and the App business service, can realize seamless code implantation and is an actual execution party of byte code insertion. After the research and development personnel develop and complete the service codes, the Android tool compiles, and the data acquisition tool can implant any required codes including acquisition format data and the like into the class codes in the compiling process. The collected data can be uniformly stored in a database or a file, and the service specified time is uploaded to a server through Http, so that the data can be uniformly collected. Therefore, the embodiment can directly insert data in the code compiling stage, realize the purpose of data acquisition, greatly simplify the complexity of data acquisition, reduce the workload of data acquisition, improve the data acquisition efficiency and basically realize the once-for-all data acquisition effect.
To sum up, the application provides a data acquisition system, which first obtains a plurality of application programs determined in advance or in real time, independently acquires service data of each application program according to data acquisition middleware services, and transmits the acquired service data to a database for summarizing; and uploading the service data summarized in the database to a server side based on a data unification rule determined in advance or in real time. Therefore, the system can directly insert data in the code compiling stage, achieves the purpose of data acquisition, greatly simplifies the complexity of data acquisition, not only reduces the workload of data acquisition, but also improves the efficiency of data acquisition, and basically achieves the effect of data acquisition once and for all. Research personnel normally develop business functions (logic layer, app business service), java byte code insertion can be independent of current business codes (data acquisition middleware service) and interacts with App business service single items, and a byte code insertion tool is based on the data acquisition middleware and the App business service, can realize seamless code implantation and is an actual execution party of byte code insertion. After the research and development personnel develop and complete the service codes, the Android tool compiles, and the data acquisition tool can implant any required codes including acquisition format data and the like into the class codes in the compiling process. The collected data can be uniformly stored in a database or a file, and the service specified time is uploaded to a server through Http, so that the data can be uniformly collected. Therefore, the system can directly insert data in the code compiling stage to achieve the purpose of data acquisition, greatly simplifies the complexity of data acquisition, reduces the workload of data acquisition, improves the data acquisition efficiency, and basically achieves the once-for-all data acquisition effect.
It should be noted that the data acquisition system provided in the foregoing embodiment and the data acquisition method provided in the foregoing embodiment belong to the same concept, and the specific manner in which each module performs operation has been described in detail in the method embodiment, and is not described here again. In practical applications, the data acquisition system provided in the above embodiment may distribute the above functions through different functional modules according to needs, that is, divide the internal structure of the system into different functional modules to complete all or part of the above described functions, which is not limited herein.
An embodiment of the present application further provides an electronic device, including: one or more processors; a storage device, configured to store one or more programs, which when executed by the one or more processors, cause the electronic device to implement the data acquisition method provided in the above-described embodiments.
FIG. 7 illustrates a schematic structural diagram of a computer system suitable for use in implementing the electronic device of an embodiment of the present application. It should be noted that the computer system 1000 of the electronic device shown in fig. 7 is only an example, and should not bring any limitation to the functions and the application scope of the embodiments of the present application.
As shown in fig. 7, the computer system 1000 includes a Central Processing Unit (CPU) 1001 that can perform various appropriate actions and processes, such as performing the methods described in the above embodiments, according to a program stored in a Read-Only Memory (ROM) 1002 or a program loaded from a storage portion 1008 into a Random Access Memory (RAM) 1003. In the RAM1003, various programs and data necessary for system operation are also stored. The CPU 1001, ROM 1002, and RAM1003 are connected to each other by a bus 1004. An Input/Output (I/O) interface 1005 is also connected to the bus 1004.
The following components are connected to the I/O interface 1005: an input section 1006 including a keyboard, a mouse, and the like; an output section 1007 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage portion 1008 including a hard disk and the like; and a communication section 1009 including a Network interface card such as a LAN (Local Area Network) card, a modem, or the like. The communication section 1009 performs communication processing via a network such as the internet. The driver 1010 is also connected to the I/O interface 1005 as necessary. A removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 1010 as necessary, so that a computer program read out therefrom is mounted into the storage section 1008 as necessary.
In particular, according to embodiments of the application, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising a computer program for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from the network through the communication part 1009 and/or installed from the removable medium 1011. When the computer program is executed by a Central Processing Unit (CPU) 1001, various functions defined in the system of the present application are executed.
It should be noted that the computer readable medium shown in the embodiments of the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM), a flash Memory, an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer-readable signal medium may comprise a propagated data signal with a computer-readable computer program embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. Each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present application may be implemented by software, or may be implemented by hardware, and the described units may also be disposed in a processor. Wherein the names of the elements do not in some way constitute a limitation on the elements themselves.
Another aspect of the present application also provides a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor of a computer, causes the computer to perform the data acquisition method as described above. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist separately without being incorporated in the electronic device.
Another aspect of the application also provides a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the data acquisition method provided in the above embodiments.
The above-described embodiments are merely illustrative of the principles and utilities of the present application and are not intended to limit the application. Any person skilled in the art can modify or change the above-described embodiments without departing from the spirit and scope of the present application. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical concepts disclosed in the present application shall be covered by the claims of the present application.
Claims (10)
1. A method of data acquisition, the method comprising the steps of:
acquiring a plurality of application programs determined in advance or in real time, wherein the service of each application program is independent of the service of other application programs;
independently collecting the service data of each application program according to the data collection middleware service, and transmitting the collected service data to a database for summarizing;
and uploading the service data summarized in the database to a server side based on a data unification rule determined in advance or in real time.
2. The data collection method of claim 1, wherein the process of independently collecting the business data of each application according to the data collection middleware service comprises:
presenting the source code of each application program by taking a preset computer code as a suffix to serve as a data acquisition middleware;
converting the presented source code into a class file containing a constant pool and a logic class by using a compiling tool;
selecting a linking tool associated with the compiling tool, converting the class file into an executable file based on the selected linking tool, and merging part of repeated data on the basis of the class file, so that the collected business data comprises constant pool, logic and header data.
3. The data collection method of claim 1, wherein the process of independently collecting the business data of each application according to the data collection middleware service comprises:
presenting the source code of each application program by taking a preset computer code as a suffix to serve as a data acquisition middleware;
converting the presented source code into a first class file containing a constant pool and a logic class by using a compiling tool, and directly implanting data into the first class file by using a data acquisition tool to form a second class file;
selecting a linking tool associated with the compiling tool, converting the second type of file into an executable file based on the selected linking tool, and merging part of repeated data on the basis of the second type of file so as to enable the collected business data to comprise constant pool, logic and header data.
4. The data collection method of claim 2, wherein the preset computer code comprises java code, the class file comprises a class file, the executable file comprises a dex file, the compilation tool comprises an Android compilation tool, and the linking tool comprises an Android linking tool.
5. The data collection method of claim 3, wherein the predetermined computer code comprises java code, the first class file and the second class file comprise class files, the executable file comprises a dex file, the compilation tool comprises an Android compilation tool, the linking tool comprises an Android linking tool, and the data collection tool comprises ASM instrumentation.
6. A data acquisition system, comprising:
the system comprises an application program module, a data acquisition module and a data acquisition module, wherein the application program module is used for determining a plurality of application programs needing data acquisition in advance or in real time, and the service of each application program is independent of the service of other application programs;
the data acquisition module is used for independently acquiring the service data of each application program according to the data acquisition middleware service and transmitting the acquired service data to the database for summarizing;
and the data uploading module is used for uploading the service data summarized in the database to a server side according to a data unification rule determined in advance or in real time.
7. The data collection system of claim 6, wherein the process of independently collecting the business data of each application according to the data collection middleware service comprises:
presenting the source code of each application program by taking a preset computer code as a suffix to serve as a data acquisition middleware;
converting the presented source code into a class file containing a constant pool and a logic class by using a compiling tool;
selecting a linking tool associated with the compiling tool, converting the class file into an executable file based on the selected linking tool, and merging part of repeated data on the basis of the class file, so that the collected business data comprises constant pool, logic and header data.
8. The data collection system of claim 6, wherein the process of independently collecting the business data of each application according to the data collection middleware service comprises:
presenting the source code of each application program by taking a preset computer code as a suffix to serve as a data acquisition middleware;
converting the presented source code into a first class file containing a constant pool and a logic class by using a compiling tool, and directly implanting data into the first class file by using a data acquisition tool to form a second class file;
selecting a linking tool associated with the compiling tool, converting the second type of file into an executable file based on the selected linking tool, and merging part of repeated data on the basis of the second type of file so as to enable the collected business data to comprise a constant pool, logic and header data.
9. An electronic device, characterized in that the electronic device comprises:
one or more processors;
storage means for storing one or more programs that, when executed by the one or more processors, cause the electronic device to implement the data acquisition method of any of claims 1-5.
10. A computer-readable storage medium, on which a computer program is stored which, when executed by a processor of a computer, causes the computer to carry out a data acquisition method as claimed in any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210852215.0A CN115145585A (en) | 2022-07-19 | 2022-07-19 | Data acquisition method, system, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210852215.0A CN115145585A (en) | 2022-07-19 | 2022-07-19 | Data acquisition method, system, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115145585A true CN115145585A (en) | 2022-10-04 |
Family
ID=83412089
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210852215.0A Pending CN115145585A (en) | 2022-07-19 | 2022-07-19 | Data acquisition method, system, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115145585A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110173501A1 (en) * | 2008-09-26 | 2011-07-14 | Microsoft Corporation | Memory management techniques selectively using mitigations to reduce errors |
| CN109144834A (en) * | 2017-06-27 | 2019-01-04 | 深圳市Tcl高新技术开发有限公司 | Acquisition method and device, the Android system and terminal device of user behavior data |
| CN110134400A (en) * | 2019-04-02 | 2019-08-16 | 北京奇艺世纪科技有限公司 | Data capture method, device, electronic equipment and computer readable storage medium |
| US20200125668A1 (en) * | 2018-10-19 | 2020-04-23 | Oracle International Corporation | Language interoperable runtime adaptable data collections |
| CN111190573A (en) * | 2018-11-14 | 2020-05-22 | 北京字节跳动网络技术有限公司 | Application program point burying method and device and electronic equipment |
| CN111897540A (en) * | 2020-07-23 | 2020-11-06 | 平安证券股份有限公司 | Data collection method and device based on private cloud, server and storage medium |
| CN111913756A (en) * | 2020-07-08 | 2020-11-10 | 北京达佳互联信息技术有限公司 | Application program optimization method, device and system, electronic equipment and storage medium |
| CN112306501A (en) * | 2020-11-30 | 2021-02-02 | 杭州网易云音乐科技有限公司 | Business data acquisition method and device, storage medium and computing equipment |
-
2022
- 2022-07-19 CN CN202210852215.0A patent/CN115145585A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110173501A1 (en) * | 2008-09-26 | 2011-07-14 | Microsoft Corporation | Memory management techniques selectively using mitigations to reduce errors |
| CN109144834A (en) * | 2017-06-27 | 2019-01-04 | 深圳市Tcl高新技术开发有限公司 | Acquisition method and device, the Android system and terminal device of user behavior data |
| US20200125668A1 (en) * | 2018-10-19 | 2020-04-23 | Oracle International Corporation | Language interoperable runtime adaptable data collections |
| CN111190573A (en) * | 2018-11-14 | 2020-05-22 | 北京字节跳动网络技术有限公司 | Application program point burying method and device and electronic equipment |
| CN110134400A (en) * | 2019-04-02 | 2019-08-16 | 北京奇艺世纪科技有限公司 | Data capture method, device, electronic equipment and computer readable storage medium |
| CN111913756A (en) * | 2020-07-08 | 2020-11-10 | 北京达佳互联信息技术有限公司 | Application program optimization method, device and system, electronic equipment and storage medium |
| CN111897540A (en) * | 2020-07-23 | 2020-11-06 | 平安证券股份有限公司 | Data collection method and device based on private cloud, server and storage medium |
| CN112306501A (en) * | 2020-11-30 | 2021-02-02 | 杭州网易云音乐科技有限公司 | Business data acquisition method and device, storage medium and computing equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110019350B (en) | Data query method and device based on configuration information | |
| CN112104709B (en) | Intelligent contract processing method, device, medium and electronic equipment | |
| CN109815107B (en) | Method and device for automatic testing | |
| CN104049991A (en) | Method and system for converting network applications into mobile applications | |
| CN111367791B (en) | Method, device, medium and electronic equipment for generating test case | |
| CN112631588A (en) | File generation method and device, electronic equipment and computer readable medium | |
| CN112860662B (en) | Automatic production data blood relationship establishment method, device, computer equipment and storage medium | |
| CN113992641B (en) | Data processing method, device, equipment and storage medium | |
| CN111767498B (en) | Method and device for realizing file information sharing | |
| CN115145585A (en) | Data acquisition method, system, electronic equipment and storage medium | |
| CN116781586A (en) | gRPC flow analysis method, device, equipment and medium | |
| CN116204428A (en) | Test case generation method and device | |
| CN115756448A (en) | Method, device, equipment and medium for acquiring vehicle cloud data acquisition system architecture | |
| CN110858240A (en) | Front-end module loading method and device | |
| CN111796878B (en) | Resource splitting and loading method and device applied to single-page application | |
| CN116263690A (en) | Method and device for virtual machine to read data from external system and relevant written data | |
| CN112433752A (en) | Page parsing method, device, medium and electronic equipment | |
| CN111008346A (en) | Data acquisition method and device | |
| CN116880901B (en) | Application page parsing method, device, electronic device and computer-readable medium | |
| CN114968605B (en) | Data synchronization method, device, equipment, medium and program product | |
| CN113468065B (en) | Test case operation method and device | |
| CN110825438A (en) | Method and device for simulating data processing of artificial intelligence chip | |
| CN111736805B (en) | Method and device for processing Excel table | |
| CN116828073A (en) | Message conversion method, device, electronic equipment and computer readable storage medium | |
| CN113760348A (en) | Code packaging method and device, electronic equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |