CN105589881B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN105589881B CN105589881B CN201410573710.3A CN201410573710A CN105589881B CN 105589881 B CN105589881 B CN 105589881B CN 201410573710 A CN201410573710 A CN 201410573710A CN 105589881 B CN105589881 B CN 105589881B
- Authority
- CN
- China
- Prior art keywords
- office data
- shared memory
- static
- data
- static office
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 24
- 230000003068 static effect Effects 0.000 claims abstract description 429
- 238000000034 method Methods 0.000 claims abstract description 106
- 230000008569 process Effects 0.000 claims abstract description 63
- 230000006870 function Effects 0.000 claims description 31
- 230000004044 response Effects 0.000 abstract description 6
- 230000008676 import Effects 0.000 description 7
- 201000007694 Athabaskan brainstem dysgenesis syndrome Diseases 0.000 description 4
- 208000020006 human HOXA1 syndromes Diseases 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data processing method and a data processing device, wherein the data processing method comprises the following steps: obtaining static office data required by a service from a physical database, loading the static office data into a shared memory, exporting the static office data loaded into the shared memory to a temporary file, and importing the static office data into a file database through the temporary file for first backup; when the static office data required by the service in the physical database is determined to be updated, copying the static office data in the shared memory into the file database for second backup, acquiring the updated static office data and loading the updated static office data into the shared memory; and exporting the updated static office data loaded into the shared memory to a temporary file, and importing the updated static office data into a file database through the temporary file to update the static office data backed up by the first backup. The invention can not only reduce the burden of the physical library, but also shorten the response processing time of the business process.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a data processing method and apparatus.
Background
The telecommunication service system is stored with a large amount of static office data, such as city information and user mobile phone number attribution information, which provides data support for the charging service of the telecommunication service system.
Currently, static office data is stored in a physical library, such as an Oracle database. However, a large number of service processes that need to query such static office data information may be present at the same time in the charging service, and the database needs to be accessed every time the service processes respond, and obviously, frequent database access is required for responding to the service processes.
It can be seen that, in the existing scheme of storing static office data in a physical library and acquiring data required by a service request by querying the physical library, a large amount of I/O operations are performed on the physical library, the operation burden of the physical library is heavy, and the response processing of a service process is time-consuming.
Disclosure of Invention
The invention provides a data processing method and a data processing device, which are used for solving the problems that the burden of a physical library is heavy and the response processing of a business process is long in time in the conventional data processing scheme.
In order to solve the above problem, the present invention discloses a data processing method, comprising: obtaining static office data required by the service from a relational database and/or a file database; loading the obtained static office data into a shared memory, exporting the static office data loaded into the shared memory to a temporary file, and importing the static office data into a file database through the temporary file for first backup; when determining that static office data required by the service in a relational database and/or a file database is updated, copying the static office data in the shared memory into the file database for second backup, acquiring the updated static office data, and loading the updated static office data into the shared memory so as to simultaneously store the static office data before updating and the updated static office data in the shared memory; and exporting the updated static office data loaded into the shared memory to the temporary file, and importing the updated static office data into the file database through the temporary file to update the static office data backed up by the first backup.
Preferably, the step of obtaining updated static office data includes: when the updated static office data is obtained from the relational database, searching the updated static office data in the relational database through a header file for reading the data of the relational database, and importing the searched updated static office data into the temporary file; and when the updated static office data is obtained from the file database, searching the updated static office data through a database tool arranged in the file database to obtain the updated static office data.
Preferably, the step of searching for the updated static office data in the relational database by using the header file for reading the data of the relational database includes: and positioning a relational database to be searched by an OTL method in the header file so as to search the updated static office data from the relational database.
Preferably, the step of loading the obtained static office data into a shared memory includes: judging whether the created shared memory exists or not; if the static office data exists, the created shared memory is directly connected, and the static office data is loaded according to a set rule; if the static office data does not exist in the shared memory, a shared memory is created through the Shmaget function, an entrance function is called to create all static class objects in the newly-created shared memory, and the static office data is loaded into the created static class objects.
Preferably, the step of loading the static office data according to a set rule includes: judging whether the remaining memory space in the hooked shared memory is enough to load the static office data or not; if not, applying for a new shared memory space to an operating system, calling an entry function to create all static class objects of the new shared memory space, loading the static office data in the shared memory from an address behind the tail address of the loaded data, and loading the static office data by using the idle static class objects in the original shared memory space of the shared memory and the newly created static class objects; and if so, directly hooking the shared memory, and loading the static office data from an address behind the tail address of the loaded data in the shared memory so as to load the static office data into an idle static class object in the shared memory.
Preferably, after the step of loading the updated static office data into the shared memory, the method further includes: judging whether business process connection exists or not; if the static office data exists in the service process, after the set time, the data version ID of the service process is forcibly converted into a new data version ID, so that the updated shared memory area where the static office data is located is used for providing data for the service process connection, wherein the data version ID and the shared memory area have a corresponding relation.
In order to solve the above problem, the present invention also discloses a data processing apparatus, comprising: the first acquisition module is used for acquiring static office data required by the service from a relational database and/or a file database; the first loading module is used for loading the acquired static office data into a shared memory, exporting the static office data loaded into the shared memory to a temporary file, and importing the static office data into a file database through the temporary file for first backup; the updating module is used for copying the static office data in the shared memory into a file database for second backup when determining that the static office data required by the service in the relational database and/or the file database is updated, acquiring the updated static office data, and loading the updated static office data into the shared memory so as to simultaneously store the static office data before updating and the updated static office data in the shared memory; and the backup module is used for exporting the updated static office data loaded into the shared memory to the temporary file, and importing the updated static office data into the file database through the temporary file to update the static office data backed up by the first backup.
Preferably, when the update module obtains the updated static office data from the relational database, the updated static office data in the relational database is searched through a header file for reading the data of the relational database, and the searched updated static office data is imported into the temporary file; and when the updating module acquires the updated static office data from the file database, searching the updated static office data through a database tool built in the file database to acquire the updated static office data.
Preferably, when the updated static office data in the relational database is searched for by a header file for reading the data of the relational database, the updating module locates the relational database to be searched for by an OTL method in the header file, so as to search for the updated static office data from the relational database.
Preferably, the first loading module comprises: the judging module is used for judging whether the created shared memory exists or not; the first execution module is used for directly hooking the created shared memory and loading the static office data according to a set rule when the judgment result of the judgment module is that the shared memory exists; a second execution module, configured to create a shared memory through the Shmaget function, call an entry function to create all static class objects in the newly created shared memory, and load the static office data into the created static class objects when the determination result of the determination module is absent; and the data export module is used for exporting the static office data loaded into the shared memory to a temporary file, and importing the static office data into a file database through the temporary file for first backup.
Compared with the prior art, the invention has the following advantages:
the data processing scheme provided by the invention loads static office data required by the service from the relational database and/or the file database to the shared memory, and backups the static office data required by the service in the file database. When the static office data required by the service is updated, the updated static office data is loaded to the shared memory, and then the updated static office data is backed up in the file database. Therefore, through the data processing scheme provided by the invention, on one hand, static office data required by business in a relational database and/or a file database, namely a physical library, is loaded into a shared memory, a business process can acquire the required static office data from the shared memory without accessing the physical library, and only the static office data in the shared memory needs to be accessed when being updated, so that the input and output of the physical library, namely the I/O operation times can be reduced, and the burden of the physical library is reduced. On the other hand, according to the data processing scheme provided by the invention, the service process directly acquires data from the shared memory, and the time required for accessing the shared memory is shorter than the time required for accessing the physical library, so that the response processing time of the service process can be shortened. In another aspect, according to the data processing scheme provided by the present invention, the static office data loaded into the shared memory and the updated static office data are backed up in the file database, so as to provide data required by the shared memory when the shared memory has a problem.
Drawings
FIG. 1 is a flow chart of steps of a data processing method according to a first embodiment of the invention;
FIG. 2 is a flow chart of steps of a data processing method according to a second embodiment of the invention;
fig. 3 is a block diagram of a data processing apparatus according to a third embodiment of the present invention;
FIG. 4 is an architecture diagram of a data processing system according to a fourth embodiment of the present invention;
FIG. 5 is a flow chart illustrating steps for performing a static office data update using the data processing system described in FIG. 4.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
Example one
Referring to fig. 1, a flowchart illustrating steps of a data processing method according to a first embodiment of the present invention is shown.
The data processing method of the embodiment comprises the following steps:
step S102: and acquiring static office data required by the service from a relational database and/or a file database.
The data processing method in this embodiment is applicable to various service systems, for example: a telecommunications service system, and static office data is data that provides service support. Taking the telecommunication service as an example, the static office data is the data satisfying the telecommunication service, such as: data required for long distance and local telephone service in fixed network, data required for mobile network telephone and data service, and data required for satellite communication and satellite mobile communication service, etc.
When obtaining static office data, extraction needs to be performed according to business requirements, for example: the service system only provides the domestic long-distance and local telephone service of the fixed network, so that when the static office data required by the service is acquired, only the static office data required by the domestic long-distance and local telephone service of the fixed network is extracted.
In practical implementation, only the static office data required by the service may be acquired from the relational database, only the static office data required by the service may be acquired from the file database, a part of the static office data may be acquired from the relational database, and the static office data required by the service and not extracted from the relational database may be acquired from the file database. The specific extraction scheme can be set according to the storage position of the required static office data. The relational database and the file database in this embodiment are both physical databases.
Step S104: and loading the obtained static office data into a shared memory.
Step S106: and exporting the static office data loaded into the shared memory to a temporary file, and importing the static office data into a file database through the temporary file for first backup.
Step S108: and when determining that the static office data required by the service in the relational database and/or the file database is updated, copying the static office data in the shared memory into the file database for second backup.
The first backup and the second backup of static office data required by the service can be in different folders of the same file database and can be respectively stored in the two file databases. For example: the first backup is stored in a first file database and the second backup is stored in a second file database, the first file database and the second file database being different databases.
The first backup of the static office data can provide the static office data for the shared memory when the shared memory has a problem, and the updated static office data can cover the first backup after the updated static office data is loaded into the shared memory, and at this moment, the updated static office data is provided for the shared memory when the shared memory has a problem. The static office data before updating is provided by the second backup of the static office data before loading the updated office data, and as can be seen, the first backup is to provide the updated static office data for the shared memory, and the second backup is to provide the static office data before updating for the shared memory.
Step S110: and acquiring the updated static office data, and loading the updated static office data into the shared memory so as to simultaneously store the static office data before updating and the updated static office data in the shared memory.
The updated static office data can be obtained from only the relational database or only the file database, or can be obtained from two databases, wherein a part of the updated static office data is respectively obtained from the two databases. The static office data before updating and the static office data after updating are simultaneously stored in the shared memory, wherein the static office data before updating provides data support for a service process accessed during the static office data updating period, and the static office data after updating provides data support for a service accessed after the static office data is updated, so that the consistency of the data accessed by the same process is ensured.
Step S112: and exporting the updated static office data loaded into the shared memory to a temporary file, and importing the updated static office data into a file database through the temporary file to update the static office data backed up by the first backup.
By the data processing method provided by the embodiment, the static office data required by the service is loaded to the shared memory from the physical library, and the static office data required by the service is backed up in the file database. When the static office data required by the service is updated, the updated static office data is loaded to the shared memory, and then the updated static office data is backed up in the file database. Therefore, through the data processing method provided by the invention, on one hand, the static office data required by the service in the physical library is loaded into the shared memory, the service process can acquire the required static office data from the shared memory, the physical library does not need to be accessed, and the physical library only needs to be accessed when the static office data in the shared memory is updated, so that the I/O times of the physical library can be reduced, and the burden of the physical library is reduced. On the other hand, by the data processing method provided by the invention, the service process directly acquires the data from the shared memory, and the time required for accessing the shared memory is shorter than the time required for accessing the physical library, so that the response processing time of the service process can be shortened. In another aspect, by the data processing method provided by the present invention, the static office data loaded into the shared memory and the updated static office data are backed up in the file database, so as to provide the data required by the shared memory when the shared memory has a problem.
Example two
Referring to fig. 2, a flowchart illustrating steps of a data processing method according to a second embodiment of the present invention is shown.
The data processing method of the embodiment specifically comprises the following steps:
step S202: the office data manager obtains static office data needed by the business from a relational database and/or a file database.
The office data manager selects from which database to fetch the static office data according to the storage location of the static office data required by the service.
When the office data manager acquires office data from the relational database, the office data manager can directly acquire static office data required by the service from the relational database, and can also search the static office data required by the service stored in the relational database through a header file for reading the data of the relational database, import the searched static office data into a temporary file, and acquire the static office data required by the service from the temporary file. For example: the office data manager can adopt an Otlv4.h header file and use an OTL mode to read data in the relational database. Of course, the office data manager may also read the data in the relational database through other header files, and those skilled in the art may set the header files according to actual requirements, which is not specifically limited in this application.
When the bureau data manager obtains the static bureau data needed by the service from the file database, the static bureau data needed by the service is searched through the database tool arranged in the file database so as to obtain the static bureau data needed by the service.
Step S204: the office data manager judges whether the created shared memory exists or not; if yes, go to step S206; if not, step S208 is directly executed.
After acquiring static office data required by a service, the office data manager loads the acquired static office data into a shared memory, and before loading, whether the created shared memory exists needs to be judged.
Step S206: when the office data manager determines that the created shared memory exists, the created shared memory is directly attached, static office data is loaded according to a set rule, and then step S210 is executed.
In this step, if the office data manager does not load static office data into the shared memory for the first time, it indicates that the created shared memory exists, and at this time, the created shared memory may be hooked through the Shmat method. Of course, the created shared memory may also be attached by other methods, and the application does not specifically limit this.
In practical implementation, a preferred method for loading static office data according to a set rule is as follows:
the office data manager judges whether the remaining memory space in the hooked shared memory is enough to load the static office data.
When the remaining memory space is judged to be insufficient for loading the static office data, applying for a new shared memory space to the operating system, calling an entry function to create all static class objects of the new shared memory space (the entry function may be provided by a dynamic library in the system, for example, an entry function provided by an ABSD dynamic library which is a dynamic library of a software design method based on an architecture), loading the static office data in the shared memory from an address after the last address of the loaded data, and loading the static office data by using the idle static class objects in the original shared memory space of the shared memory and the newly created static class objects. . For example: before the static office data is stored, 5 static class objects exist in the shared memory space, and only two idle static class objects exist, but the two idle static class objects cannot load all the static office data. At this time, a new shared memory space needs to be applied to the operating system, and if 5 static class objects are also created for the new shared memory space, when static office data is loaded, static data is loaded into the original two static class objects in the shared memory space and the newly created static class object.
And when the residual memory space is judged to be enough to load the static office data, directly hooking the shared memory, and loading the static office data from the address behind the tail address of the loaded data in the shared memory so as to load the static office data into the idle static class object in the shared memory.
It should be noted that, when the office data manager loads the static office data into the shared memory, the office data manager is not limited to start to load the static office data at an address after the end address of the loaded data, and a person skilled in the art may select an address to start loading according to actual requirements, for example: the static office data can be selected to be loaded from a newly created static class object, which is not specifically limited in this application.
Step S208: when the office data manager judges that the created shared memory does not exist, the shared memory is created through the Shmaget function, the entry function is called to create all static class objects in the newly created shared memory, the static office data is loaded into the created static class objects, and then step S210 is executed.
Wherein the entry function may be an entry function provided by an ABSD dynamic library.
Step S210: and the office data manager exports the static office data loaded into the shared memory to a temporary file, and imports the static office data into a file database through the temporary file for first backup.
Step S212: and when determining that the static office data required by the service in the relational database and/or the file database is updated, copying the static office data in the shared memory into the file database for second backup.
In this step, before loading the updated static office data into the shared memory, the static office data in the shared memory is backed up. When the shared memory has problems, the static office data can be directly obtained from the backup file.
Step S214: the office data manager obtains updated static office data from the relational database and/or the file database.
In practice, preferably, when the office data manager obtains the updated static office data from the relational database, the office data manager searches for the updated static office data in the relational database through the header file for reading the data in the relational database, and imports the searched updated static office data into the temporary file. The office data manager can locate the relational database to be searched by using an OTL method through an Otlv4.h header file so as to search the updated static office data from the relational database. Of course, the relational database to be searched can be located through other header files, and the updated static office data can be searched from the relational database.
It should be noted that, for the OTL method, those skilled in the art have the ability to implement the method, and therefore, detailed description of the implementation method is omitted here.
In practice, preferably, when the office data manager obtains the updated static office data from the file database, the office data manager searches the updated static office data through the database tool built in the file database to obtain the updated static office data.
It should be noted that the office data manager may also directly obtain the updated static office data from the relational database without using a temporary file.
Step S216: the office data manager judges whether the residual memory space in the shared memory is enough to load the updated static office data; if yes, go to step S218; if not, go to step S220.
Step S218: when the remaining memory space in the shared memory of the office data manager is not enough to load the updated static office data, a new shared memory space is applied to the operating system to load the updated static office data, and step S222 is executed.
Specifically, after applying for a new shared memory space to the operating system, the office data manager calls an entry function to create all static class objects of the new shared memory space, where the entry function may be provided by a dynamic library in the system, loads updated static office data in the shared memory from an address after the last address of the loaded static office data, and loads the updated static office data using a free static class object in the original shared memory space of the shared memory and the newly created static class object.
It should be noted that the office data manager may also directly load the updated static office data into the newly created static class object, instead of loading the updated static office data into the original static class object of the shared memory.
Step S220: when the remaining memory space in the shared memory of the office data manager is sufficient for loading the updated static office data, the shared memory is directly hooked to load the updated static office data, and step S222 is executed.
Specifically, when the updated static office data is loaded in the shared memory, the updated static office data is loaded in the shared memory from an address after the end address of the loaded static office data, so as to load the updated static office data into the idle static class object in the shared memory.
It should be noted that, when the office data manager loads the updated static office data into the shared memory, the office data manager is not limited to start loading the updated static office data at an address after the end address of the loaded static office data, and a person skilled in the art may select an address to start loading according to actual requirements, for example: the updated static office data may be loaded from a newly created static class object, which is not specifically limited in this application.
Step S222: and the office data manager exports the updated static office data loaded into the shared memory to a temporary file, and imports the updated static office data into a file database through the temporary file to update the static office data backed up by the first backup.
It should be noted that, in this embodiment, the updated static office data is imported into the file database to update the static office data backed up by the first backup, and in actual implementation, the new static office data may also be imported into another file database without updating the static office data backed up by the first backup. Alternatively, even if the updated static office data is imported into the same file database, the static office data backed up by the first backup is not updated.
Step S224: the office data manager judges whether business process connection exists or not; if yes, go to step S226; if not, executing the setting program.
And when the service process connection is the process connection established by the service system for the access of the user when the user accesses the service system. Wherein, the business process connection comprises: the data version ID. The memory area for storing the static office data in the shared memory also has a version, the data version ID has a corresponding relation with the shared memory area, the shared memory area for storing the static office data before updating corresponds to the old version data ID, and the shared memory area for storing the static office data after updating corresponds to the new version data ID.
The setting program is waiting for business process connection, and when a business process is connected, the updated static office data shared memory area is directly adopted to provide data support for the business process connection.
Step S226: when the office data manager judges that the business process connection exists, after the set time, the business process data version ID is converted into a new data version ID forcibly, so that the updated static office data is located in the shared memory area to provide data for the business process connection. In practical implementation, the set time can be set by a person skilled in the art according to practical requirements, for example: the updated static office data may be set to be forcibly switched after being updated into the shared memory for 10 seconds, and may also be set to be 15 seconds, 20 seconds, and the like, which is not specifically limited in the present application.
In the shared memory, the shared memory area storing the static office data before and after update corresponds to different data versions, and the same business process also has a data version ID, that is, the data acquired by the business process needs to be acquired from the corresponding shared memory area according to the data version ID, so as to ensure the consistency of the data versions requested to be acquired by the same business process.
And when the shared memory area for providing data for the business process is switched forcibly, the business process data version ID is converted into a new data version ID, so that the program can provide data for the business process connection through the shared memory area corresponding to the new static office data corresponding to the new data version ID and the updated static office data, and the forced switching of the business process is completed.
And after the service process is switched to the shared memory area where the updated static local data is located, releasing the shared memory area where the static local data is located in the shared memory for the next static local data updating.
Through the data processing method provided by the embodiment, on one hand, static office data required by the service in the physical library is loaded into the shared memory, the service process can acquire the required static office data from the shared memory, the physical library does not need to be accessed, and the physical library only needs to be accessed when the static office data in the shared memory is updated, so that the I/O frequency of the physical library can be reduced, and the burden of the physical library is reduced. On the other hand, by the data processing method provided by the invention, the service process directly acquires the data from the shared memory, and the time required for accessing the shared memory is shorter than the time required for accessing the physical library, so that the response processing time of the service process can be shortened. In another aspect, by the data processing method provided by the present invention, the static office data loaded into the shared memory and the updated static office data are backed up in the file database, so as to provide the data required by the shared memory when the shared memory has a problem. On the other hand, by the data processing method provided by the embodiment, when the shared memory area is switched, when the service process exists, the service process is switched after waiting for a period of time and having no data request, so that the consistency of data access of the same process is ensured, and the dynamic update of static data required by the service is supported while the real-time service processing is not influenced.
EXAMPLE III
Referring to fig. 3, a block diagram of a data processing apparatus according to a third embodiment of the present invention is shown.
The data processing apparatus of the present embodiment includes: a first obtaining module 302, configured to obtain static office data required by a service from a relational database and/or a file database; a first loading module 304, configured to load the obtained static office data into a shared memory, export the static office data loaded into the shared memory to a temporary file, and import the static office data into a file database through the temporary file to perform a first backup; the updating module 306 is configured to copy the static office data in the shared memory into the file database for second backup when it is determined that the static office data required by the service in the relational database and/or the file database is updated, acquire the updated static office data, and load the updated static office data into the shared memory, so as to simultaneously store the static office data before updating and the updated static office data in the shared memory; the backup module 308 is configured to export the updated static office data loaded into the shared memory to a temporary file, and import the updated static office data into a file database through the temporary file to update the static office data backed up by the first backup.
Preferably, when the updating module 306 obtains the updated static office data from the relational database, the updated static office data in the relational database is searched through a header file for reading the data of the relational database, and the searched updated static office data is imported into the temporary file; when the update module 306 obtains the updated static office data from the file database, the updated static office data is searched by the database tool built in the file database to obtain the updated static office data.
Preferably, when searching the updated static office data in the relational database through the header file for reading the data of the relational database, the updating module 306 locates the relational database to be searched through the OTL method in the header file, so as to search the updated static office data from the relational database.
On the basis of the data processing apparatus provided in this embodiment, the apparatus may be further optimized to obtain a more preferable data processing apparatus, and the specific optimization method is as follows:
preferably, the first load module 304 includes: a determining module 3042, configured to determine whether there is a created shared memory; a first executing module 3044, configured to directly hook the created shared memory and load static office data according to a set rule when the determination result of the determining module is present; a second executing module 3046, configured to create a shared memory through the Shmaget function, call an entry function to create all static class objects in the newly created shared memory, and load static office data into the created static class objects when the determination result of the determining module is absent; the data export module 3048 is configured to export the static office data loaded into the shared memory to a temporary file, and import the static office data into the file database through the temporary file to perform the first backup.
Preferably, the first execution module 3044 includes: a memory space determining module 30442, configured to determine whether the remaining memory space in the hooked shared memory is enough to load the static office data; a newly-built module 30444, configured to, if the determination result of the memory space determining module 30442 is negative, apply for a new shared memory space to the operating system, invoke an entry function to create all static class objects of the new shared memory space, load static office data in the shared memory from an address after the last address of the loaded data, and load the static office data by using the idle static class object in the original shared memory space of the shared memory and the newly-built static class object; a memory direct hooking module 30446, configured to hook the shared memory directly when the determination result of the memory space determining module 30442 is yes, and load static office data in the shared memory from an address after the end address of the loaded data, so as to load the static office data into an idle static class object in the shared memory.
Preferably, the data processing apparatus in this embodiment further includes: a process connection determining module 310, configured to determine whether a service process connection exists after the update module loads the updated static office data into the shared memory; a first process switching module 312, configured to, when the determination result of the process connection determining module 310 is present, after a set time, forcibly convert the data version ID of the business process into a new data version ID, so as to provide data for business process connection through the shared memory area where the updated static office data is located; and the data version ID and the shared memory area have a corresponding relation.
The data processing apparatus of this embodiment is used to implement the corresponding device cross-version upgrading method in the foregoing multiple method embodiments, and has the beneficial effects of being implemented by the corresponding method, which are not described herein again.
Example four
In this embodiment, a data processing system including the data processing apparatus described in the third embodiment and a specific flow of data processing performed by the data processing system are described.
As shown in fig. 4, the data processing system in the present embodiment includes: the Database 402, i.e. a relational Database, is used for storing static office data, the Sqlite disk file 404, i.e. a file Database, is used for storing static office data, the office data manager 406, i.e. a data processing device, the shared memory 408 is used for loading the static office data, the Sqlite backup disk file 410 and the business process 412.
The data processing system in FIG. 4 has two basic functions: the function of loading the static office data required by the service to the shared memory and the function of dynamically updating the static office data.
In this embodiment, when the data processing system loads the static office data required by the service into the shared memory, the office data manager extracts the static office data required by the service from a Database DB for short and a Sqlite disk file, and loads the extracted static office data into the shared memory. And meanwhile, storing the acquired static office data to a Sqlite disk file for backup, namely, loading the static office data required by the service to the shared memory. When a business process needs to acquire data, the data can be directly acquired from the shared memory.
What has been described above is only the process of loading the static office data required by the service to the shared memory, and since the static office data required by the service is updated, the static office data in the shared memory needs to be updated. As shown in fig. 4, when performing data update, the office data manager needs to extract the loaded static office data from the shared memory, and then backup the static office data to the Sqlite backup disk file, and a specific flow of loading the static office data and dynamically updating is described with reference to fig. 5.
In a specific implementation, the loading and dynamic updating of the static office data are performed through a program, however, no matter the loading or updating of the static office data, the static office data needs to be acquired from the DB database and the Sqlite disk file, and when the static office data is imported from the Sqlite disk file, a Sqlite built-in database tool needs to be used, and a specific query method is as follows:
CBSSqliteStmt stmtSelect；
if (stmtselect. open (this- > m _ strSelectSqlite. c _ str ()), connSqlite | ═ SQLITE _ OK), where m _ strSelectSqlite is a standard SQL query statement.
When importing office data from a DB database, an Otlv4.h header file needs to be referred to, and data is read by using an OTL mode, and the specific method is as follows:
otl_stream
otlSelect(OTL_READ_BUFFER_SIZE,m_strSelectSqlDB.c_str(),con)。
when the static office data required by the service to be loaded into the shared memory is acquired by the method, the acquired static office data needs to be loaded into the shared memory. If the program is run for the first time, that is, there is no created shared memory, then at this time, the Shmget method needs to be used for creating the shared memory, and the specific method is as follows: shmid ═ shmlet (this- > base _ shm _ key, round _ bytes + shm _ table _ offset, this- > file _ perms | IPC _ create | IPC _ EXCL).
If the program is not operated for the first time, that is, the shared memory already exists, the program only needs to hook the shared memory created by the office data memory management before, that is, the shared memory keyid configured in the configuration file, and a Shmat method needs to be used for hooking the shared memory, and the specific method is as follows: this- > base _ addr ═ shmat (shmid, render _ cast < char > (this- > base _ addr), 0).
After the program creates or attaches the shared memory, the obtained static office data is added after the address of the static office data loaded in the shared memory, and if the shared memory space is not enough, the program can apply for a new shared memory space from the operating system. The management of the local data memory calls an entry function reg _ data _ fun provided by the ABSD dynamic library to create all static class objects in the shared memory, so as to load the acquired static local data. The method for calling the ABSD dynamic library to provide the entry function by the program is as follows:
pStaticDataFunc＝(STATIC_DATA_FUNC)dlsym(libinfo.lib_handle,
itr->second.lib_enterfunc.c_str())；
wherein pStaticDataFunc is a function pointer, in which an entry function to a dynamic library is located by using a Dlsym method, wherein the Dlsym method is a method of returning an address corresponding to a symbol according to a dynamic link library operation handle.
It should be noted that, at present, the number of shared memory blocks is unified into at most 30 blocks, and the size of each block is 32M, and in actual implementation, the application for the shared memory space is actually the application for the shared memory block. When the data is less, the number of blocks which can be actually needed is applied, and the shared memory of the system cannot be wasted. It should be noted that the first shared memory block of the program application is a control block, and the limited 30 shared memory blocks are not counted.
After loading the static office data into the shared memory, when the static office data is updated, the program will sequentially call the following methods:
CEventProc:load_Data2Shm(CMD_REQUEST_DATA&cmd_data)
namely, step S502: the Dump backs up the static office data in the original shared memory.
In this step, when the static office data is requested to be loaded into the shared memory, the static office data in the current shared memory is backed up to obtain the static class of all the current static office data in the shared memory. The Dump backup is to record the content of the whole storage device or a certain part of the storage device in another storage device at a specific time, that is, all static classes of the static office data stored in the backup shared memory in this step are backed up to backup the static office data stored in the shared memory.
In this embodiment, the office data stored in the shared memory is old static office data a and old static office data B. Then, correspondingly, the old static office data a + B is backed up to the disk file during backup, the backed-up disk file may be a Sqlite backed-up disk file, and certainly, the old static office data a + B may also be backed up to the Sqlite disk file during backup.
In this embodiment, since the updated static office data needs to be acquired from the DB database and the Sqlite disk file, the updated static data a read from the DB database and the updated static data B read from the Sqlite disk file are needed.
ret＝down_shm_to_file(cont_addr_map,tmp_load_bak_pathfile)
Namely, step S504: and loading the updated static office data A read from the DB database, and loading the static office data A into a temporary disk file, namely a Temp disk file.
Aret＝load_db_to_file(cmd_db2mem,tmp_work_pathfile)
Namely, step S506: and loading the updated static office data A read by the temporary file into the shared memory.
ret＝load_file_to_shm(cmd_file2mem,0,from_file_cont_addr_map)
Namely, step S508: and loading the updated static office data B from the Sqlite disk file to the shared memory.
It should be noted that step S506 and step S508 may be executed simultaneously, or step S506 may be executed after step S508 is executed.
ret＝load_file_to_shm(cmd_file2mem,0,from_file_cont_addr_map)
Namely, step S510: and storing the static office data of the latest version of the shared memory, namely the updated static office data A + B into a temporary disk file.
ret＝down_shm_to_file(from_file_cont_addr_map,tmp_work_pathfile)
At this point, the update operation of the static office data is completed.
Step S512: and backing up the updated local data A + B in the temporary disk file in the Sqlite disk file.
ret＝update_shm_stc_version()
rename_file(tmp_load_bak_pathfile,load_bak_pathfile)
And finally, converting the backed-up disk file, the disk Temp file and the Sqlite disk file into formal files. When the shared memory has a problem, the old static office data A + B of the shared memory and the updated static office data A + B can be provided without influencing the business process to acquire the office data.
In this embodiment, after the updated static office data a + B is loaded, if there is a service process connection, the program is forced to switch after being loaded successfully for 10 seconds, and then the shared memory space occupied by the old static office data a + B is released to provide next data update use, but the released shared memory space is not handed back to the operating system. At this point, the updated static office data has been loaded from the DB database and the Sqlite disk file to the shared memory, and all the updated static office data is saved in the Sqlite disk file.
By adopting the method for storing static office data of telecommunication service by using the shared memory designed by the embodiment, when data is updated at a service, static office data is loaded from a relational database management system (RDBMS) again from the idle position of the current shared memory, if the applied memory block is not enough, a new shared memory block is applied from an operating system, but the maximum number of blocks of the applicable shared memory is limited, and if the maximum limit is reached or no idle memory is available for the operating system, CORE is generated.
By the method for storing static office data required by telecommunication services by using a shared memory provided by the embodiment, the static office data required by the services is loaded into the shared memory, so that the centralized storage of the static office data is realized, and the method is a service process such as: the processes of business analysis, pricing, etc. provide static office data. The business process can directly acquire data from the shared memory, thereby greatly improving the speed of business processing and reducing the I/O overhead of the physical database. Meanwhile, the data processing system provided by the embodiment supports automatic timing loading of updated office data, when the office data is updated, static office data in the physical database is imported into the Sqlite disk file, the whole process is an atomic transaction, meanwhile, version control is performed on the data, registration management is performed on processes, and consistency of data access of the same process is guaranteed. It can be seen that the data processing system provided by the embodiment supports dynamic update of static office data while not affecting real-time service processing.
The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.
The data processing method and apparatus provided by the present invention are described in detail above, and the principle and the implementation of the present invention are explained in detail herein by applying specific examples, and the descriptions of the above examples are only used to help understanding the method and the core idea of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (10)
1. A data processing method, comprising:
obtaining static office data required by the service from a relational database and/or a file database;
loading the obtained static office data into a shared memory, exporting the static office data loaded into the shared memory to a temporary file, and importing the static office data into a file database through the temporary file for first backup;
when determining that static office data required by the service in a relational database and/or a file database is updated, copying the static office data in the shared memory into the file database for second backup, acquiring the updated static office data, and loading the updated static office data into the shared memory so as to simultaneously store the static office data before updating and the updated static office data in the shared memory;
and exporting the updated static office data loaded into the shared memory to the temporary file, and importing the updated static office data into the file database through the temporary file to update the static office data backed up by the first backup.
2. The method of claim 1, wherein the step of obtaining updated static office data comprises:
when the updated static office data is obtained from the relational database, searching the updated static office data in the relational database through a header file for reading the data of the relational database, and importing the searched updated static office data into the temporary file;
and when the updated static office data is obtained from the file database, searching the updated static office data through a database tool arranged in the file database to obtain the updated static office data.
3. The method of claim 2, wherein the step of searching for updated static office data in the relational database via a header file used to read the relational database data comprises:
and positioning a relational database to be searched by an OTL method in the header file so as to search the updated static office data from the relational database.
4. The method according to claim 1, wherein the step of loading the obtained static office data into a shared memory comprises:
judging whether the created shared memory exists or not;
if the static office data exists, the created shared memory is directly connected, and the static office data is loaded according to a set rule;
if the static office data does not exist in the shared memory, establishing a shared memory through an Shmaget function, calling an entry function to establish all static class objects in the newly-established shared memory, and loading the static office data into the established static class objects; wherein, the creating the shared memory through the Shmaget function comprises: and the shared memory is created by using the Shmget method.
5. The method of claim 4, wherein the step of loading the static office data according to a set rule comprises:
judging whether the remaining memory space in the hooked shared memory is enough to load the static office data or not;
if not, applying for a new shared memory space to an operating system, calling an entry function to create all static class objects of the new shared memory space, loading the static office data in the shared memory from an address behind the tail address of the loaded data, and loading the static office data by using the idle static class objects in the original shared memory space of the shared memory and the newly created static class objects;
and if so, directly hooking the shared memory, and loading the static office data from an address behind the tail address of the loaded data in the shared memory so as to load the static office data into an idle static class object in the shared memory.
6. The method according to claim 1, further comprising, after the step of loading the updated static office data into the shared memory:
judging whether business process connection exists or not;
if the static office data exists in the service process, after the set time, the data version ID of the service process is forcibly converted into a new data version ID, so that the updated shared memory area where the static office data is located is used for providing data for the service process connection, wherein the data version ID and the shared memory area have a corresponding relation.
7. A data processing apparatus, comprising:
the first acquisition module is used for acquiring static office data required by the service from a relational database and/or a file database;
the first loading module is used for loading the acquired static office data into a shared memory, exporting the static office data loaded into the shared memory to a temporary file, and importing the static office data into a file database through the temporary file for first backup;
the updating module is used for copying the static office data in the shared memory into a file database for second backup when determining that the static office data required by the service in the relational database and/or the file database is updated, acquiring the updated static office data, and loading the updated static office data into the shared memory so as to simultaneously store the static office data before updating and the updated static office data in the shared memory;
and the backup module is used for exporting the updated static office data loaded into the shared memory to the temporary file, and importing the updated static office data into the file database through the temporary file to update the static office data backed up by the first backup.
8. The apparatus of claim 7, wherein:
when the updating module acquires the updated static office data from the relational database, searching the updated static office data in the relational database through a header file for reading the data of the relational database, and importing the searched updated static office data into the temporary file;
and when the updating module acquires the updated static office data from the file database, searching the updated static office data through a database tool built in the file database to acquire the updated static office data.
9. The apparatus according to claim 8, wherein the updating module locates the relational database to be searched by an OTL method in the header file to search the updated static office data from the relational database when searching the updated static office data in the relational database by the header file for reading the data of the relational database.
10. The apparatus of claim 7, wherein the first load module comprises:
the judging module is used for judging whether the created shared memory exists or not;
the first execution module is used for directly hooking the created shared memory and loading the static office data according to a set rule when the judgment result of the judgment module is that the shared memory exists;
a second execution module, configured to create a shared memory through the Shmaget function, call an entry function to create all static class objects in the newly created shared memory, and load the static office data into the created static class objects when the determination result of the determination module is absent; wherein, the creating the shared memory through the Shmaget function comprises: creating a shared memory by using a Shmget method;
and the data export module is used for exporting the static office data loaded into the shared memory to a temporary file, and importing the static office data into a file database through the temporary file for first backup.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410573710.3A CN105589881B (en) | 2014-10-23 | 2014-10-23 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410573710.3A CN105589881B (en) | 2014-10-23 | 2014-10-23 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105589881A CN105589881A (en) | 2016-05-18 |
| CN105589881B true CN105589881B (en) | 2020-01-24 |
Family
ID=55929467
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410573710.3A Active CN105589881B (en) | 2014-10-23 | 2014-10-23 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105589881B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107465944A (en) * | 2017-08-31 | 2017-12-12 | 江苏省公用信息有限公司 | The gray scale delivery system and method for a kind of IPTV Content Management |
| CN108038171B (en) * | 2017-12-07 | 2020-07-03 | 杭州电魂网络科技股份有限公司 | Data writing method and device and data server |
| CN108304461A (en) * | 2017-12-26 | 2018-07-20 | 山东浪潮通软信息科技有限公司 | A kind of method and device of data sharing |
| CN108958732A (en) * | 2018-06-28 | 2018-12-07 | 上海恺英网络科技有限公司 | A kind of data load method and equipment based on PHP |
| CN109756704A (en) * | 2018-11-30 | 2019-05-14 | 安徽清新互联信息科技有限公司 | A kind of inter-process communication methods and system of inserted audio-video monitoring system |
| CN110795508B (en) * | 2019-11-04 | 2023-04-07 | 中国建设银行股份有限公司 | Data copying method, device, equipment and storage medium |
| CN111200692B (en) * | 2019-12-24 | 2021-10-26 | 广州市高科通信技术股份有限公司 | Voice equipment, processing method, device and storage medium for network telephone |
| CN112256457A (en) * | 2020-11-06 | 2021-01-22 | 开普云信息科技股份有限公司 | A shared memory-based data loading acceleration method, device, electronic device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6941295B2 (en) * | 1999-08-12 | 2005-09-06 | International Business Machines Corporation | Data access system |
| CN101126920A (en) * | 2007-09-30 | 2008-02-20 | 浙江中控技术有限公司 | Method and device implementing on-line upgrading of a fixed programme |
| CN101754156A (en) * | 2008-12-05 | 2010-06-23 | 中国移动通信集团辽宁有限公司 | Office data management method, device and charging system |
| CN101853288A (en) * | 2010-05-19 | 2010-10-06 | 马晓普 | Configurable full-text retrieval service system based on document real-time monitoring |
-
2014
- 2014-10-23 CN CN201410573710.3A patent/CN105589881B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6941295B2 (en) * | 1999-08-12 | 2005-09-06 | International Business Machines Corporation | Data access system |
| CN101126920A (en) * | 2007-09-30 | 2008-02-20 | 浙江中控技术有限公司 | Method and device implementing on-line upgrading of a fixed programme |
| CN101754156A (en) * | 2008-12-05 | 2010-06-23 | 中国移动通信集团辽宁有限公司 | Office data management method, device and charging system |
| CN101853288A (en) * | 2010-05-19 | 2010-10-06 | 马晓普 | Configurable full-text retrieval service system based on document real-time monitoring |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105589881A (en) | 2016-05-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105589881B (en) | Data processing method and device | |
| US20120158795A1 (en) | Entity triggers for materialized view maintenance | |
| CN109492053B (en) | Method and device for accessing data | |
| CN102073526B (en) | Software transfer installation method and system for mobile terminal | |
| CN113297320A (en) | Distributed database system and data processing method | |
| CN111367925A (en) | Data dynamic real-time updating method, device and storage medium | |
| CN108205560B (en) | Data synchronization method and device | |
| CN101727475B (en) | Method, device and system for acquiring database access process | |
| CN106325870B (en) | Method and device for realizing dynamic form API (application program interface) | |
| CN108667866A (en) | A kind of request processing method and device based on API intelligent adaptations | |
| CN109857723B (en) | Dynamic data migration method based on expandable database cluster and related equipment | |
| WO2019161620A1 (en) | Application dependency update method, terminal and device, and storage medium | |
| CN110134738A (en) | Distributed storage system resource estimation method and device | |
| CN112463170B (en) | Method, device and equipment for making mirror image on cloud platform and storage medium | |
| CN111046106A (en) | Cache data synchronization method, device, device and medium | |
| CN105094787B (en) | Method and device for processing enterprise internet application | |
| CN113660359A (en) | Management method and device for domain name resolution record, storage medium and electronic equipment | |
| CN109325057B (en) | Middleware management method, device, computer equipment and storage medium | |
| US20030162559A1 (en) | Mobile communications terminal, information transmitting system and information receiving method | |
| CN116628027A (en) | Cross-database query method, apparatus, device, medium and computer program product | |
| CN115510204A (en) | Intelligent water affair data resource catalog management method and device | |
| WO2019153547A1 (en) | Database operation method, apparatus and device, and computer-readable storage medium | |
| CN114675776A (en) | Resource storage method and apparatus, storage medium and electronic device | |
| CN117093597B (en) | Data processing method and device | |
| KR101573663B1 (en) | Method and system for data integration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |