[go: up one dir, main page]

CN109783276B - Data backup and recovery device and method based on special compression card - Google Patents

Data backup and recovery device and method based on special compression card Download PDF

Info

Publication number
CN109783276B
CN109783276B CN201811559002.9A CN201811559002A CN109783276B CN 109783276 B CN109783276 B CN 109783276B CN 201811559002 A CN201811559002 A CN 201811559002A CN 109783276 B CN109783276 B CN 109783276B
Authority
CN
China
Prior art keywords
data
compression
backup
card
database
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201811559002.9A
Other languages
Chinese (zh)
Other versions
CN109783276A (en
Inventor
潘多
赵峰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yuanguang Software Co Ltd
Original Assignee
Yuanguang Software Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yuanguang Software Co Ltd filed Critical Yuanguang Software Co Ltd
Priority to CN201811559002.9A priority Critical patent/CN109783276B/en
Publication of CN109783276A publication Critical patent/CN109783276A/en
Application granted granted Critical
Publication of CN109783276B publication Critical patent/CN109783276B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention relates to a data backup and recovery device and a method based on a special compression card, which belong to the technical field of information and solve the problems of data backup and storage efficiency and storage cost. According to the invention, through the special compression card, under the condition that the CPU and the storage performance are not influenced, the space storage efficiency is improved, and the storage cost is saved, especially the storage cost of high-performance media such as a flash memory and the like; and the compression process works in the storage layer, and is completely transparent to the access of the database service of the upper layer.

Description

Data backup and recovery device and method based on special compression card
Technical Field
The invention relates to the technical field of information, in particular to a data backup and recovery device and a data backup and recovery method based on a special compression card.
Background
With the development of enterprise informatization, the business function, the number of users and the number of units increase year by year, the informatization data volume increases rapidly, and backup storage of database data files and filing logs is the key point of database operation and maintenance work. According to the traditional data backup storage, the backup data is directly written into a storage layer without compression, the storage process time is long, the occupied storage space is large, the cost of storage on a flash memory medium is high, and the operation and maintenance cost of an enterprise is increased.
Disclosure of Invention
In view of the foregoing analysis, the present invention is directed to provide a data backup and recovery apparatus and method based on a dedicated compression card, which utilize the dedicated compression card to improve space storage efficiency and save storage cost without affecting CPU and storage performance.
The purpose of the invention is mainly realized by the following technical scheme:
the invention provides a data backup and recovery device based on a special compression card, which comprises a compression/recovery interface module, a compression card driving module, a special compression card and a backup storage layer;
the compression/recovery interface module is connected with the database platform and used for receiving the data file to be backed up and the filing log file output by the database or outputting the recovered data file and the filing log file to corresponding positions of the database;
the special compression card driving module is internally integrated with a compression card driving and compression algorithm library, is connected with the special compression card and is used for starting the compression/recovery function of the special compression card;
the special compression card is connected with the compression/recovery interface module and the backup storage layer, and on one hand, the special compression card is used for receiving file data output by the database through the compression/recovery interface module and sending the compressed file data to the backup storage layer; on the other hand, the method is used for reading the compressed data to be restored from the backup storage layer and sending the corresponding position of the database platform after restoration;
and the backup storage layer is used for storing the compressed backup data file and the filing log file.
In another aspect of the present invention, a data backup method based on a dedicated compression card is provided, which includes the following steps:
connecting the backup data storage layer with the database by using a special compression card;
formatting the data backup storage layer into an xfs file system, designating data files to be backed up and filing log file information in a database on the file system, and setting a special compression card into a compression mode;
after receiving the data backup instruction, the special compression card compresses the data files to be backed up and the filing log files in the read database;
the special compression card outputs the compressed data to a page cache region for caching;
the page cache region writes cache data to a corresponding storage layer.
In another aspect of the present invention, a data recovery method based on a dedicated compression card is provided, which is characterized by comprising the following steps:
connecting a storage layer storing backup data with a database by using a special compression card;
setting the special compression card to a recovery mode; an xfs file system for reading the backup data file and filing log file information in the storage layer; appointing the information of backup data files and filing log files to be restored in the storage layer to specific positions in a database;
under the control of a data recovery instruction, the storage layer sends backup data to be recovered to a page cache region for caching;
the page cache region sends the cache data to a special compression card for data decompression and recovery; and the card stores the recovered data into a corresponding position of a database according to the information of the data file and the filing log file in the xfs file system.
In another aspect of the present invention, a data processing method based on a dedicated compression card is provided, which includes the above data compression method and recovery method based on a dedicated compression card.
The invention has the following beneficial effects:
1. the whole data compression/recovery process does not affect business operation, storage performance and CPU computing resources, compressed data are written into a storage layer again, backup time is short, occupied storage space is small, storage space can be effectively utilized, and under the condition of not affecting performance, hardware cost is reduced, especially storage cost of high-performance media such as flash memories and the like; and the compression process works in the storage layer, and is completely transparent to the access of the database service of the upper layer.
2. And a good compression effect display is provided by utilizing a graphical interface, so that the operation and maintenance cost of an enterprise is saved, and the operation and maintenance efficiency is improved.
3. Backup data may be compressed to a size of 1/2 or 1/5;
4. and a general gzip algorithm is adopted, so that even if the special compression card is damaged, the CPU can still be used for decompressing data, and the robustness is improved.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
The drawings are only for purposes of illustrating particular embodiments and are not to be construed as limiting the invention, wherein like reference numerals are used to designate like parts throughout.
FIG. 1 is a schematic diagram illustrating the connection between components of a data backup and restore apparatus according to an embodiment of the present invention;
FIG. 2 is a flowchart of a data backup method according to an embodiment of the present invention;
FIG. 3 is a flowchart of a data recovery method according to an embodiment of the present invention.
Detailed Description
The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form a part hereof, and which together with the embodiments of the invention serve to explain the principles of the invention.
A specific embodiment of the present invention discloses a data backup and recovery device based on a dedicated compression card, as shown in fig. 1, including a compression/recovery interface module, a compression card driving module, a dedicated compression card, and a backup storage layer;
the compression/recovery interface module is connected with the database platform and used for receiving the data file to be backed up and the filing log file output by the database or outputting the recovered data file and the filing log file to corresponding positions of the database;
the special compressed card driving module is internally integrated with a compressed card driving and zlib compression algorithm library (including gzip compression algorithm), is connected with the special compressed card, and starts the compression/recovery function of the special compressed card through an external instruction; the external instructions include a compress data instruction and a restore data instruction, which may be from a user interface of a database.
The special compression card is connected with the compression/recovery interface module and the backup storage layer, and on one hand, when receiving a compressed data command, the special compression card is used for receiving file data in a database which is required to be compressed in the compressed data command through the compression/recovery interface module, compressing the file data and then sending the compressed file data to the backup storage layer; on the other hand, when receiving the data recovery command, the data recovery module is used for reading the compressed data required to be recovered in the recovery backup command from the backup storage layer and sending the corresponding position of the database platform after recovery;
and the backup storage layer is used for storing the compressed backup data file and the filing log file.
Specifically, the dedicated compression card is a card dedicated to compression independent of a database platform, and performs data compression by using a gzip compression algorithm of a Zlib library.
In the embodiment, a special compression card is adopted, and the gzip compression algorithm of the ZLib library is called to be responsible for compression, so that the calculation work of a CPU is replaced, and the additional CPU consumption caused by calling the gzip library for compression by using the CPU in the conventional compression is avoided. Therefore, no extra CPU is consumed, and the CPU load is lower because the amount of data written becomes smaller.
Moreover, due to the adoption of the general gzip algorithm, even if the special compression card is damaged, the data can still be decompressed by using the CPU, the space occupation is not changed, and the robustness is improved.
In the embodiment, a special compression card is used to replace a CPU to perform compression operation, when a gzip compression algorithm is operated, the compression operation performance of the special compression card is 2 times that of the compression operation performed by the CPU under OLTP, and the compression ratio can reach 4 times under OLAP because a storage layer adopts a larger block. The archiving compression ratio can reach more than 10 times.
Specifically, the special compression card of the embodiment can achieve a compression ratio of about 1:2 for data files and a compression ratio of about 1:5 for archive log files. Under the condition of not influencing the performance, the utilization rate of storage is improved, and the hardware cost is reduced.
Optionally, the backup storage layer is formatted as an xfs file system, and the backed-up data file and the archive log file information are stored in a file directory created on the xfs file system.
The data file and archive log file information comprises:
database default data files are data files of table spaces such as SYSTEM, TEMP, EXAMPLE, SYSAUX, UNDOTBS1, USERS and the like.
Business system data files in database instances: dbf (different business system data file names)
The database archives log files. Redo Log Files (different database platforms, different filing Log file names);
in order to adapt to the storage and reading of high-speed data, the embodiment further includes a page cache module, where the page cache module is connected to the dedicated compression card and the backup storage layer, and is configured to cache data compressed by the dedicated compression card or cache backup data read from the backup storage layer.
And writing the compressed data into a cache:
when the special compression card initiates a compression data write cache request, the compression data is directly written into the page cache module, and the content in the backup storage cannot be directly updated. The cache page (page) being written is marked dirty and added to dirty list. The cached pages (pages) in dirty list are then periodically written back to disk, so that the data on disk and the data of the page cache module are consistent.
Caching the read backup data:
when the special compression card initiates a request for reading the backup data, whether the requested backup data is cached in the page cache module is checked firstly, if so, the requested backup data is directly read from the page cache module without accessing a disk. If the requested data is not available in the page cache module, the data must be read from disk. And then caching the read backup data into the page cache module, and reading the backup data from the page cache module by the special compression card.
Optionally, in order to further improve human-computer interaction and enable an operator to visually know specific content and progress of data backup, the embodiment further includes a graphical module, where the graphical module is connected to the compression/recovery interface module and is used to collect and display progress of data backup/recovery; and estimating the time required by backup/recovery according to the size and the type of the file to be backed up/recovered, and displaying the backup/recovery progress in a progress bar mode.
The embodiment also discloses a data backup method based on the special compressed card, as shown in fig. 2, comprising the following steps:
step S201, connecting a backup data storage layer with a database by using a special compression card;
step S202, formatting the data backup storage layer into an xfs file system, designating data files to be backed up and filing log file information in a database on the file system, and setting a special compression card as a compression mode;
step S203, after receiving the data backup instruction, the special compression card compresses the data file to be backed up and the filing log file in the read database;
step S204, the special compression card outputs the compressed data to a page cache region for caching;
and step S205, the page cache region writes the cache data into the corresponding storage layer.
Optionally, the dedicated compression card performs compression by using a gzip compression algorithm of the Zlib library, instead of the calculation work of the CPU, and avoids the need of consuming an additional CPU by using the CPU to call the gzip library for compression in the conventional compression. Therefore, no extra CPU is consumed, and the CPU load is lower because the amount of data written becomes smaller.
Specifically, the dedicated compression card performs data compression by using a gzip compression algorithm of a Zlib library.
In order to realize the connection between the special compression card and the database, a compression card drive is integrated in the database, and during data compression, the database starts a compression function of the special compression card through the compression card drive.
The embodiment also discloses a data recovery method based on the special compressed card, as shown in fig. 3, including the following steps:
step S301, connecting a storage layer storing backup data with a database by using a special compression card;
step S302, setting the special compression card to be in a recovery mode; an xfs file system for reading the backup data file and filing log file information in the storage layer; appointing the information of backup data files and filing log files to be restored in the storage layer to specific positions in a database;
step S303, under the control of the data recovery instruction, the storage layer sends the backup data to be recovered to a page cache region for caching;
step S304, the page cache region sends the cache data to a special compression card for data decompression and recovery; and the card stores the recovered data into a corresponding position of a database according to the information of the data file and the filing log file in the xfs file system.
The embodiment also discloses a data processing method based on the special compression card, which comprises the compression/recovery method in the embodiment.
By compressing/restoring the data, the data backup is realized without influencing business operation, storage performance and CPU computing resources, the time is short, the occupied storage space is small, the storage space can be more effectively utilized, and the hardware cost is reduced under the condition of not influencing the performance; a good compression effect display is provided by using a graphical interface, so that the operation and maintenance cost is saved for enterprises, and the operation and maintenance efficiency is improved; backup data may be compressed to a size of 1/2 or 1/5; and a general gzip algorithm is adopted, so that even if the special compression card is damaged, the CPU can still be used for decompressing data, and the robustness is improved.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention.

Claims (7)

1. A data backup and recovery device based on a special compression card is characterized by comprising a compression/recovery interface module, a compression card driving module, the special compression card and a backup storage layer;
the compression/recovery interface module is connected with the database platform and used for receiving the data file to be backed up and the filing log file output by the database or outputting the recovered data file and the filing log file to corresponding positions of the database;
the compression card driving module is internally integrated with a compression card driving and compression algorithm library, is connected with a special compression card and is used for starting the compression/recovery function of the special compression card; the database is connected with the database platform through the compression/recovery interface module, a compression card drive is integrated in the database, and during data compression, the database starts a special compression card compression function through the compression card drive;
the special compression card is connected with the backup storage layer and is connected with the compression/recovery interface module through the compression card driving module; on one hand, the method is used for receiving file data output by the database through the compression/recovery interface module, and sending the compressed file data to the backup storage layer; on the other hand, the method is used for reading the compressed data to be restored from the backup storage layer and sending the corresponding position of the database platform after restoration;
the backup storage layer is used for storing compressed backup data files and filing log files;
the page cache module is connected with the special compression card and the backup storage layer and used for caching data compressed by the special compression card or caching backup data read from the backup storage layer;
and writing the compressed data into a cache:
when the special compression card initiates a compression data write cache request, the compression data is directly written into the page cache module; the written cache page is marked as dirty and added to dirty; periodically writing the cached pages in dirtylist back to the disk, so that the data on the disk is consistent with the data of the page cache module;
caching the read backup data:
when the special compression card initiates a request for reading the backup data, firstly, whether the requested backup data is cached in the page cache module is checked, if so, the requested backup data is directly read from the page cache module without accessing a disk; if the page cache module does not have the requested data, the data must be read from the disk; and then caching the read backup data into the page cache module, and reading the backup data from the page cache module by the special compression card.
2. The data backup and recovery device according to claim 1, wherein the dedicated compression card is a card dedicated for compression independent of the database platform, and data compression is performed by gzip compression algorithm of Zlib library.
3. The data backup and restore apparatus of claim 1 wherein the backup storage tier is formatted as an xfs file system and the backed up data files, archive log file information are stored in a file directory created on the xfs file system.
4. The data backup and recovery device according to any one of claims 1 to 3, further comprising a graphical module, connected to the compression/recovery interface module, for collecting and displaying the progress of data backup/recovery; and estimating the time required by backup/recovery according to the size and the type of the file to be backed up/recovered, and displaying the backup/recovery progress in a progress bar mode.
5. A data backup method based on the data backup and restore apparatus according to any one of claims 1 to 4, comprising the steps of:
connecting the backup storage layer with the database by using a special compression card;
formatting the backup storage layer into an xfs file system, designating data files to be backed up and filing log file information in a database on the file system, and setting a special compression card into a compression mode;
after receiving the data backup instruction, the special compression card compresses the data files to be backed up and the filing log files in the read database;
the special compression card outputs the compressed data to a page cache region for caching;
the page cache region writes cache data to a corresponding storage layer.
6. A data restoring method based on the data backup and restoring apparatus according to any one of claims 1 to 4, comprising the steps of:
connecting the backup storage layer with the database by using a special compression card;
setting the special compression card to a recovery mode; an xfs file system for reading the backup storage layer containing the backed-up data files and the archived log file information; appointing the information of backup data files and filing log files to be restored in the backup storage layer to specific positions in a database;
under the control of a data recovery instruction, the backup storage layer sends backup data to be recovered to a page cache region for caching;
the page cache region sends the cache data to a special compression card for data decompression and recovery; and storing the recovered data into a corresponding position of a database according to the information of the data file and the filing log file in the xfs file system.
7. A data processing method based on a dedicated compression card, characterized by comprising the data backup method of claim 5 and the data recovery method of claim 6.
CN201811559002.9A 2018-12-19 2018-12-19 Data backup and recovery device and method based on special compression card Active CN109783276B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201811559002.9A CN109783276B (en) 2018-12-19 2018-12-19 Data backup and recovery device and method based on special compression card

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201811559002.9A CN109783276B (en) 2018-12-19 2018-12-19 Data backup and recovery device and method based on special compression card

Publications (2)

Publication Number Publication Date
CN109783276A CN109783276A (en) 2019-05-21
CN109783276B true CN109783276B (en) 2021-03-02

Family

ID=66497318

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201811559002.9A Active CN109783276B (en) 2018-12-19 2018-12-19 Data backup and recovery device and method based on special compression card

Country Status (1)

Country Link
CN (1) CN109783276B (en)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2006125271A1 (en) * 2005-05-27 2006-11-30 Damit Australia Pty Ltd A digital asset management system
US7158998B2 (en) * 2002-07-31 2007-01-02 Cingular Wireless Ii, Llc Efficient synchronous and asynchronous database replication
US7389314B2 (en) * 2004-08-30 2008-06-17 Corio, Inc. Database backup, refresh and cloning system and method
WO2009025923A1 (en) * 2007-08-23 2009-02-26 Vhayu Technologies Corporation A system and method for data compression using compression hardware
CN101727291A (en) * 2008-10-14 2010-06-09 北京众志和达信息技术有限公司 System and method for embedded integrated virtual tape library
CN103309704A (en) * 2013-05-09 2013-09-18 苏州泽佑科技有限公司 Android driving program loading method
CN103778258A (en) * 2014-02-27 2014-05-07 华为技术有限公司 Method for sending and receiving data of database, client terminal and server
CN107357848A (en) * 2017-06-27 2017-11-17 中国电子科技集团公司第二十八研究所 Database synchronization method based on driving encapsulation
US9977716B1 (en) * 2015-06-29 2018-05-22 Veritas Technologies Llc Incremental backup system

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020143764A1 (en) * 2001-04-03 2002-10-03 Martin Andrew R. Data management system and method for intercepting and changing database instructions between a database back end and an application front end
JP4166056B2 (en) * 2002-08-16 2008-10-15 富士通株式会社 Database operation history management device, database operation history management method, and database operation history management program
US7200603B1 (en) * 2004-01-08 2007-04-03 Network Appliance, Inc. In a data storage server, for each subsets which does not contain compressed data after the compression, a predetermined value is stored in the corresponding entry of the corresponding compression group to indicate that corresponding data is compressed
CN100464334C (en) * 2007-06-06 2009-02-25 中国工商银行股份有限公司 Data incremental backup and recovery method applied to data bank
CN101587451A (en) * 2008-05-20 2009-11-25 研祥智能科技股份有限公司 Data backup and recovery method based on extensible firmware interface (EFI) firmware
CN101859231A (en) * 2009-04-09 2010-10-13 北京众志和达信息技术有限公司 Method and system for realizing data compression and decompression in embedded virtual tape library environment
CN101694637A (en) * 2009-11-06 2010-04-14 金蝶软件(中国)有限公司 Method and system for restoring database
US9235464B2 (en) * 2012-10-16 2016-01-12 Microsoft Technology Licensing, Llc Smart error recovery for database applications
CN102902607A (en) * 2012-11-01 2013-01-30 山东大学 Data backup and recovery method for Android thin client
CN106168919A (en) * 2016-07-06 2016-11-30 浪潮电子信息产业股份有限公司 A kind of database backup method, Apparatus and system
CN107800917A (en) * 2017-12-15 2018-03-13 定远县网萌电子商务有限公司 Archival digitalization Machining Intelligent process flow line platform

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7158998B2 (en) * 2002-07-31 2007-01-02 Cingular Wireless Ii, Llc Efficient synchronous and asynchronous database replication
US7389314B2 (en) * 2004-08-30 2008-06-17 Corio, Inc. Database backup, refresh and cloning system and method
WO2006125271A1 (en) * 2005-05-27 2006-11-30 Damit Australia Pty Ltd A digital asset management system
WO2009025923A1 (en) * 2007-08-23 2009-02-26 Vhayu Technologies Corporation A system and method for data compression using compression hardware
CN101727291A (en) * 2008-10-14 2010-06-09 北京众志和达信息技术有限公司 System and method for embedded integrated virtual tape library
CN103309704A (en) * 2013-05-09 2013-09-18 苏州泽佑科技有限公司 Android driving program loading method
CN103778258A (en) * 2014-02-27 2014-05-07 华为技术有限公司 Method for sending and receiving data of database, client terminal and server
US9977716B1 (en) * 2015-06-29 2018-05-22 Veritas Technologies Llc Incremental backup system
CN107357848A (en) * 2017-06-27 2017-11-17 中国电子科技集团公司第二十八研究所 Database synchronization method based on driving encapsulation

Also Published As

Publication number Publication date
CN109783276A (en) 2019-05-21

Similar Documents

Publication Publication Date Title
CN109791520B (en) Physical media aware spatially coupled logging and replay
JP6495568B2 (en) Method, computer readable storage medium and system for performing incremental SQL server database backup
US20110022811A1 (en) Information backup/restoration processing apparatus and information backup/restoration processing system
US20110087640A1 (en) Data Compression and Storage Techniques
US10853199B2 (en) Techniques for managing context information for a storage device while maintaining responsiveness
US9146928B1 (en) Techniques for storing metadata of a filesystem in persistent memory
CN108628542B (en) A file merging method and controller
CN109683825B (en) Storage system online data compression method, device and equipment
US11176033B2 (en) Data storage devices and data processing methods
CN113076220A (en) Data processing method and device, electronic equipment and computer readable medium
CN106708960A (en) Restoration method for file system of NAS device and NAS device
US8151068B2 (en) Data copy management for faster reads
CN111124258B (en) Data storage method, device and equipment of full flash memory array and readable storage medium
CN101201774A (en) Method for snapshot of magnetic disk
CN106681659A (en) Data compression method and device
CN109783276B (en) Data backup and recovery device and method based on special compression card
CN115268767A (en) Data processing method and device
US10430115B2 (en) System and method for optimizing multiple packaging operations in a storage system
US6510499B1 (en) Method, apparatus, and article of manufacture for providing access to data stored in compressed files
US10852997B1 (en) Relocation target selection
KR20170133778A (en) Apparatus and method for consistency of database
CN101739308B (en) Image file generation method and its storage system
US20190107952A1 (en) Logical address history management in memory device
US20090187719A1 (en) Data backup method for logical volume manager
CN115033425A (en) A method to improve the success rate of data backup

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant