CN113448929A - Data storage method and device, equipment and storage medium - Google Patents
Data storage method and device, equipment and storage medium Download PDFInfo
- Publication number
- CN113448929A CN113448929A CN202110815618.3A CN202110815618A CN113448929A CN 113448929 A CN113448929 A CN 113448929A CN 202110815618 A CN202110815618 A CN 202110815618A CN 113448929 A CN113448929 A CN 113448929A
- Authority
- CN
- China
- Prior art keywords
- file
- information
- folder
- files
- levels
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/168—Details of user interfaces specifically adapted to file systems, e.g. browsing and visualisation, 2d or 3d GUIs
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/148—File search processing
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Library & Information Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本申请实施例公开了一种数据存储方法、装置、设备及存储介质，所述方法包括：对分布式文件系统中存储的至少一个文件进行解析，得到每一所述文件的属性信息；其中，所述属性信息至少包括每一所述文件的相对路径和文件大小信息；基于每一所述文件的相对路径和文件大小信息，确定对应所述文件所在的各级文件夹的文件夹信息；将每一所述文件的属性信息和对应的所述各级文件夹信息以轻量级数据交换格式存储在搜索数据库中；其中，每一所述文件的属性信息和对应的所述各级文件夹信息用于通过前端交互界面按层级展示每一所述文件的内容。
The embodiments of the present application disclose a data storage method, device, device, and storage medium. The method includes: parsing at least one file stored in a distributed file system to obtain attribute information of each file; wherein, The attribute information includes at least the relative path and file size information of each of the files; based on the relative path and file size information of each of the files, determine the folder information corresponding to the various levels of folders where the file is located; The attribute information of each of the files and the corresponding folder information at all levels are stored in the search database in a lightweight data exchange format; wherein, the attribute information of each of the files and the corresponding folders at all levels The information is used to display the content of each said file hierarchically through the front-end interactive interface.
Description
Technical Field
The present application relates to the field of computer technology, and relates to, but is not limited to, a data storage method and apparatus, a device, and a storage medium.
Background
Currently, files on a viewing server can only be viewed through command lines such as ls (list directory contents), cat (register files and print on the standard output), etc. For example, files stored on a file system (lustre) or a distributed file system (ceph) can only be viewed through a command line, are not visual, cannot view pictures and videos, and cannot be displayed through a graphical interface.
Disclosure of Invention
The embodiment of the application provides a data storage method, a data storage device, data storage equipment and a data storage medium.
The technical scheme of the embodiment of the application is realized as follows:
in a first aspect, an embodiment of the present application provides a data storage method, where the method includes:
analyzing at least one file stored in a distributed file system to obtain attribute information of each file; wherein the attribute information at least comprises relative path and file size information of each file;
determining folder information of all levels of folders corresponding to the files based on the relative path and file size information of each file;
storing the attribute information of each file and the corresponding folder information of each level in a search database in a lightweight data exchange format; and the attribute information of each file and the corresponding folder information at each level are used for displaying the content of each file according to the hierarchy through a front-end interactive interface.
In some possible embodiments, the parsing at least one file stored in the distributed file system to obtain attribute information of each file includes: recording the stored meta information of the at least one file in the process of uploading the at least one file to the distributed file system; and analyzing the meta information line by line to obtain the attribute information of each file. Therefore, the file attribute information in the lightweight data exchange format can be obtained by recording the meta information of the stored file during uploading the file and analyzing the meta information line by line, and the file attribute information is convenient to store in a search database subsequently.
In some possible embodiments, determining folder information corresponding to each level folder in which the file is located based on the relative path and file size information of each file includes: analyzing the relative path of each file layer by layer to obtain the relative path of each level of folder corresponding to the file; determining the folder levels of the folders at all levels based on the relative paths of the folders at all levels; accumulating the file size information of the files contained in the folders at all levels layer by layer according to the folder levels to obtain the file size information of the folders at all levels; and taking the file size information of the folders at all levels as the folder information. Therefore, the relative path of the file and the file folder hierarchy are obtained by analyzing the relative path of the file, and the size information of all levels of file folders is accumulated layer by layer from deep to shallow to obtain the file folder information, so that the file folder information is conveniently recorded under the node information of the corresponding file folder.
In some possible embodiments, the storing the attribute information of each file and the corresponding folder information of each level in a lightweight data exchange format in a search database includes: recording the attribute information of each file under the node of each file in the lightweight data exchange format; storing the relative paths of the folders at all levels and corresponding folder information in a mapping file in a key value pair mode; based on the mapping file, recording the folder information of each level of folder under the node of the corresponding folder in the lightweight data exchange format; and sequentially inserting the nodes of each file and the nodes of all levels of folders into the search database according to the folder hierarchy. Therefore, the attribute information of the file node storage file is created firstly, then the folder node storage folder information is created, and finally the file node storage folder information is sequentially inserted into the search database according to the folder hierarchy, so that the file hierarchy structure information is stored in the search database, and the information of the file can be displayed according to the hierarchy through the front-end display page.
In some possible embodiments, each of the files includes at least one of: pictures, videos, text files; the nodes of each file at least comprise the following fields: the file updating method comprises the following steps of (1) storing information of a father file directory, a file name, a file size, a file type, the number of files, an updating timestamp and a file; wherein the storage information of the file at least comprises one of the following information: region, container, object key. Therefore, the file attribute information obtained by analysis is respectively recorded in the corresponding fields of the file nodes, so that a certain field can be conveniently retrieved from the search database to check the corresponding file attribute information, and the data checking efficiency is improved.
In some possible embodiments, the method further comprises: in response to receiving a parent folder path input by a user on the front-end interactive interface, retrieving folder information of each level below the parent folder path from the search database; and calling and displaying files included under each level of folders from the distributed file system based on the folder information of each level. Therefore, files under each level of folder can be visually displayed by searching according to the path of the parent folder on the front-end interactive interface by storing the information of each level of folder in the search database.
In some possible embodiments, the method further comprises: retrieving a relative path of a first file from the search database in response to receiving a user-entered parent folder name and a file name of the first file on the front-end interactive interface; and calling and displaying the content of the first file from the distributed file system based on the relative path of the first file. Therefore, fuzzy search can be carried out in the search database by combining the parent folder name and the file name, the content of the first file can be obtained and displayed through the distributed file system, and the operating efficiency is improved.
In some possible embodiments, the method further comprises: and in the process of displaying the folder information of each level under the parent file path, responding to the received downloading operation of a second file, and downloading the second file through the storage information of the second file stored in the search database. Therefore, the storage information for downloading of the files stored in the search database is adopted, the files can be downloaded and checked directly in response to interface operation, and other downloading services do not need to be installed.
In a second aspect, an embodiment of the present application provides a data storage device, including:
the analysis module is used for analyzing at least one file stored in the distributed file system to obtain attribute information of each file; wherein the attribute information at least comprises relative path and file size information of each file;
the determining module is used for determining folder information of all levels of folders corresponding to the files based on the relative path and file size information of each file;
the storage module is used for storing the attribute information of each file and the corresponding folder information of each level in a search database in a lightweight data exchange format; and the attribute information of each file and the corresponding folder information at each level are used for displaying the content of each file according to the hierarchy through a front-end interactive interface.
In a third aspect, an embodiment of the present application provides an electronic device, which includes a memory and a processor, where the memory stores a computer program that is executable on the processor, and the processor implements the steps in the data storage method when executing the program.
In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps in the data storage method.
The beneficial effects brought by the technical scheme provided by the embodiment of the application at least comprise:
in the embodiment of the application, at least one file stored in a distributed file system is analyzed to obtain attribute information of each file; then determining folder information of all levels of folders corresponding to the files based on the relative path and file size information of each file; finally, storing the attribute information of each file and the corresponding folder information of each level in a search database in a lightweight data exchange format; therefore, the distributed file system is used for storing files, the attribute information and the folder information of the files are recorded in the searching database, a visual storage structure is constructed, details of file data can be previewed and checked visually in a front-end display mode, the operation working efficiency is improved, and the data checking speed of researchers is increased.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without inventive efforts, wherein:
fig. 1 is a schematic flowchart of a data storage method according to an embodiment of the present application;
fig. 2 is a schematic flowchart of a data storage method according to an embodiment of the present application;
fig. 3 is a schematic flowchart of a data storage method according to an embodiment of the present application;
fig. 4 is a schematic flowchart of a data storage method according to an embodiment of the present application;
fig. 5A is a schematic flowchart of a data storage method according to an embodiment of the present application;
fig. 5B is a schematic diagram illustrating document data provided in an embodiment of the present application;
fig. 5C is a schematic diagram of a preview text file provided in the embodiment of the present application;
fig. 5D is a schematic diagram illustrating document data provided in an embodiment of the present application;
fig. 5E is a schematic diagram of a preview picture file provided in the embodiment of the present application;
FIG. 6 is a logic flow diagram of a data storage method provided by an embodiment of the present application;
FIG. 7 is a schematic diagram illustrating a structure of a data storage device according to an embodiment of the present disclosure;
fig. 8 is a schematic diagram of a hardware entity of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. The following examples are intended to illustrate the present application but are not intended to limit the scope of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.
It should be noted that the terms "first \ second \ third" referred to in the embodiments of the present application are only used for distinguishing similar objects and do not represent a specific ordering for the objects, and it should be understood that "first \ second \ third" may be interchanged under specific ordering or sequence if allowed, so that the embodiments of the present application described herein can be implemented in other orders than illustrated or described herein.
It will be understood by those within the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which embodiments of the present application belong. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
All data in the computer are 0 and 1, and a series of combinations of 0 and 1 stored on a hardware medium is completely impossible for a user to distinguish and manage. Thus, the concept of "files" is used to organize the data, and all data for the same purpose are organized into different types of files (different types are usually referred to by different suffixes) according to the structural manner required by different applications, and then each file is given a name that is convenient for understanding. When there are many files, the files are grouped according to some dividing way, and each group of files is placed in the same directory (or called folder), but it is also necessary to give the directory an easily understood and memorized name. Besides the files, the directory can have next level directories (called sub-directories or sub-folders), and all the files and directories form a tree structure.
For the convenience of searching, the names of the directories, sub-directories and files are spliced together by using special characters (such as "\" for Windows or DOS System and "/" -for Unix-like System) from the root node to the files themselves, and such a string of characters is called the relative path of the files, such as "/etc/System/system.conf" in Linux System or "C: \ Windows System32\ taskmr.exe" in Windows System. The path may be used as a unique identifier to access a particular file.
The embodiment of the application provides a data storage method, which is applied to computer equipment, such as terminal equipment, a server or other processing equipment. The terminal device includes, but is not limited to, a mobile phone, a notebook computer, a tablet computer, a web pad, a multimedia device, a streaming media device, a mobile internet device, a wearable device, or other types of devices. The functions implemented by the method may be implemented by calling program code by a processor in a computer device, which may, of course, be stored in a computer storage medium, which may comprise at least a processor and a storage medium. The processor may be used for processing of data storage procedures, and the memory may be used for storing data required and data generated during the data storage procedures.
Fig. 1 is a schematic flow chart of a data storage method according to an embodiment of the present application, and as shown in fig. 1, the method at least includes the following steps:
step S110, analyzing at least one file stored in a distributed file system to obtain attribute information of each file;
here, the distributed file system may be Lustre, Ceph, where Lustre is a large-scale, safe and reliable cluster file system with high availability, and Ceph is a unified distributed storage system, and is designed to provide better performance, reliability and extensibility.
Each of the files includes at least one of: pictures, videos, text files, viewing these files in a distributed file system can only be previewed via a command line and do not visually display the picture or video content.
The attribute information of the file includes at least relative path and file size information of each of the files. In some embodiments, the attribute information of the file may further include information such as a file name, a file type, a file number, a timestamp, and the like; in other embodiments, the attribute information of the file may further include storage information of the file, such as an area, a container, and an object key, where the file is stored, and this is not limited in this embodiment of the present application.
Step S120, determining folder information of all levels of folders corresponding to the files based on the relative path and file size information of each file;
here, the relative path of the file indicates which folders the file is recorded under, and the file can be directly found by retrieving the relative path of the file. For example, the relative path a/b/c/d.jpg indicates that the picture d.jpg is recorded under the folder c, where the folder c is a subfolder under the folder b, and the folder b is a subfolder under the folder a, that is, the folder a, the folder b, and the folder c are sequentially a primary folder, a secondary folder, and a tertiary folder.
File size information is used to measure the size of a computer file, usually in prefixed byte numbers. For a file, the specified file size in bytes is returned. For a folder, the size of all files and subfolders contained in the folder is returned in bytes. The actual disk space taken up by a file depends on the file system. The maximum file size of the file system depends on the number of bits that hold the storage size information, and the total size of the file system.
The folder information of each level of folders is the number of files under the corresponding folder and the size information of the folders, wherein the size information of the folders can be obtained by accumulating the size information of each level of subfiles and the size information of the included files layer by layer.
Step S130, storing the attribute information of each file and the corresponding folder information of each level in a search database in a lightweight data exchange format;
here, the attribute information of each file and the corresponding folder information of each level are used for displaying the content of each file in a hierarchical manner through a front-end interactive interface, for example, the attribute information of the file can be retrieved from a search database by inputting an operation instruction on the front-end interactive interface, and the corresponding file is called from a distributed file system and displayed in a hierarchical manner.
The lightweight data interchange format (JSON) is easy for human reading and writing, and also easy for machine parsing and generation. JSON is a string representation of a JavaScript object that uses text to represent the information of a JavaScript object, essentially a string. JSON can convert a set of data represented in a JavaScript object into a string, and then can easily transfer the string between networks or programs and restore it to a data format supported by each programming language when necessary.
The search database (elastic search) is an open-source, distributed, REST-style search and data analysis engine, belonging to a non-relational database. The search database is stored as a distributed, real-time document, and each field can be indexed and searched. Thus, the attribute information and the folder information of the file are recorded in the search database, a visual storage structure is constructed, the folder structure and the content of the file can be previewed, viewed and retrieved intuitively by using a front-end display method, and the relevance of a large amount of data can be displayed on a page clearly.
In the embodiment of the application, at least one file stored in a distributed file system is analyzed to obtain attribute information of each file; then determining folder information of all levels of folders corresponding to the files based on the relative path and file size information of each file; finally, storing the attribute information of each file and the corresponding folder information of each level in a search database in a lightweight data exchange format; therefore, the distributed file system is used for storing files, the attribute information and the folder information of the files are recorded in the searching database, a visual storage structure is constructed, details of file data can be previewed and checked visually in a front-end display mode, the operation working efficiency is improved, and the data checking speed of researchers is increased.
Fig. 2 is a schematic flow chart of a data storage method according to an embodiment of the present application, and as shown in fig. 2, the method at least includes the following steps:
step S210, recording the stored meta information of the at least one file in the process of uploading the at least one file to the distributed file system;
here, the meta-information (meta-information) describes specific information of the attribute of the file, such as information of the owner of the file, contact address, organization name, mail address, and the like. Where the meta (meta) is an auxiliary tag of the header (head) region of hypertext markup language (html), located in the header of the document, and does not contain any content, the meta may provide meta information of the related page, such as descriptions and keywords for search engine and update frequency. The meta information refers to information of the data itself, for example, meta information of a file, which may include creation time of the file, file size, and the like, and is different from information of the text itself.
The at least one file is stored in the distributed file system as a data set, and since the graphical interface presentation is visualized in units of the data set consisting of a group of files or folders, the meta information of the stored at least one file, i.e. the data set, is recorded.
Step S220, analyzing the meta information line by line to obtain attribute information of each file;
here, since the meta information of at least one file stored in the distributed file system is in the form of jsonl (jsonnines), that is, each line is JSON, the meta information is parsed line by line, so that the file attribute information in the lightweight data exchange format can be obtained, and the file attribute information can be conveniently stored in the search database subsequently.
Step S230, determining folder information corresponding to all levels of folders in which the files are located based on the relative path and file size information of each file;
step S240, storing the attribute information of each file and the corresponding folder information of each level in a search database in a lightweight data exchange format;
here, the attribute information of each file and the corresponding folder information of each level are used for displaying the content of each file in a hierarchy manner through a front-end interactive interface.
In the embodiment of the application, the file attribute information in the lightweight data exchange format can be obtained by recording the stored meta information of a group of files during uploading the files and analyzing the meta information line by line, so that the files can be conveniently stored in the search database subsequently.
Fig. 3 is a schematic flow chart of a data storage method according to an embodiment of the present application, and as shown in fig. 3, the method may be implemented by the following steps:
step S310, analyzing at least one file stored in a distributed file system to obtain attribute information of each file;
here, the attribute information includes at least relative path and file size information of each of the files.
Step S320, analyzing the relative path of each file layer by layer to obtain the relative path of each level of folder corresponding to the file;
illustratively, the relative path/a/b/c/d.jpg of the file is analyzed layer by layer, and the folder a and the relative path/, the folder b and the relative path a/, the folder c and the relative path/a/b/, are analyzed sequentially.
Step S330, determining the folder levels of all levels of folders based on the relative paths of all levels of folders;
for example, the folder a, the folder b, and the folder c may be sequentially a first level, a second level, and a third level, and the file hierarchy structure of each level of folders may be obtained by sorting the folders in the reverse order of the levels.
Step S340, accumulating the file size information of the files contained in the folders at all levels layer by layer according to the folder levels to obtain the file size information of the folders at all levels;
here, for each level of folder, the file size information of the subfolders and the files included under the subfolders are accumulated to obtain the file size information of the folder.
Illustratively, accumulating the file size information of all files in the folder c to obtain the file size information of the folder c; accumulating the file size information of each subfolder under the folder b and including the folder c to obtain the file size information of the folder b; and analogizing in turn to obtain the file size information of the folder a.
Step S350, taking the file size information of each level of folder as the folder information;
step S360, storing the attribute information of each file and the corresponding folder information of each level in a search database in a lightweight data exchange format.
In the embodiment of the application, the relative path of the file and the file hierarchy are obtained by analyzing the relative path of the file, and then the size information of all levels of files is accumulated layer by layer from deep to shallow to obtain the file information, so that the file information is conveniently recorded under the node information of the corresponding file. Therefore, the file hierarchical structure information is stored in the search database, and the information of the file can be displayed according to the hierarchy through the front-end display page.
Fig. 4 is a schematic flow chart of the data storage method according to the embodiment of the present application, and as shown in fig. 4, the step S130 or S240 "storing the attribute information of each file and the corresponding folder information of each level in the search database in the lightweight data exchange format" may be implemented by the following steps:
step S410, recording the attribute information of each file under the node of each file in the lightweight data exchange format;
here, a node is created in advance in the search database for each file, and then the attribute information of the file obtained by the parsing is arranged in each field of the node of the corresponding file.
The nodes of each file at least comprise the following fields: parent directory (parentDirectory), file name (fileName), file size (size), file type (fileType), number of files (fileNum), update time stamp (updateTime), and storage information of files; wherein the storage information of the file at least comprises one of the following information: area (area), container (bucket), object key (objectKey).
The parent folder directory is used for recording the folder under which the file is, the file type is used for recording whether the file is the file or the folder, and correspondingly, the number of the files is 1 when the file type is the file, and the number of the files is all the files under the folder when the file type is the folder. The time stamp indicates the time of the file update, which is convenient for managing the file. The storage information of the file is used for downloading the file, and the file stored in the distributed file system can be directly downloaded by calling the storage information of the file.
Therefore, the file attribute information obtained by analysis is respectively recorded in the corresponding fields of the file nodes, so that a certain field can be conveniently retrieved from the search database to check the corresponding file attribute information, and the data checking efficiency is improved.
Step S420, storing the relative path of each level of folder and the corresponding folder information in a mapping file in a key value pair mode;
here, the map file that maps a file to a folder is a combination of a set of key-value pairs, which is colloquially understood to resemble a particular array. In implementation, the relative path of each level of folder is stored in the key (key) of the mapping file, and the corresponding folder information is stored in the value (value) of the mapping file.
Step S430, recording the folder information of all levels of folders under the nodes of the corresponding folders in the lightweight data exchange format based on the mapping file;
the folder information in the mapping file is sequentially recorded under the nodes of the corresponding folders from deep to shallow according to the hierarchy, and if each folder has other files, the folder information is also recorded under the corresponding nodes.
Step S440, sequentially inserting the node of each file and the nodes of the folders at each level into the search database according to the folder hierarchy.
It is noted that the nodes of each file are inserted first, and then the folder nodes of each level are inserted under the root node "/" of the entire dataset according to the folder hierarchy.
In the embodiment of the application, the attribute information of the file node storage file is firstly created, then the folder node storage folder information is created, and finally the folder node storage folder information is sequentially inserted into the search database according to the folder hierarchy, so that the file hierarchy structure information is stored in the search database, and the information of the file can be displayed according to the hierarchy through the front-end display page.
Fig. 5A is a schematic flowchart of a data storage method according to an embodiment of the present application, and as shown in fig. 5A, the method includes the following steps:
step S510, in response to receiving a parent folder path input by a user on the front-end interactive interface, retrieving folder information of each level below the parent folder path from the search database;
here, the folder information of each level includes the number of files under the corresponding folder and file size information of the folder. Compared with the prior art that the file names can only be checked through the command line, the file searching method and the file searching device can respond to the input of the front end of a user to quickly search the number of files included in each level of folder and the file size information of the folder by utilizing the attribute information of the files stored in the searching database and the information of each level of folder, namely the hierarchical structure information of the files.
Step S520, based on the folder information of each level, calling and displaying files included under the folders of each level from the distributed file system;
here, the search by the parent folder path may show information of folders of each level. As shown in fig. 5B, in the column "data management 51" of the front-end interactive page, a tab page of "my data set 511" is selected, and basic information such as a name 513, a file format 514, a file number 515, and a data size 516 of a parent folder "VOT 2018" is displayed by searching through a file path 512 "lustre/sunny/VOT 2018", and all files 517 included under the parent folder "VOT 2018" such as subfolders "butterfly", "leaves", "matrix", and text files "list.txt", JSON files "VOT 2018_ new.json", information such as a file size and an upload time may be further displayed for each file, and the number of files included in a corresponding folder may be further displayed for files of folder types.
Therefore, files under each level of folders can be visually displayed by searching according to the path of the parent folder on the front-end interactive interface by searching the information of the folders at all levels in the database, namely, a group of data sets comprising a plurality of files can be displayed according to the hierarchy in a front-end display mode.
Step S530, responding to the receiving of the name of the parent folder and the name of the first file input by the user on the front-end interactive interface, and retrieving the relative path of the first file from the search database;
here, the fuzzy search may be performed on the first file through the parent file name and the file name of the first file based on the search database, and a relative path of the first file is obtained.
Step S540, based on the relative path of the first file, calling and displaying the content of the first file from the distributed file system;
therefore, fuzzy search can be carried out in the search database by combining the parent folder name and the file name, the content of the first file can be obtained and displayed through the distributed file system, and the operating efficiency is improved.
As shown in fig. 5B, the text file "list.txt" is used as the first file for explanation, the relative path of the file is "VOT 2018/", the "list.txt" 518 is clicked on the front-end interaction page, and the "file preview 52" interface shown in fig. 5C is popped up, and the content in the text file "list.txt" 518 is shown on the interface.
As shown in fig. 5D, a picture file "0000003. jpg" is taken as the first file, the relative path of the picture file is "VOT 2018/basetball/color", and "0000003. jpg" 519 is clicked on the front-end interaction page, so that a "file preview 52" interface shown in fig. 5E is popped up, and the content in the picture file "0000003. jpg" 519 is shown on the interface.
Step S550, in the process of displaying each level of folder information in the parent file path, in response to receiving a download operation for a second file, downloading the second file through the storage information of the second file stored in the search database.
Therefore, the storage information for downloading of the files stored in the search database is adopted, the files can be downloaded and checked directly in response to interface operation, and other downloading services do not need to be installed.
In the embodiment of the application, a distributed file system is used for storing files, attribute information and folder information of the files are recorded in a search database, a visual storage structure is constructed, data can be visually displayed according to levels in a front-end display mode, and details of the file data can be visually previewed and viewed; meanwhile, files can be directly downloaded by utilizing file storage information stored in the search database, so that the operating efficiency is improved, and the data checking speed of researchers is increased.
It should be noted that the various embodiments in this specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and like parts between various embodiments may be referred to each other.
The data storage method is described below with reference to a specific embodiment, but it should be noted that the specific embodiment is only for better describing the present application and is not to be construed as limiting the present application.
According to the file hierarchical structure information in the search database, the file information can be displayed according to the hierarchy, and pictures and videos can be obtained to be viewed. Fig. 6 is a schematic flowchart of a data storage method according to an embodiment of the present application, and as shown in fig. 6, the method includes the following steps:
step S610, recording the attribute information of the file when uploading the file;
here, when uploading a file to a distributed file system such as ceph, attribute information of the file is recorded, and stored meta information is analyzed line by line when all files are uploaded to obtain the attribute information of the file, two pieces of attribute information are mainly required, one is a relative path of the file, and the other is the size of the file.
Step S620, storing the attribute information and the folder information of the file in a search database in a JSON form;
here, the attribute information of the file is processed in sequence and recorded under the nodes of the file, and the nodes of the file are inserted into a search database; and processing the folder information, recording the folder information under the nodes of the folder, and inserting the nodes of the folder into a search database. Wherein, each node has the following information:
parentDirectory, a parent folder directory for recording under which folder;
filename for recording a file name;
size, file size;
fileType, file type, record whether the file is a file or a folder;
the number of files is 1 if the number of files is a file, and the number of files in the folder is the number of files in the folder if the number of files is a folder;
updateTime, update time stamp;
an area region, a bucket container and an objectKey object key, namely storage information of the file, which are used for downloading the file;
level, which is used to count the size of each folder (used during processing, not stored in the database).
The process of analyzing the attribute information of the file and storing the attribute information in the search database can be realized by the following steps:
s1, create a root node "/" of a set of files;
s2, the relative path of each file is analyzed to obtain the folder where the file is located and the folder information. Each folder information is put into a mapping file of the folder, the keys of the mapping file store the relative path of the folder, and the values store the folder information for recording the size of the folder and the number of files. When the last layer is resolved, the information of the file is added. For example, for a file a/b/c/d.jpg, the folders a, a/b/, a/b/c and the file a/b/c/d.jpg are sequentially analyzed, and if each folder has another file, the file is also recorded under the corresponding node information;
s3, processing the attribute information of the file in batch, recording the attribute information under the node of the file, and inserting the node of the file into a search database;
s4, after analyzing the attribute information of all files, sorting the folder information in reverse order according to the hierarchy, and accumulating the size information of the subfolders;
and S5, after finishing the folder information, inserting folder nodes into the database, wherein the folder nodes comprise the root nodes.
Step S630, the file is viewed in a front-end display manner.
Here, there are several functions mainly when exposed through the front-end interactive interface: searching information which can display each level of folder according to a parent folder path; fuzzy search can be carried out on the files by combining the parent folder and the file names; clicking on the json/video/picture file can download and view the file through stored information such as a container and an object key.
In the embodiment of the application, the attribute information of the file is recorded by using the search database, the file is stored by using the distributed file system, a visual storage is constructed, and fuzzy search can be carried out. Therefore, the data user can know the file data more fully and more carefully, and can search files fuzzily according to the file name. Meanwhile, the File data can be prevented from being downloaded by installing services such as FTP (File Transfer Protocol), SCP (Secure Copy, Secure shell Protocol login based Secure remote File Copy command in Linux system) and the like.
The file storage method can be used for visually checking the file content, so that whether the file has errors or not can be known and found conveniently in time, and the file data can be downloaded by the data storage method provided by the embodiment of the application, so that the file storage method is more convenient to download than a server.
Based on the foregoing embodiments, an embodiment of the present application further provides a data storage device, where the data storage device includes modules, and sub-modules and units included in the modules, and may be implemented by a processor in an electronic device; of course, the implementation can also be realized through a specific logic circuit; in the implementation process, the Processor may be a Central Processing Unit (CPU), a microprocessor Unit (MPU), a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), or the like.
Fig. 7 is a schematic structural diagram of a data storage device according to an embodiment of the present application, and as shown in fig. 8, the device 700 includes a parsing module 710, a determining module 720, and a storage module 730, where:
the parsing module 710 is configured to parse at least one file stored in the distributed file system to obtain attribute information of each file; wherein the attribute information at least comprises relative path and file size information of each file;
the determining module 720 is configured to determine, based on the relative path and file size information of each file, folder information of each level of folder in which the corresponding file is located;
the storage module 730 is configured to store the attribute information of each file and the corresponding folder information of each level in a search database in a lightweight data exchange format; and the attribute information of each file and the corresponding folder information at each level are used for displaying the content of each file according to the hierarchy through a front-end interactive interface.
In some possible embodiments, the parsing module 710 includes: the first recording submodule is used for recording the stored meta information of the at least one file in the process of uploading the at least one file to the distributed file system; a first parsing submodule for parsing the meta information line by line to obtain attribute information of each file
In some possible embodiments, the determining module 720 includes: the second analysis submodule is used for analyzing the relative path of each file layer by layer to obtain the relative path of each level of folder corresponding to the file; the first determining submodule is used for determining the folder hierarchy of each level of folder based on the relative path of each level of folder; the accumulation submodule is used for accumulating the file size information of the files contained in the folders at all levels layer by layer according to the folder levels to obtain the file size information of the folders at all levels; and the second determining submodule is used for taking the file size information of the folders at all levels as the folder information.
In some possible embodiments, the storage module 730 includes a second recording sub-module, configured to record attribute information of each of the files under a node of each of the files in the lightweight data exchange format; the first storage submodule is used for storing the relative paths of all levels of folders and corresponding folder information in a mapping file in a key-value pair mode; the third recording submodule is used for recording the folder information of all levels of folders under the nodes of the corresponding folders in the lightweight data exchange format based on the mapping file; and the second storage submodule is used for sequentially inserting the node of each file and the nodes of all levels of folders into the search database according to the folder hierarchy.
In some possible embodiments, each of the files includes at least one of: pictures, videos, text files; the nodes of each file at least comprise the following fields: the file updating method comprises the following steps of (1) storing information of a father file directory, a file name, a file size, a file type, the number of files, an updating timestamp and a file; wherein the storage information of the file at least comprises one of the following information: region, container, object key.
In some possible embodiments, the apparatus further comprises a first retrieving module and a first presenting module, wherein: the first retrieval module is used for responding to a parent folder path input by a user and retrieving folder information of each level below the parent folder path from the search database; the first display module is used for calling and displaying files included under each level of folders from the distributed file system based on the folder information at each level.
In some possible embodiments, the apparatus further comprises a second retrieving module and a second presenting module, wherein: the second retrieval module is used for responding to the name of a parent folder and the name of a first file which are input by a user and received on the front-end interactive interface, and retrieving the relative path of the first file from the search database; the second display module is used for calling and displaying the content of the first file from the distributed file system based on the relative path of the first file.
In some possible embodiments, the apparatus further includes a downloading module, configured to, in the process of displaying information of each level of folders in the parent file path, in response to receiving a downloading operation for a second file, download the second file through the storage information of the second file stored in the search database.
Here, it should be noted that: the above description of the apparatus embodiments, similar to the above description of the method embodiments, has similar beneficial effects as the method embodiments. For technical details not disclosed in the embodiments of the apparatus of the present application, reference is made to the description of the embodiments of the method of the present application for understanding.
It should be noted that, in the embodiment of the present application, if the data storage method is implemented in the form of a software functional module and is sold or used as a standalone product, the data storage method may also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling an electronic device (which may be a smartphone with a camera, a tablet computer, etc.) to execute all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read Only Memory (ROM), a magnetic disk, or an optical disk. Thus, embodiments of the present application are not limited to any specific combination of hardware and software.
Correspondingly, the present application provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps in the data storage method described in any of the above embodiments. Correspondingly, in an embodiment of the present application, a chip is further provided, where the chip includes a programmable logic circuit and/or program instructions, and when the chip runs, the chip is configured to implement the steps in the data storage method in any of the above embodiments. Correspondingly, in an embodiment of the present application, there is also provided a computer program product, which is used to implement the steps in the data storage method in any of the above embodiments when the computer program product is executed by a processor of an electronic device.
Based on the same technical concept, the embodiment of the present application provides an electronic device, which is used for implementing the data storage method described in the above method embodiment. Fig. 8 is a hardware entity diagram of an electronic device according to an embodiment of the present application, as shown in fig. 8, the electronic device 800 includes a memory 810 and a processor 820, the memory 810 stores a computer program that can be executed on the processor 820, and the processor 820 implements steps in any data storage method according to the embodiment of the present application when executing the computer program.
The Memory 810 is configured to store instructions and applications executable by the processor 820, and may also buffer data (e.g., image data, audio data, voice communication data, and video communication data) to be processed or already processed by the processor 820 and modules in the electronic device, and may be implemented by a FLASH Memory (FLASH) or a Random Access Memory (RAM).
The steps of any of the above-described data storage methods are implemented when the processor 820 executes a program. The processor 820 generally controls the overall operation of the electronic device 800.
The Processor may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understood that the electronic device implementing the above-mentioned processor function may be other electronic devices, and the embodiments of the present application are not particularly limited.
The computer storage medium/Memory may be a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Programmable Read Only Memory (EPROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a magnetic Random Access Memory (FRAM), a Flash Memory (Flash Memory), a magnetic surface Memory, an optical Disc, or a Compact Disc Read-Only Memory (CD-ROM), and the like; and may be various electronic devices such as mobile phones, computers, tablet devices, personal digital assistants, etc., including one or any combination of the above-mentioned memories.
Here, it should be noted that: the above description of the storage medium and device embodiments is similar to the description of the method embodiments above, with similar advantageous effects as the method embodiments. For technical details not disclosed in the embodiments of the storage medium and apparatus of the present application, reference is made to the description of the embodiments of the method of the present application for understanding.
It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in the various embodiments of the present application, the sequence numbers of the above-mentioned processes do not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described device embodiments are merely illustrative, for example, the division of the unit is only a logical functional division, and there may be other division ways in actual implementation, such as: multiple units or components may be combined, or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the coupling, direct coupling or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between the devices or units may be electrical, mechanical or other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units; can be located in one place or distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiments of the present application.
In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be separately regarded as one unit, or two or more units may be integrated into one unit; the integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.
Alternatively, the integrated units described above in the present application may be stored in a computer-readable storage medium if they are implemented in the form of software functional modules and sold or used as independent products. Based on such understanding, the technical solutions of the embodiments of the present application may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing an automatic test line of a device to perform all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a removable storage device, a ROM, a magnetic or optical disk, or other various media that can store program code.
The methods disclosed in the several method embodiments provided in the present application may be combined arbitrarily without conflict to obtain new method embodiments.
The features disclosed in the several method or apparatus embodiments provided in the present application may be combined arbitrarily, without conflict, to arrive at new method embodiments or apparatus embodiments.
The above description is only for the embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (11)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110815618.3A CN113448929A (en) | 2021-07-19 | 2021-07-19 | Data storage method and device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110815618.3A CN113448929A (en) | 2021-07-19 | 2021-07-19 | Data storage method and device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113448929A true CN113448929A (en) | 2021-09-28 |
Family
ID=77816738
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110815618.3A Pending CN113448929A (en) | 2021-07-19 | 2021-07-19 | Data storage method and device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113448929A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114168534A (en) * | 2021-11-04 | 2022-03-11 | 广州广电运通信息科技有限公司 | Method, system, device and medium for accelerating ES file to carry out local full-text retrieval |
| CN116011404A (en) * | 2022-08-17 | 2023-04-25 | 深圳市沃享科技有限公司 | Json file reading and writing method based on simplified data format |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105912710A (en) * | 2016-04-28 | 2016-08-31 | 深圳市金立通信设备有限公司 | File management method and terminal |
| CN107729415A (en) * | 2017-09-27 | 2018-02-23 | 郑州云海信息技术有限公司 | A kind of compressed file content displaying method and device |
| US20180165300A1 (en) * | 2015-06-17 | 2018-06-14 | Qumulo, Inc. | Filesystem capacity and performance metrics and visualizations |
| US20190121874A1 (en) * | 2017-10-23 | 2019-04-25 | Dropbox, Inc. | Storage space manager |
| CN109983431A (en) * | 2016-12-15 | 2019-07-05 | 甲骨文国际公司 | System and method for storing the list retrieval in equipment |
| CN110674101A (en) * | 2019-09-27 | 2020-01-10 | 北京金山云网络技术有限公司 | Data processing method, device and cloud server for file system |
| CN111045994A (en) * | 2019-12-25 | 2020-04-21 | 山东方寸微电子科技有限公司 | KV database-based file classification retrieval method and system |
-
2021
- 2021-07-19 CN CN202110815618.3A patent/CN113448929A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180165300A1 (en) * | 2015-06-17 | 2018-06-14 | Qumulo, Inc. | Filesystem capacity and performance metrics and visualizations |
| CN105912710A (en) * | 2016-04-28 | 2016-08-31 | 深圳市金立通信设备有限公司 | File management method and terminal |
| CN109983431A (en) * | 2016-12-15 | 2019-07-05 | 甲骨文国际公司 | System and method for storing the list retrieval in equipment |
| CN107729415A (en) * | 2017-09-27 | 2018-02-23 | 郑州云海信息技术有限公司 | A kind of compressed file content displaying method and device |
| US20190121874A1 (en) * | 2017-10-23 | 2019-04-25 | Dropbox, Inc. | Storage space manager |
| CN110674101A (en) * | 2019-09-27 | 2020-01-10 | 北京金山云网络技术有限公司 | Data processing method, device and cloud server for file system |
| CN111045994A (en) * | 2019-12-25 | 2020-04-21 | 山东方寸微电子科技有限公司 | KV database-based file classification retrieval method and system |
Non-Patent Citations (1)
| Title |
|---|
| 陈东辉等: "气象台站历史沿革信息检索可视化系统设计与实现", 计算机应用, vol. 41, no. 1, pages 119 - 124 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114168534A (en) * | 2021-11-04 | 2022-03-11 | 广州广电运通信息科技有限公司 | Method, system, device and medium for accelerating ES file to carry out local full-text retrieval |
| CN116011404A (en) * | 2022-08-17 | 2023-04-25 | 深圳市沃享科技有限公司 | Json file reading and writing method based on simplified data format |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR101653268B1 (en) | Processing method of tagged information and the client-server system for the same | |
| US8122069B2 (en) | Methods for pairing text snippets to file activity | |
| US8713078B2 (en) | Method for building taxonomy of topics and categorizing videos | |
| US7797350B2 (en) | System and method for processing downloaded data | |
| JP5147947B2 (en) | Method and system for generating search collection by query | |
| TW200928811A (en) | Collecting and presenting temporal-based action information | |
| BRPI0520649A2 (en) | search for structured data | |
| CN108255972A (en) | A kind of text searching method and system | |
| US20100211570A1 (en) | Distributed system | |
| US20140188931A1 (en) | Lexicon based systems and methods for intelligent media search | |
| CN102077179A (en) | document management system | |
| US10878020B2 (en) | Automated extraction tools and their use in social content tagging systems | |
| US8082276B2 (en) | Techniques using captured information | |
| EP2257895A1 (en) | Electronic resource annotation | |
| US20180307744A1 (en) | Named entity-based category tagging of documents | |
| US9870422B2 (en) | Natural language search | |
| US8595183B2 (en) | Systems and methods for providing enhanced content portability in a word page module | |
| US10719492B1 (en) | Automatic reconciliation and consolidation of disparate repositories | |
| CN113448929A (en) | Data storage method and device, equipment and storage medium | |
| CN101089853B (en) | Apparatus and method for browsing contents | |
| TW201820181A (en) | Method and device for processing the accumulation searching | |
| TW200928812A (en) | System and method for assisting a user in constructing of a search query | |
| CN106815248B (en) | Website analysis method and device | |
| CN115630170A (en) | Document recommendation method, system, terminal and storage medium | |
| US20050262054A1 (en) | Item type specific structured search |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210928 |