CN117149711A - Asynchronous generation and downloading method, system and equipment for large files of distributed system - Google Patents
Asynchronous generation and downloading method, system and equipment for large files of distributed system Download PDFInfo
- Publication number
- CN117149711A CN117149711A CN202311035879.9A CN202311035879A CN117149711A CN 117149711 A CN117149711 A CN 117149711A CN 202311035879 A CN202311035879 A CN 202311035879A CN 117149711 A CN117149711 A CN 117149711A
- Authority
- CN
- China
- Prior art keywords
- file
- generation
- record
- generated
- configuration record
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000012544 monitoring process Methods 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 1
- 230000006378 damage Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method, a system and equipment for asynchronously generating and downloading large files of a distributed system, which relate to the field of file generation and downloading; generating a unique number according to the file generation request; storing the unique number, the request time and the configuration record into a file of a relational database to generate a configuration table; recording the current file generation state as not generated; acquiring a configuration record with a current file generation state being not generated; executing the generation logic to generate a file according to the configuration record, the file processing class name and the method name, and storing the file; then updating the file according to the unique number corresponding to the current configuration record to generate a configuration table state; and downloading the corresponding file according to the unique number. The invention can avoid competing with the service system for server resources and improve the availability of the service system.
Description
Technical Field
The invention relates to the field of file generation and downloading, in particular to a method, a system and equipment for asynchronously generating and downloading large files in a distributed system.
Background
There is often a functional need in distributed systems for users to create and download large files, such as in a financial company ledger system, where the user at the end of the month creates a ledger PDF file for the entire ledger and downloads it for printing. Because the account cover file is large, the generated file has long duration, and resources such as server memory, CPU, I/O and the like are occupied in the process of generating the file, so that the response of other functions of the system is very slow, and the memory overflows out of the system when serious.
Therefore, a method is needed to separate the generation and downloading of large files from the service system to operate independently, and to avoid influencing the normal operation of the service system through physical isolation.
Disclosure of Invention
The invention aims to provide a method, a system and equipment for asynchronously generating and downloading large files of a distributed system, which can avoid competing with a service system for server resources and improve the usability of the service system.
In order to achieve the above object, the present invention provides the following solutions:
a method for asynchronously generating and downloading a large file of a distributed system comprises the following steps:
acquiring a file generation request and a configuration record of a user by utilizing a service system; the configuration record includes: the generated file name, file processing class name and method name, file storage path, current user ID and storage record;
generating a unique number according to the file generation request;
storing the unique number, the request time and the configuration record into a file of a relational database to generate a configuration table; recording the current file generation state as not generated;
acquiring a configuration record with a current file generation state being not generated; executing the generation logic to generate a file according to the configuration record, the file processing class name and the method name, and storing the file; then updating the file according to the unique number corresponding to the current configuration record to generate a configuration table state;
and downloading the corresponding file according to the unique number.
Optionally, the obtaining, by using the service system, the file generation request and the configuration record of the user specifically includes:
acquiring a file generation request of a user;
and encapsulating the generated file request into json character strings in a key value mode.
Optionally, the current file generation state is obtained as an undepicted configuration record; and executing the generation logic to generate a file according to the configuration record, the file processing class name and the method name, and storing the file, wherein the method specifically comprises the following steps:
querying all records of the file generation state from the file generation configuration table at intervals of set time;
storing records with file generation states not generated in a memory queue according to a first-in first-out principle;
monitoring a memory queue, and inquiring a file to generate a configuration table according to a unique number corresponding to a generated record after monitoring that the new generated record exists;
and executing the generation logic to generate a file according to the file processing class name and the method name after the record with the file generation state being not generated exists, and storing the file under the catalog corresponding to the configuration record.
Optionally, the downloading the corresponding file according to the unique number specifically includes:
acquiring corresponding storage catalogs and file names in the file generation configuration table query according to the unique numbers;
and acquiring the file in the form of a file stream from the storage directory according to the file name.
An asynchronous generation and download system for large files in a distributed system, comprising:
the file generation request acquisition module is used for acquiring a file generation request and a configuration record of a user by utilizing a service system; the configuration record includes: the generated file name, file processing class name and method name, file storage path, current user ID and storage record;
the file processing request receiving module is used for generating a unique number according to the file generating request;
the file generation module is used for storing the unique number, the request time and the configuration record into a file generation configuration table of the relational database; recording the current file generation state as not generated; acquiring a configuration record with a current file generation state being not generated; executing the generation logic to generate a file according to the configuration record, the file processing class name and the method name, and storing the file; then updating the file according to the unique number corresponding to the current configuration record to generate a configuration table state;
and the file downloading module is used for downloading the corresponding file according to the unique number.
An asynchronous generation and download device for large files in a distributed system, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, which when executed by the processor, implement the method as described.
Optionally, the memory is a computer readable storage medium.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects:
the invention provides a method, a system and equipment for asynchronously generating and downloading large files of a distributed system, wherein a service system is utilized to acquire a file generation request and a configuration record of a user, and a configuration table is generated according to a unique number, request time and configuration record generated by the file generation request and the file generation configuration record stored in a relational database; further, a thread for generating the file is realized by utilizing the file generation configuration table; the file generation method uses the thread pool to reduce the resource consumption of thread creation and destruction, and the two threads logically decouple data scanning and file generation through memory queue communication, thereby reducing the complexity of the system. The invention physically isolates the large file generating and downloading processing process from the service system, avoids competing with the service system for server resources, and improves the availability of the service system.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a method for asynchronously generating and downloading large files in a distributed system;
FIG. 2 is a flow chart of file generation;
fig. 3 is a schematic diagram of a distributed system large file asynchronous generation and downloading system according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention aims to provide a method, a system and equipment for asynchronously generating and downloading large files of a distributed system, which can avoid competing with a service system for server resources and improve the usability of the service system.
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
As shown in FIG. 1, the method for asynchronously generating and downloading the large file of the distributed system provided by the invention comprises the following steps:
s101, acquiring a file generation request and a configuration record of a user by using a service system; the configuration record includes: the generated file name, file processing class name and method name, file storage path and current user ID and storage record.
S101 specifically includes:
and acquiring a file generation request of the user. Generating file requests includes, but is not limited to: account cover mechanism, month.
And encapsulating the generated file request into json character strings in a key value mode. For example, { "orgCode":1000, "monta": "202101" }.
S102, generating a unique number according to the file generation request.
S103, storing the unique number, the request time and the configuration record into a file generation configuration table of a relational database; and records the current file generation status as not generated, which can be represented by status=0.
S104, acquiring a configuration record with the current file generation state being not generated; executing the generation logic to generate a file according to the configuration record, the file processing class name and the method name, and storing the file; and then updating the file according to the unique number corresponding to the current configuration record to generate a configuration table state.
S105 specifically includes:
and querying all records of the file generation state from the file generation configuration table at intervals of set time.
Records whose file generation status is not generated are stored in a memory queue according to a first-in first-out principle.
And monitoring the memory queue, and after monitoring that a new generation record exists, inquiring the file to generate a configuration table according to the unique number corresponding to the generation record.
And executing the generation logic to generate a file according to the file processing class name and the method name after the record with the file generation state being not generated exists, and storing the file under the catalog corresponding to the configuration record.
S104 and S105 are thread pools of two threads for file generation, as shown in FIG. 2, thread 1 queries all records of file generation state status=0 from a file generation configuration table every 2 minutes, and stores the records in a memory queue according to a first-in first-out principle; and the thread 2 monitors the memory queue, queries the file generation configuration table according to the unique number of the record after monitoring that a new processing record exists, if the file generation state status=1 is queried, the file generation is not processed, if the file generation state status=0 is queried, then the generation logic is executed according to the file processing class name and the method name to generate the file, the file is stored in the directory appointed by the record, and the file generation configuration table state status=1 is updated according to the unique number of the current processing record.
S105, downloading the corresponding file according to the unique number.
S105 specifically includes:
and acquiring corresponding storage catalogs and file names in the file generation configuration table query according to the unique numbers.
And acquiring the file in the form of a file stream from the storage directory according to the file name.
S105 further includes: providing a file downloading interface embedded service system, wherein the interface receives the access user ID, queries all file generation records with the user ID equal to the current user ID from a file generation configuration table, and arranges and displays the file generation records according to the request time in a reverse order.
The file downloading interface receives a file downloading request from a user, transmits a unique number of a downloaded file to the file downloading module, inquires a storage directory and a file name from a file generating configuration table, acquires the file from the storage directory, and returns the file to the browser in a file stream form.
The file downloading module provides a unified file downloading interface, and a user can track all the historical file generating states through the interface, so that the working efficiency is improved.
As shown in fig. 3, the system for asynchronously generating and downloading large files in a distributed system provided by the present invention includes:
the file generation request acquisition module is used for acquiring a file generation request and a configuration record of a user by utilizing a service system; the configuration record includes: the generated file name, file processing class name and method name, file storage path and current user ID and storage record.
And the file processing request receiving module is used for generating a unique number according to the file generating request.
The file generation module is used for storing the unique number, the request time and the configuration record into a file generation configuration table of the relational database; recording the current file generation state as not generated; acquiring a configuration record with a current file generation state being not generated; executing the generation logic to generate a file according to the configuration record, the file processing class name and the method name, and storing the file; and then updating the file according to the unique number corresponding to the current configuration record to generate a configuration table state.
And the file downloading module is used for downloading the corresponding file according to the unique number.
In order to execute the method corresponding to the above embodiment to achieve the corresponding functions and technical effects, the present invention further provides a distributed system large file asynchronous generation and downloading device, including: at least one processor, at least one memory, and computer program instructions stored in the memory, which when executed by the processor, implement the method.
The memory is a computer-readable storage medium.
Based on the above description, the technical solution of the present invention may be embodied in essence or a part contributing to the prior art or a part of the technical solution in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method of the embodiments of the present invention. And the aforementioned computer storage medium includes: various media capable of storing program codes, such as a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to assist in understanding the methods of the present invention and the core ideas thereof; also, it is within the scope of the present invention to be modified by those of ordinary skill in the art in light of the present teachings. In view of the foregoing, this description should not be construed as limiting the invention.
Claims (7)
1. A method for asynchronously generating and downloading a large file in a distributed system, comprising:
acquiring a file generation request and a configuration record of a user by utilizing a service system; the configuration record includes: the generated file name, file processing class name and method name, file storage path, current user ID and storage record;
generating a unique number according to the file generation request;
storing the unique number, the request time and the configuration record into a file of a relational database to generate a configuration table; recording the current file generation state as not generated;
acquiring a configuration record with a current file generation state being not generated; executing the generation logic to generate a file according to the configuration record, the file processing class name and the method name, and storing the file; then updating the file according to the unique number corresponding to the current configuration record to generate a configuration table state;
and downloading the corresponding file according to the unique number.
2. The method for asynchronously generating and downloading large files in a distributed system according to claim 1, wherein said obtaining the request and the configuration record of the generated files by the user by using the service system specifically comprises:
acquiring a file generation request of a user;
and encapsulating the generated file request into json character strings in a key value mode.
3. The method for asynchronously generating and downloading large files in a distributed system according to claim 1, wherein the obtaining the current file generation status is an un-generated configuration record; and executing the generation logic to generate a file according to the configuration record, the file processing class name and the method name, and storing the file, wherein the method specifically comprises the following steps:
querying all records of the file generation state from the file generation configuration table at intervals of set time;
storing records with file generation states not generated in a memory queue according to a first-in first-out principle;
monitoring a memory queue, and inquiring a file to generate a configuration table according to a unique number corresponding to a generated record after monitoring that the new generated record exists;
and executing the generation logic to generate a file according to the file processing class name and the method name after the record with the file generation state being not generated exists, and storing the file under the catalog corresponding to the configuration record.
4. The method for asynchronously generating and downloading large files in a distributed system according to claim 1, wherein said downloading the corresponding files according to the unique numbers comprises:
acquiring corresponding storage catalogs and file names in the file generation configuration table query according to the unique numbers;
and acquiring the file in the form of a file stream from the storage directory according to the file name.
5. An asynchronous generation and download system for large files in a distributed system, comprising:
the file generation request acquisition module is used for acquiring a file generation request and a configuration record of a user by utilizing a service system; the configuration record includes: the generated file name, file processing class name and method name, file storage path, current user ID and storage record;
the file processing request receiving module is used for generating a unique number according to the file generating request;
the file generation module is used for storing the unique number, the request time and the configuration record into a file generation configuration table of the relational database; recording the current file generation state as not generated; acquiring a configuration record with a current file generation state being not generated; executing the generation logic to generate a file according to the configuration record, the file processing class name and the method name, and storing the file; then updating the file according to the unique number corresponding to the current configuration record to generate a configuration table state;
and the file downloading module is used for downloading the corresponding file according to the unique number.
6. An apparatus for asynchronously generating and downloading large files in a distributed system, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, which when executed by the processor, implement the method of any one of claims 1-4.
7. A distributed system large file asynchronous generation and download device according to claim 6, wherein the memory is a computer readable storage medium.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311035879.9A CN117149711A (en) | 2023-08-17 | 2023-08-17 | Asynchronous generation and downloading method, system and equipment for large files of distributed system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311035879.9A CN117149711A (en) | 2023-08-17 | 2023-08-17 | Asynchronous generation and downloading method, system and equipment for large files of distributed system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117149711A true CN117149711A (en) | 2023-12-01 |
Family
ID=88885908
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311035879.9A Pending CN117149711A (en) | 2023-08-17 | 2023-08-17 | Asynchronous generation and downloading method, system and equipment for large files of distributed system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117149711A (en) |
-
2023
- 2023-08-17 CN CN202311035879.9A patent/CN117149711A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12141183B2 (en) | Dynamic partition allocation for query execution | |
| US11586692B2 (en) | Streaming data processing | |
| US11860874B2 (en) | Multi-partitioning data for combination operations | |
| CN109997126B (en) | Event driven extraction, transformation, and loading (ETL) processing | |
| US11151137B2 (en) | Multi-partition operation in combination operations | |
| US11232100B2 (en) | Resource allocation for multiple datasets | |
| US11461334B2 (en) | Data conditioning for dataset destination | |
| US11416528B2 (en) | Query acceleration data store | |
| US10726009B2 (en) | Query processing using query-resource usage and node utilization data | |
| US11163758B2 (en) | External dataset capability compensation | |
| Varia | Cloud architectures | |
| CN101243445B (en) | Data change notifications | |
| US6505200B1 (en) | Application-independent data synchronization technique | |
| US20180089262A1 (en) | Dynamic resource allocation for common storage query | |
| EP2469410A2 (en) | System and method for scaling for a large number of concurrent users by storing a user's session state on external storage | |
| CN105095365A (en) | Information flow data processing method and device | |
| EP2590090B1 (en) | Dynamic interface to read database through remote procedure call | |
| US8751469B2 (en) | System and method for scaling for a large number of concurrent users | |
| US20220245093A1 (en) | Enhanced search performance using data model summaries stored in a remote data store | |
| CN111752959A (en) | Real-time database cross-database SQL interaction method and system | |
| US20090328043A1 (en) | Infrastructure of data summarization including light programs and helper steps | |
| US9760576B1 (en) | System and method for performing object-modifying commands in an unstructured storage service | |
| US20220245091A1 (en) | Facilitating generation of data model summaries | |
| KR101888131B1 (en) | Method for Performing Real-Time Changed Data Publish Service of DDS-DBMS Integration Tool | |
| CN112765200A (en) | Data query method and device based on Elasticissearch |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |