CN121284025A - Light-weight universal file uploading system capable of realizing second transmission - Google Patents
Light-weight universal file uploading system capable of realizing second transmissionInfo
- Publication number
- CN121284025A CN121284025A CN202511851019.1A CN202511851019A CN121284025A CN 121284025 A CN121284025 A CN 121284025A CN 202511851019 A CN202511851019 A CN 202511851019A CN 121284025 A CN121284025 A CN 121284025A
- Authority
- CN
- China
- Prior art keywords
- file
- uploading
- task
- storage
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a lightweight universal file uploading system capable of realizing second transmission, and relates to the technical field of computer data transmission and storage. The system includes a system program module, a database module, and an object storage module. The invention provides file uploading and task management functions through an HTTP interface by a system program module, and comprises uploading task generating, simple uploading and second uploading units, wherein a database module consists of a file uploading task table and a file storage table and is used for recording tasks and file information, and an object storage module is used as an independent permanent storage medium. And realizing the call of a stateless interface through uploading the certificate without a distributed session or a cache, and realizing the second transmission through judging the weight of the unified task model and the file characteristic value. The whole only depends on two basic components of the database and the object storage, the framework is light, the deployment is simplified, the uploading practicability of the light-weight general file is improved, and the problem that the uploading practicability of the light-weight general file is low in the prior art is solved.
Description
Technical Field
The invention relates to the technical field of computer data transmission and storage, in particular to a lightweight universal file uploading system capable of realizing second transmission.
Background
Firstly, a user triggers a File selection dialog box through an interactive interface of a webpage or an application, after selecting a local File, a front-end verification mechanism is triggered, for example, javaScript is utilized to carry out preliminary verification on the type and the size of the File, and File metadata (such as MIME type and size attribute) are read through a File API of the HTML5 so as to realize quick feedback and prevent invalid transmission.
Next, the file enters a preparation stage, so as to improve transmission efficiency and reliability, a modern technology often introduces a slicing process, namely, cutting a large file into data blocks with fixed length (such as 1 MB), calculating a unique file identifier by using a hash algorithm such as MD5 or SHA-256, and the like, for checking the file integrity of the server, performing a core transmission stage through an HTTP/HTTPs protocol, and uploading a standard form or more efficient asynchronous uploading, wherein the front end sends the file slices in a binary stream form to a preset server interface in a block-by-block or concurrent manner through a Fetch API or a dedicated library, and when integrating load balancing and CDN (Content Delivery Network, content distribution network) acceleration, routing a request to an optimal server node through a distributed gateway, so as to improve throughput.
After the server receives the data, firstly, carrying out security scanning, then temporarily storing the fragments in a temporary directory, triggering merging operation by the server after all the fragments arrive, restoring the original file, and checking whether the hash value of the original file is consistent with that of the client again to ensure that the data is error-free. Finally, the file is persisted to a target location, such as a local disk, and a unique access URL (Uniform Resource Locator ) is generated and returned to the client to complete the entire lightweight universal file upload loop.
The file uploading method and the file uploading system are disclosed in CN115412547A, and comprise the steps of enabling a distributed storage service to conduct slicing operation and return storage information by calling the distributed storage service deployed by a storage server in response to a slicing request sent by the front-end service, sending the storage information to the front-end service to enable the front-end service to upload each slicing file of a file to be stored to the storage server according to slicing information and storage address links, enabling the distributed storage service to conduct merging operation on each stored slicing file and return a merging result by calling the distributed storage service in response to a merging request sent by the front-end service, and sending the merging result to the front-end service to enable the front-end service to present the merging result.
The file uploading method and device for the patent application publication number CN115865894B comprise the steps of calling a target plug-in to judge the type of an object to be uploaded when the object to be uploaded is detected to be input into a file uploading tag, calling the target plug-in to obtain a network file stored in a network address when the object to be uploaded is the network address, wherein the target plug-in comprises a preloaded first dynamic scripting language program which is used for obtaining the network file stored in the network address, and uploading the network file to a database in a back-end server.
The above technology has at least the following technical problems:
In the prior art, in order to realize file deduplication and quick uploading, a complex system architecture is generally adopted, so that a plurality of outstanding problems exist in floor deployment and practical application, and particularly, the existing system generally needs to introduce a large number of external mature components such as a distributed message queue, an independent file deduplication system, a distributed cache and the like, so that the system architecture is redundant, deployment units are numerous, the resource occupancy rate is high, and the operation and maintenance cost and stability risk are obvious. In addition, the client must integrate a special SDK (Software Development Kit ) that is strongly bound to the system, resulting in strong code invasiveness, high adaptation cost, and poor portability when migrating between different platforms. Meanwhile, the system is dependent on external session or cache when processing the uploading task state, the stateful design not only increases the complexity of the system, but also severely restricts the stateless expansion capability of the system, so that the system is difficult to adapt to the scene of high availability and elastic expansion, and the problem of low uploading practicability of the lightweight general file exists.
Disclosure of Invention
In order to solve the technical problem of low practicability of uploading lightweight universal files in the prior art, the embodiment of the invention provides a lightweight universal file uploading system capable of realizing second uploading. The technical scheme is as follows:
The system comprises a system program module, a database module and an object storage module, wherein the system program module is used for providing a group of HTTP interfaces to the outside so as to realize file uploading logic and task management logic, the system program module comprises an uploading task generating unit, a simple uploading unit and a second uploading unit, the database module is used for recording file uploading tasks and file storage information and is composed of two logic data tables, the two logic data tables are a file uploading task table and a file storage table, the object storage module is used as an independent file content permanent storage medium, the object storage module is used for realizing the stateless of interface calling through the uploading credentials, a server side does not need to rely on a distributed session or a cache component to maintain task states, the simple uploading and the second uploading are abstracted into a unified file uploading task model, the file storage table is used for judging and re-judging so as to realize the second uploading function of the file, and finally, the object storage module is only dependent on the whole architecture of the object storage module.
The technical scheme provided by the embodiment of the invention has the beneficial effects that at least:
1. By means of a refined modular design, core dependence is reduced to only two basic components of database and object storage, and no distributed message queue, independent deduplication system or cache component is required to be introduced. The database layer can support complete file and task management by only two logic tables, so that the deployment unit is minimized, and the occupation of the system to server calculation, memory and storage resources is greatly reduced. The architecture simplification directly brings standardization and light weight of the deployment flow, and obviously reduces configuration complexity, operation and maintenance difficulty and comprehensive cost under the privately-owned or independent deployment scene.
2. By adopting a stateless design based on an HTTP standard interface, a server does not need to rely on a distributed session or a cache to maintain a state by transferring a task context through self-contained uploading credentials. The client does not need to integrate any special SDK, can complete all uploading flows only through conventional network requests, greatly reduces integration cost and code invasiveness, and has the advantages of being cross-platform and portable in nature. The stateless architecture simultaneously endows the system with good horizontal expansion capability, and when the system faces high concurrent requests, the service instance can be simply increased to realize elastic expansion, so that the overall usability and stability of the system are effectively improved.
3. By constructing a unified file uploading task model, simple uploading and second uploading are abstractly fused at a logic level, and efficient judgment is carried out in a file storage table by utilizing file characteristic values. The mechanism enables repeated files to be accurately identified during uploading, and realizes second transmission only by creating new task records instead of repeatedly transmitting file data, so that network bandwidth consumption is greatly saved, and particularly second-level uploading is realized under a large file scene. Meanwhile, the mechanism avoids redundant data in object storage from the source, improves the utilization rate of storage space, and realizes double optimization of transmission efficiency and storage economy.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic structural diagram of a lightweight general file uploading system capable of realizing second transmission according to an embodiment of the present invention;
FIG. 2 is a flowchart of generating a unified uploading credential of a lightweight general file uploading system capable of realizing second uploading according to an embodiment of the present invention;
FIG. 3 is a simple uploading flow chart of a lightweight general file uploading system capable of realizing second uploading provided by the embodiment of the invention;
fig. 4 is a second upload flowchart of a lightweight universal file upload system capable of implementing second upload according to an embodiment of the present invention.
Detailed Description
The technical scheme of the invention is described below with reference to the accompanying drawings.
In embodiments of the invention, words such as "exemplary," "such as" and the like are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, the term use of an example is intended to present concepts in a concrete fashion. Furthermore, in embodiments of the present invention, the meaning of "and/or" may be that of both, or may be that of either, optionally one of both.
In the embodiments of the present invention, "image" and "picture" may be sometimes used in combination, and it should be noted that the meaning of the expression is consistent when the distinction is not emphasized. "of", "corresponding (corresponding, relevant)" and "corresponding (corresponding)" are sometimes used in combination, and it should be noted that the meaning of the expression is consistent when the distinction is not emphasized.
In order to make the technical problems, technical solutions and advantages to be solved more apparent, the following detailed description will be given with reference to the accompanying drawings and specific embodiments.
The embodiment of the invention provides a lightweight general file uploading system capable of realizing second uploading, which is shown in a structural schematic diagram of the lightweight general file uploading system capable of realizing second uploading, and comprises a system program module, a database module and an object storage module, wherein the system program module is used for providing a group of HTTP interfaces to the outside so as to realize file uploading logic and task management logic, the system program module comprises an uploading task generating unit, a simple uploading unit and a second uploading unit, the database module is used for recording file uploading tasks and file storage information and is composed of two logic data tables, the two logic data tables are a file uploading task table and a file storage table, the object storage module is used as an independent file content permanent storage medium, the server side does not need to rely on a distributed session or a cache component to maintain task states, the system program module is used for uploading the simple uploading and the second to be a unified file uploading task model, the whole is used for judging file characteristic values to carry out file uploading in the whole database module, and the object storage module is only used for realizing file uploading function, and the object storage module is similar to a final framework.
In this embodiment, the system is composed of three core parts, namely a system program module, a database module and an object storage module. The system program module is responsible for providing a group of standard HTTP interfaces to the outside to realize file uploading logic and task management logic, wherein the system program module internally comprises an uploading task generating unit, a simple uploading unit and a second uploading unit which are respectively used for generating uploading certificates, processing regular file stream uploading and realizing second uploading judgment based on file characteristic values, the database module is used for recording file uploading tasks and file storage information and is only composed of two logic data tables of a file uploading task table and a file storage table to respectively manage the mapping relation between the life cycle of the task and the file characteristics and a storage path, and the object storage module is used as an independent file content permanent storage medium to be responsible for the lasting storage of actual files. By abstracting simple uploading and second uploading into a file uploading task model and judging the weight in a file storage table by utilizing a file characteristic value, the system realizes extremely simplified architecture, light deployment and low threshold of client integration on the premise of only relying on a database and an object storage two-class basic component, and meanwhile, by means of a stateless uploading credential mechanism, the dependence on a distributed session or a cache component is eliminated, the expansibility, the stability and the uploading efficiency of the system are obviously improved, and the system has obvious second transmission advantage and storage economy especially in a large file scene.
Further, the uploading task generating unit is specifically configured to receive an uploading initialization request initiated by a client, where the request at least includes a file name, generate a globally unique file uploading task identifier in response to the request, summarize and encrypt at least one of the file uploading task identifier, the file name, a client device identifier, and service source information to form a self-contained uploading certificate, and return the uploading certificate to the client for the client to carry in a subsequent uploading or second uploading operation.
It should be explained that, as shown in fig. 2, the embodiment of the invention provides a unified uploading credential generation flow chart of a lightweight general file uploading system capable of realizing second uploading, which specifically comprises that after a client initiates an uploading initialization request, an uploading task generation unit receives request information at least containing a file name, then generates a globally unique file uploading task identifier, and performs summarizing encryption on parameters such as the task ID, the file name and a client device identifier to form a self-contained security uploading credential, and the self-contained security uploading credential is returned to the client, wherein the client can carry the client to execute subsequent simple uploading flow or second uploading flow operation after acquiring the credential.
In this embodiment, the uploading task generating unit is used as a core component of the system program module, and its specific functions include receiving an uploading initialization request including at least a file name initiated by a client, generating a globally unique file uploading task identifier in response to the request, summarizing and encrypting the identifier and parameters such as the file name, the client device identifier, service source information, etc., to form a self-contained uploading credential, and returning the credential to the client for carrying in subsequent operations. The design realizes complete stateless interface call through the encrypted and packaged task credential, so that a server does not need to maintain a session state or rely on a distributed cache assembly, the security of data transmission is ensured, the integration complexity of a client is obviously reduced, the complete process access can be completed only by using a standard HTTP request, and a key technology foundation is laid for the lightweight deployment and high-availability expansion of a system.
The simple uploading unit is specifically used for receiving an uploading request initiated by a client, wherein parameters of the request are the uploading certificate and binary stream data of a file, analyzing the uploading certificate, decrypting and restoring a file uploading task identifier and other context information in the file uploading task identifier, inquiring in a file uploading task table according to the file uploading task identifier to check whether the task is submitted for the first time, creating a task record in the file uploading task table after the task passes the verification, marking the state of the task record as initialized, uploading the received file binary stream data to the object storage module, calculating a characteristic value of the file after the uploading is successful, recording the characteristic value in the file uploading task record, inquiring the file storage table according to the characteristic value, and if a record matched with the characteristic value does not exist in the file storage table, inserting a new file record in the file storage table, and updating the state of the file uploading task to be completed.
It should be explained that, as shown in fig. 3, a simple uploading flow chart of a lightweight general file uploading system capable of realizing second uploading is provided in the embodiment of the invention, and the specific flow chart is that after a client carries an uploading certificate and a file stream initiating request, a simple uploading unit firstly analyzes and decrypts the uploading certificate to restore task ID and context information, then inquires whether a file uploading task table check task is first submitted, if the task is repeated, it is determined to be abnormal, if the task is first submitted, a task record is created and the state is marked as "initialized", then the binary stream data of a file is uploaded to an object storage module, a global access address returned by the binary stream data is obtained, then a file characteristic value is calculated, and the file storage table check characteristic value is inquired, if the binary stream data does not exist, a new file record is inserted into the file storage table, and finally the file uploading task state is updated to be "completed", thereby completing the whole uploading flow chart.
In the embodiment, a simple uploading unit is used as a core functional unit for realizing conventional uploading of files by a system, and the complete workflow comprises the steps of receiving a transmission request of a client carrying uploading certificates and file binary stream data, analyzing and decrypting the uploading certificates to restore a file uploading task identifier and a task context, carrying out unique verification in a file uploading task table based on the task identifier to ensure that the task is submitted for the first time, establishing a task record after the verification is passed and setting the task record as an initialization state, uploading the file stream data to an object storage module, calculating a file characteristic value after the uploading is completed and backfilling the file characteristic value to the task record, inquiring the file storage table based on the characteristic value, creating a new file storage record if a repeated file is not found, and finally updating the task state to be completed. The unit realizes decoupling of service logic and transmission logic through a credential analysis mechanism, establishes system-level file deduplication capability through eigenvalue calculation and comparison, ensures the integrity and safety of a file transmission process, fundamentally avoids redundancy occupation of storage resources, and provides key technical support for realizing a lightweight architecture and high-efficiency storage management of a system.
The second uploading unit is specifically configured to receive a second uploading request initiated by a client, wherein a parameter of the request is an uploading certificate containing a file characteristic value and does not carry binary stream data of a file, analyze the uploading certificate, decrypt and restore the file characteristic value and a file uploading task identifier in the file uploading certificate, query in a file uploading task table according to the file uploading task identifier to check whether the task is submitted for the first time, query in a file storage table according to the restored file characteristic value after the task is checked to pass, and directly create a new file uploading task record in the file uploading task table if a record matched with the file characteristic value is queried in the file storage table, and directly mark the state of the task as completed, thereby realizing a second uploading effect without transmitting the file data.
It should be explained that, as shown in fig. 4, the embodiment of the invention provides a second uploading flow chart of a lightweight general file uploading system capable of realizing second uploading, which specifically comprises the steps that after a client initiates a second uploading request with an uploading certificate without a file stream, a second uploading unit first analyzes and decrypts the uploading certificate to restore a file characteristic value and a task ID, then queries whether a file uploading task table check task is first submitted, if the task is repeated, judges that the task is abnormal, if the task is first submitted, continues querying whether a file storage table check characteristic value exists, if the characteristic value does not exist, returns an empty data prompt client, the client can be converted into a simple uploading flow, if the characteristic value exists, directly creates a new task record in the file uploading task table, marks the state as completed, and realizes the second uploading success with zero transmission overhead.
In the embodiment, an uploading unit is used as a core functional module for realizing file extremely-fast transmission, and the complete workflow comprises the steps of receiving a second uploading request which is initiated by a client and only contains file characteristic value uploading certificates and does not carry any file binary stream data, analyzing and decrypting the uploading certificates, restoring key information such as file characteristic values, file uploading task identifiers and the like, carrying out validity check in a file uploading task table based on the task identifiers to ensure that the tasks are not repeatedly submitted, carrying out accurate matching query in a file storage table according to the restored file characteristic values after the check is passed, and directly creating new task records in the file uploading task table and marking the state as completed by a system when confirming that the file records with the same characteristic values exist, so that a second uploading effect is realized by completely skipping a physical file transmission link. The mechanism thoroughly eliminates network bandwidth consumption generated by repeated file transmission on the premise of ensuring data consistency through intelligent eigenvalue comparison and task state reconstruction, so that millisecond-level response is realized by uploading a large file, the system transmission efficiency and user experience are greatly improved, writing load and storage redundancy of object storage are remarkably reduced, and an efficient and energy-saving intelligent file transmission solution is formed.
Further, the file uploading task list is used for recording full life cycle information taking a task as a core, and the included fields are at least used for recording a unique identifier of the file uploading task as a service main key of the list, a file characteristic value, an uploading task type used for distinguishing simple uploading from second uploading, an uploading task state used for identifying that the task is in a state of initialization, in progress or completed and the like, at least one of a file name, a file size, a file extension and a file classification, and at least one of a client device identifier, a client operating system platform and a service source module.
In this embodiment, the file upload task table is used as a core data table of the system database module, and plays a key role in recording the full life cycle information of the file upload task. The table structure design comprises the following core fields of a file uploading task unique identifier as a service main key to ensure the global uniqueness of the task, a file characteristic value used for establishing an association relation with a file storage table, an uploading task type field for clearly distinguishing two modes of simple uploading and second uploading, an uploading task state field for completely tracking the state transition of the whole process from initialization, in progress to completion, and simultaneously recording basic file attributes such as file names, file sizes, extension names, file classification and the like, and context information such as a client device identifier, an operating system platform, a service source module and the like. The table realizes complete metadata management of the uploading task through a refined field design, not only provides a data-level support for a unified task model, but also creates a unified task processing framework for the system through standardization of task types and states. The collaborative work mode of the system and the file storage table enables the system to realize complex task state tracking, service tracing analysis and file deduplication logic while maintaining a very simple architecture, and provides a solid data base for efficient operation and maintenance management of the system.
Further, the file storage table is used for establishing a unique mapping relation between a file characteristic value and a physical storage position, and the included field is at least used for recording the file characteristic value, which is used as a business main key of the table and has a unique constraint, the actual storage path of the file in the object storage module, and the file size, wherein the same file characteristic value of the file storage table uniquely corresponds to one stored actual file.
In this embodiment, the file storage table is used as a core component of the system database module, and is specifically used to establish a unique mapping relationship between the file feature value and the physical storage location. The table structural design comprises the following key fields of taking a file characteristic value as a service main key and having uniqueness constraint, ensuring that each file characteristic value has uniqueness in a system, accurately pointing to a physical storage position of a file according to an actual storage path of the file in an object storage module, and obtaining basic attribute information such as file size. By means of the design, the file storage table realizes accurate mapping of the unique corresponding characteristic value of the same file to one stored actual file. The innovative design of the table forms a data base stone of a system deduplication mechanism, and provides an authoritative weight judging basis for the second transmission function through the strong correlation of the characteristic value and the storage path. The mapping mechanism not only ensures the uniqueness and consistency of file storage, but also fundamentally avoids the occupation of repeated files to the storage space, and remarkably improves the utilization rate of the storage space. Meanwhile, the cooperation of the table and the file uploading task table enables the system to realize efficient file retrieval and management while maintaining a very simple architecture, and provides a key technical guarantee for light deployment and high-performance operation of the system.
Furthermore, the file uploading task list and the file storage list are logically associated through the file characteristic values, one record exists in the file storage list at most for the same file characteristic value, and a plurality of file uploading task records can be corresponding to the file uploading task list, so that the mapping relation between the plurality of uploading task records and the single file storage record based on the same file is realized, and the file deduplication and second transmission functions are supported.
In the embodiment, a file uploading task table and a file storage table are logically associated through file characteristic values to form an efficient one-to-many mapping relation, wherein each file characteristic value in the file storage table is at most provided with a record which uniquely corresponds to a stored actual file, and in the file uploading task table, the same file characteristic value can correspond to a plurality of uploading task records, and the multiple uploading actions based on the same file are accurately recorded. The well-designed association mechanism forms a data base of a system core deduplication architecture, and through separate management of task records and file entities, not only is the business context and state track of each uploading task completely reserved, but also the uniqueness of physical file storage is ensured. The design directly supports the realization of a second transmission function, when the fact that the files with the same characteristic value exist is detected, the system only needs to newly establish task records and repeatedly transmit and store the files, so that the uploading efficiency is greatly improved, the network bandwidth consumption is reduced, the storage space utilization rate is obviously optimized, and the perfect balance of service traceability and storage economy is realized.
Further, the object storage module is an external general storage service independent of the system program module and the database module, and the system program module communicates with the object storage module through a standard application program programming interface provided by the object storage module so as to realize uploading, storage and management of files.
In this embodiment, the object storage module is used as an external general storage service independent of the program module and the database module in the system, and performs communication interaction with the system program module by providing a standardized application programming interface, so as to take charge of uploading, storing and managing functions of all entity file data. The architecture design decouples business logic, metadata management and entity file storage, so that the system can fully utilize the high reliability, high availability and elastic expansion capability of the mature object storage service. The independent deployment of the module not only avoids the complexity and maintenance cost of a self-built file storage system, but also ensures that the whole system architecture is kept highly concise, only needs to pay attention to the realization of core uploading business logic, obtains professional-level storage security guarantee and almost infinite storage expansion potential, and provides a solid storage foundation for the stable operation of the system in various deployment environments.
Further, in the workflow of the simple uploading unit, the binary stream data of the file is directly uploaded to the object storage module, and the global access address of the file is generated and returned by the object storage module, and the storage path field recorded in the file storage table is the unique identifier of the file in the object storage module or the global access address.
In the embodiment, in the workflow of the simple uploading unit, file binary stream data is directly transmitted to an object storage module through a standardized interface, and a global access address of a file is generated and returned by the module, and meanwhile, a storage path field recorded in a file storage table exactly corresponds to the global access address or a unique identifier of the file in an object storage. The mechanism realizes perfect decoupling of physical storage and logical management of the file, ensures the professionality and reliability of the file storage, and establishes strong association between the characteristic value of the file and the actual storage position through the path mapping relation. The design not only simplifies the file addressing flow and improves the access efficiency, but also provides key data support for the system to realize file deduplication and second transmission functions, and simultaneously remarkably reduces the complexity of storage management, so that the system can concentrate on business logic processing without concern about underlying storage details.
Further, when the second transmission logic passes verification, the recorded storage paths associated with the existing files in the file storage table are multiplexed, so that repeated writing operation on the object storage module is avoided.
In the embodiment, the second transmission uploading unit is characterized in that only the database module is operated in the working process, any interaction related to file data transmission between the second transmission uploading unit and the object storage module is completely avoided, and when the second transmission logic is verified to be established through file characteristic value comparison, the system directly multiplexes the storage path information which is already stored in the file storage table and is related to the files with the same characteristic value, so that repeated writing operation to the object storage module is completely avoided. The working mechanism realizes the extreme efficiency of file-level deduplication through accurate metadata management and intelligent path multiplexing, simplifies the uploading process of repeated files from physical transmission to logic recording, greatly improves the response speed of the system to millisecond level, and effectively reduces the writing pressure and network bandwidth consumption of object storage service. Meanwhile, the design remarkably improves the utilization rate of the storage space by avoiding storage redundancy, realizes double optimization of transmission efficiency and resource utilization on the premise of ensuring data consistency, and provides key technical guarantee for stable operation and cost control of the system in a high concurrency scene.
The above embodiments may be implemented in whole or in part by software, hardware (e.g., circuitry), firmware, or any other combination. When implemented in software, the above-described embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in accordance with embodiments of the present invention are produced in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website site, computer, server, or data center to another website site, computer, server, or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains one or more sets of available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium may be a solid state disk.
It should be understood that the term "and/or" is merely an association relationship describing the associated object, and means that three relationships may exist, for example, a and/or B, and may mean that a exists alone, while a and B exist alone, and B exists alone, wherein a and B may be singular or plural. In addition, the character "/" herein generally indicates that the associated object is an "or" relationship, but may also indicate an "and/or" relationship, and may be understood by referring to the context.
In the present invention, "at least one" means one or more, and "a plurality" means two or more. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (a, b, or c) of a, b, c, a-b, a-c, b-c, or a-b-c may be represented, wherein a, b, c may be single or plural.
It should be understood that, in various embodiments of the present invention, the sequence numbers of the foregoing processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
It will be clearly understood by those skilled in the art that, for convenience and brevity of description, specific working procedures of the apparatus, device and unit described above may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.
In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus, device and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another device, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. The storage medium includes a U disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, an optical disk, or other various media capable of storing program codes.
The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. The lightweight general file uploading system capable of realizing second transmission is characterized by comprising a system program module, a database module and an object storage module:
The system program module is used for providing a group of HTTP interfaces to the outside so as to realize file uploading logic and task management logic, and comprises an uploading task generating unit, a simple uploading unit and a second uploading unit;
The database module is used for recording file uploading tasks and file storage information and is composed of two logic data tables, wherein the two logic data tables are a file uploading task table and a file storage table;
The object storage module is used as an independent file content permanent storage medium, wherein stateless interface call is realized through the uploading certificate, a server does not need to rely on a distributed session or a cache component to maintain a task state, simple uploading and second uploading are abstracted into a unified file uploading task model, file characteristic values are utilized to judge weight in the file storage table, a second transmission function of a file is realized, and finally, the whole architecture only depends on two basic components of the database module and the object storage module.
2. The lightweight universal file uploading system capable of realizing second transmission according to claim 1, wherein the uploading task generating unit is specifically configured to:
receiving an uploading initialization request initiated by a client, wherein the request at least comprises a file name;
generating a globally unique file upload task identifier in response to the request;
Summarizing and encrypting at least one of the file uploading task identifier, the file name, the client device identifier and the service source information to form a self-contained uploading certificate;
and returning the uploading certificate to the client for carrying in the subsequent uploading or second uploading operation by the client.
3. The lightweight universal file upload system for enabling seconds transfer of claim 1 wherein said simple upload unit is specifically configured to:
receiving an uploading request initiated by a client, wherein parameters of the request are the uploading certificate and binary stream data of a file;
Analyzing the uploading certificate, decrypting and restoring the file uploading task identifier and other context information in the uploading certificate;
Inquiring in the file uploading task list according to the file uploading task identifier to check whether the task is submitted for the first time;
After verification is passed, creating a task record in the file uploading task table, and marking the state of the task record as initialization;
uploading the received file binary stream data to the object storage module;
after uploading is successful, calculating the characteristic value of the file, recording the characteristic value in the file uploading task record, and inquiring the file storage table according to the characteristic value;
If the record matched with the characteristic value does not exist in the file storage table, a new file record is inserted into the file storage table, and the state of the current file uploading task is updated to be completed.
4. The lightweight universal file upload system for enabling seconds upload of claim 1 wherein said seconds upload unit is specifically configured to:
Receiving a second transmission request initiated by a client, wherein parameters of the request are uploading certificates containing file characteristic values and do not carry binary stream data of a file;
analyzing the uploading certificate, decrypting and restoring the file characteristic value and the file uploading task identifier;
Inquiring in the file uploading task list according to the file uploading task identifier to check whether the task is submitted for the first time;
After verification, inquiring in the file storage table according to the restored file characteristic value;
If the record matched with the file characteristic value is inquired in the file storage table, a new file uploading task record is directly created in the file uploading task table, and the state of the task is directly marked as completed, so that a second transmission effect without file data transmission is realized.
5. The lightweight universal file upload system for enabling seconds transfer of claim 1 wherein said file upload task table is for recording full lifecycle information with tasks as cores, comprising fields for at least recording:
the file uploading task is used for identifying that a task is in a state of initialization, in progress or completed, at least one of file name, file size, file extension and file classification, and at least one of client equipment identification, a client operating system platform and a service source module.
6. The lightweight universal file upload system for enabling seconds transfer of claim 1 wherein said file storage table is used to establish a unique mapping relationship of file characteristic values to physical storage locations comprising fields for at least recording:
The file characteristic value is used as a business main key of the table and has unique constraint, the actual storage path of the file in the object storage module is provided, and the file size is provided, wherein the same file characteristic value of the file storage table uniquely corresponds to a stored actual file.
7. The lightweight universal file upload system for enabling seconds transfer of claim 1 wherein said file upload task table is logically associated with said file store table by said file feature values;
The same file characteristic value at most has one record in the file storage table, and a plurality of file uploading task records can be corresponding in the file uploading task table, so that the mapping relation between the plurality of uploading task records and the single file storage record based on the same file is realized, and the file deduplication and second transmission functions are supported.
8. A lightweight universal file upload system for enabling seconds transfer as claimed in claim 1 wherein said object storage module is an external universal storage service independent of said system program module and said database module;
The system program module communicates with the object storage module through a standard application program interface provided by the object storage module so as to realize uploading, storage and management of files.
9. The lightweight universal file uploading system capable of realizing second transmission according to claim 1, wherein in the workflow of the simple uploading unit, file binary stream data is directly uploaded to the object storage module, and a global access address of the file is generated and returned by the object storage module;
And the storage path field recorded in the file storage table is the unique identifier of the file in the object storage module or the global access address.
10. A lightweight universal file upload system for enabling seconds transfer as claimed in claim 1 wherein said seconds transfer upload unit is operative to operate only said database module without any interaction involving file data transfer with said object storage module;
And multiplexing the recorded storage paths which are associated with the existing files in the file storage table after the second transmission logic passes verification, so as to avoid repeated writing operation to the object storage module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202511851019.1A CN121284025A (en) | 2025-12-10 | 2025-12-10 | Light-weight universal file uploading system capable of realizing second transmission |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202511851019.1A CN121284025A (en) | 2025-12-10 | 2025-12-10 | Light-weight universal file uploading system capable of realizing second transmission |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN121284025A true CN121284025A (en) | 2026-01-06 |
Family
ID=98244593
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202511851019.1A Pending CN121284025A (en) | 2025-12-10 | 2025-12-10 | Light-weight universal file uploading system capable of realizing second transmission |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN121284025A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113204777A (en) * | 2021-05-07 | 2021-08-03 | 郑州信大捷安信息技术股份有限公司 | Method and system for realizing encrypted file second transmission |
| CN114546955A (en) * | 2022-02-17 | 2022-05-27 | 杭州每刻科技有限公司 | Data file uploading processing method and system |
| CN117527783A (en) * | 2023-11-09 | 2024-02-06 | 中移(苏州)软件技术有限公司 | File second transmission verification method and device, communication equipment and storage medium |
-
2025
- 2025-12-10 CN CN202511851019.1A patent/CN121284025A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113204777A (en) * | 2021-05-07 | 2021-08-03 | 郑州信大捷安信息技术股份有限公司 | Method and system for realizing encrypted file second transmission |
| CN114546955A (en) * | 2022-02-17 | 2022-05-27 | 杭州每刻科技有限公司 | Data file uploading processing method and system |
| CN117527783A (en) * | 2023-11-09 | 2024-02-06 | 中移(苏州)软件技术有限公司 | File second transmission verification method and device, communication equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8489548B2 (en) | Method, system, and device for data synchronization | |
| US8069224B2 (en) | Method, equipment and system for resource acquisition | |
| US6466968B2 (en) | Information processing system capable of file transmission and information processing apparatus in the system | |
| CN103581213B (en) | A kind of application process of shared Hosts files, equipment and system | |
| WO2014049804A1 (en) | System operation trace method in distributed system | |
| CN103888547A (en) | Bill processing method and server | |
| CN107566463A (en) | A kind of cloudy storage management system for improving storage availability | |
| CN102800015A (en) | Financial IC (Integrated Circuit) card long-distance individualized method and system | |
| CN104158889A (en) | Business card information querying method and cloud server | |
| CN110263579A (en) | A kind of data processing method, system and relevant device | |
| JP2004310371A (en) | File sharing system and method, file sharing server, file sharing service client terminal, file sharing program, recording medium recording file sharing program | |
| CN114416169B (en) | Data processing method, medium, device and computing equipment based on micro front end | |
| CN108984202A (en) | A method, device and storage medium for sharing electronic resources | |
| CN117112451B (en) | Test method, test device, electronic equipment and computer readable storage medium | |
| JP3848209B2 (en) | Data transfer device, data transfer method and program | |
| CN109040331B (en) | Electronic business card processing method and device, computing equipment and storage medium | |
| CN121284025A (en) | Light-weight universal file uploading system capable of realizing second transmission | |
| CN120492413A (en) | IPFS-based file control method, IPFS-based file control device, IPFS-based file control equipment and medium | |
| CN119561860A (en) | Kafka-based point-of-sale data transmission and storage method and system | |
| CN117235400B (en) | A unified multi-platform portal system based on Kafka technology | |
| JP3983987B2 (en) | Server-side proxy device, data transfer method, and program | |
| JP2007188514A (en) | Server side proxy device, client side proxy device, data transfer method and program | |
| JP3943868B2 (en) | Server-side proxy, data transfer method and program | |
| JP2002373107A (en) | Data transfer device, data transfer method and program | |
| JP2003108462A (en) | Data transfer device and data transfer method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |