CN114721592A - Micro-service storage platform suitable for cold data and use method thereof - Google Patents

Abstract

The present invention provides a microservice storage platform suitable for cold data and a method for using the same. The storage platform includes: a web service layer, configured with a virtual IP, for specifying a data service layer address and a unique data access port for users, and receiving user access based on the virtual IP; a data service layer, configured with at least one The microservice group and at least one storage space, at least one storage space is mapped to the corresponding microservice group by file mapping; each microservice group and the corresponding storage space are owned by a user alone, and the user can specify it when accessing based on the virtual IP. The address of the data service layer and the unique data access port, and then access to its own microservice group and storage space. The user's storage resources are isolated, which reduces the interference of external factors, and can perform hot and cold scheduling of the entire storage resources according to the user's operating habits. When the user's storage data becomes cold, the storage resources can be offline, and the computing resources and storage resources are completely released. .

Description

Microservice storage platform suitable for cold data and how to use it

technical field

The invention relates to the technical field of object storage, in particular to a microservice storage platform suitable for cold data and a method for using the same.

Background technique

Object storage, also known as object-based storage, is a form of data storage whose strategy is to manage and operate data storage as different units, where each stored data unit is called an object. These objects can be any type of data and data of any size, common are pdf, audio, image or video files. In object storage, all objects are stored in a single flat address space, i.e. in a single storage pool repository, without a folder hierarchy and not tied to files in other folders. Unlike file type and block type storage, object storage adds comprehensive metadata to objects. Metadata helps to understand and analyze large files without having to deal directly with the content contained in the files, and is the key to the success of object storage, enabling deep analysis of the usage and functionality of the data in the storage pool.

Object storage has excellent data analysis capabilities, it is driven by metadata, and once each piece of data is classified at the level, the analysis opportunities are much greater. Extremely scalable, data can be added permanently and continuously without limit. Faster data retrieval due to the categorical structure of object storage and the absence of folder hierarchies. Cost reduction, with the ability to store all data at a lower cost due to the scale-out nature of object storage. Resource optimization, since object storage has no archive hierarchy and metadata is fully customizable, it is much less restrictive than file or block storage.

Although there are many types of general object storage system implementations, in general, the user's computing resources and storage resources are confused in the backend, and no effective isolation measures are taken. First of all, because the user's resources are not separated in storage, this makes the data cold operation performed at the user level and data copying, which is inconvenient, fast, and effective. Second, even if user storage resources are separated at the back end, the cooling of data storage resources will continue to consume storage system resources due to the confusion of computing resources such as user metadata, and the storage pool where the cold data is located cannot be completely released. Resources cannot be offline. Therefore, the existing general object storage systems are not suitable for providing user-level object storage services for cold data.

SUMMARY OF THE INVENTION

Aiming at the problem that existing object storage systems cannot provide user-level object storage services for cold data, embodiments of the present invention provide a microservice storage platform suitable for cold data and a method for using the same.

In a first aspect, an embodiment of the present invention provides a microservice storage platform suitable for cold data, including:

The web service layer is configured with a virtual ip, which is used to specify a data service layer address and a unique data access port for the user, and receives the user's access based on the virtual ip;

The data service layer is configured with at least one microservice group and at least one storage space, and the at least one storage space is mapped to the corresponding microservice group by means of file mapping; each microservice group and corresponding storage space are separate for a user All, the user accesses his own microservice group and storage space through the specified data service layer address and unique data access port when accessing based on the virtual IP.

In some embodiments, the web services layer is also used to:

When receiving a scheduling request for converting the data in the storage space to cold data or hot data from the user's application, send a scheduling request to the data service layer, so that the data service layer converts the data in the user's storage space to cold data data or hot data.

In some implementations, the microservice group includes a container group.

In some embodiments, the storage space includes a disk array.

In some embodiments, the web service layer is further configured with a keep-alive group, which is used to implement the virtual IP as a unified entry for user access.

In some embodiments, the same virtual ip can be configured by multiple web service layers, and only one virtual ip in the same network at the same time can be broadcast and used by the address resolution protocol.

In a second aspect, an embodiment of the present invention provides a method for using the microservice storage platform suitable for cold data, including:

Access the web service layer through the virtual ip, and issue a request for a new user;

The web service layer specifies the address of the data service layer and the unique data access port for new users;

The data service layer allocates storage space for new users;

The data service layer starts the microservice group corresponding to the storage space for the newly added user.

In some embodiments, after the data service layer starts the microservice group corresponding to the storage space for the newly added user, the method further includes:

The data service layer updates the relevant information of the new user, including the address of the data service layer and the unique data access port designated for the new user.

In a third aspect, an embodiment of the present invention provides a method for using the microservice storage platform suitable for cold data, including:

Access the user's microservice group through virtual ip and unique data access port;

Operate the user's storage space through the microservice group.

In a fourth aspect, an embodiment of the present invention provides a method for using the microservice storage platform suitable for cold data, including:

The web service layer obtains the user's hot and cold scheduling requests for the data in its storage space;

The web service layer sends the hot and cold scheduling request to the address of the user's data service layer;

The data service layer converts the data in the user's storage space into cold data or hot data as a whole according to the cold and hot scheduling request.

Compared with the prior art, one or more embodiments of the present invention have at least the following beneficial effects:

In the present invention, each user has a unique micro-service group, a unique storage space, and a unique access port. The microservice group maps the storage space into the microservice group through file mapping. Users can access their own microservice group through their own unique access port, thereby accessing their own storage space. The use of microservices to provide users with completely independent storage space enables users to operate independently, isolate storage resources, reduce the interference of external factors, and schedule the entire storage resources hot and cold according to the user's operating habits. User storage data becomes cold, storage resources can be offline, and computing resources and storage resources are completely released.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and therefore do not should be seen as a limitation of scope.

1 is a schematic diagram of a microservice storage platform suitable for cold data provided by an embodiment of the present invention;

2 is a flowchart of a microservice storage platform suitable for cold data provided by an embodiment of the present invention;

3 is a flowchart of a microservice storage platform suitable for cold data provided by an embodiment of the present invention;

FIG. 4 is a flowchart of a microservice storage platform suitable for cold data provided by an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. The components of the embodiments of the invention generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Thus, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present invention.

MAID (ie Massive Arrays of Idle Disks) technology is a large-scale inactive disk array storage. The Storage Networking Industry Association (SNIA) defines MAID as: A storage system comprising an array of disk drives that are powered down individually or in groups when not required. MAID storage systems reduce the power consumed by a storage array.

MAID technology is proposed and developed on the basis of the working mode of tape library. The working principle of the tape library is that all tapes are placed in the slots of the tape library under normal conditions. When a tape is needed, it is placed in the tape drive, and then data is read and written. And most of the time, most tapes are inactive. MAID applies the same principle, turning some disks on only when needed, while other disks are usually powered off, like tapes in a tape library. Therefore, the disk storage of the applied MAID technology can also be called a disk library.

MAID greatly satisfies the green, environmental protection and energy-saving features required by the current popular "green storage". Not all disks in the storage device are in Active and Power ON states all the time, and most hard disks are in Power OFF state. , the hard disk will be Power ON again only when the system needs to read and write access. The boot time of a SATA hard drive in MAID storage is about 10 seconds. After the hard disk in the Power ON state has passed a certain idle time (the length of the time period can be preset according to the access characteristics and frequency of the system), it will be restarted according to the strategy and become the Power OFF state, so as to achieve node, environmental protection and extension of the hard disk. function of service life.

As more and more software projects turn to the microservice architecture due to various requirements such as horizontal expansion of system components, agile development, continuous integration, and decoupling, microservices have greatly changed the development and delivery model of software. Microservice is a software development technology and a variant of service-oriented architecture architecture style. It advocates dividing a single application into a set of small services, and the services coordinate and cooperate with each other to provide users with the ultimate value. . Each service runs in its independent process, and the services communicate with each other using a lightweight communication mechanism. Each service is built around a specific business and can be independently deployed to production environments, production-like environments, etc. In addition, try to avoid a unified, centralized service management mechanism. Containers are a common way to implement microservices. It uses a sandboxing mechanism that allows developers to package their applications and dependencies into a portable container, and distribute them to any popular Linux or Windows machine, which can also be virtualized.

In order to solve the problem that the existing object storage system cannot provide the user-level object storage service for cold data, the present invention provides a form of changing the general object storage system to be directly built on the machine operating system to provide users with a unified use entry. Combined with microservices, user computing resources and storage resources are completely independent, and after the user's storage data is cooled, the disk can be completely offline storage platform.

The specific embodiments of the present invention will be described below through several embodiments.

Example 1

Figure 1 shows a schematic diagram of a microservice storage platform suitable for cold data. As shown in Figure 1, the microservice storage platform suitable for cold data provided by this embodiment includes:

The web service layer is configured with a virtual ip, which is used to specify the address of the data service layer and the unique data access port for the user, and receives the user's access based on the virtual ip;

The data service layer is configured with at least one microservice group and at least one storage space, and at least one storage space is mapped to the corresponding microservice group through file mapping; each microservice group and the corresponding storage space are owned by a single user, Users can access their own microservice groups and storage space through the specified data service layer address and unique data access port when accessing based on the virtual IP.

In the storage platform of this embodiment, effective isolation measures can be implemented for the computing resources and storage resources of the user, data transfer operation can be performed at the user level, and data copying is very convenient, fast and effective. Moreover, due to the isolation of user storage resources, the cooling of data storage resources will not continuously consume storage system resources, and the storage space where cold data is located can be completely released, so that storage resources can be offline, and user-level object storage services for cold data can be realized.

In some embodiments, the microservice group includes a container group, which is used for user storage business, and maps the user's storage space to the container group by means of file mapping, and a group of containers is unique to one user.

In some embodiments, the storage space is implemented as a storage pool, and the storage pool includes disk arrays. The disk array is a disk group in units of users, and each disk group provides storage space for a corresponding user.

In some embodiments, the web service layer is further configured with a keep-alive group for implementing the virtual IP as a unified entry for user access. The same virtual IP can be configured by multiple web service layers, but only one virtual IP in the same network at the same time can be broadcast and used by the Address Resolution Protocol (ARP).

In some embodiments, the web services layer is also used to:

When receiving a scheduling request for converting the data in the storage space into cold data or hot data from the user, a scheduling request is sent to the data service layer, so that the data service layer converts the data in the user's storage space into cold data or hot data. Hot data.

In order to provide a user-level microservice storage platform suitable for cold data, especially to solve the problem of confusion between users' computing resources and storage resources in a common general-purpose object storage platform, the storage platform in this embodiment first builds a storage platform dedicated to each A microservice group (container group) for each user and provides a completely independent storage space, followed by an access path that allocates a globally unique port for each user.

In the storage platform of this embodiment, each user has a unique microservice group, a unique storage space, and a unique access port. The microservice group maps the storage space into the microservice group through file mapping. Users can access their own microservice group through their own unique access port, thereby accessing their own storage space.

Use microservices to build a user's unique container group to provide a completely independent storage space, so that the user's business operation is independent, the storage resources are isolated, the interference of external factors is reduced, and the entire storage resources can be cooled according to the user's operating habits. Hot scheduling. Assigning an access path to a globally unique port for each user can eliminate the confusion and relatively strong centralization of computing resources caused by the traditional provision of a unified portal, allowing users to operate storage resources through unique access ports, with clear logic and easy access. Simple, decentralized, improves reliability, and achieves the purpose of separating computing resources. It is a highly reliable and secure microservice storage platform for cold data, which makes user resources completely independent. When user storage data becomes cold , storage resources can be offline, computing resources and storage resources are completely released.

Embodiment 2

This embodiment provides a method for using the microservice storage platform suitable for cold data in the foregoing embodiment, as shown in FIG. 2 , at least including steps S201 to S204:

Step S201, accessing the web service layer through the virtual ip, and issuing a request for adding a new user;

Step S202, the web service layer specifies the address of the data service layer and the unique data access port for the newly added user;

Step S203, the data service layer allocates storage space for the newly added user;

Step S204, the data service layer starts a microservice group corresponding to the storage space for the newly added user.

In some embodiments, after the data service layer starts the microservice group corresponding to the storage space for the newly added user, it further includes:

Step S205, the data service layer updates the relevant information of the newly added user, including the address of the data service layer and the unique data access port designated for the newly added user.

This embodiment is the process of registering a new user. The user accesses the web service layer through a virtual IP and sends a request for adding a new user. When the new user is requested to be an existing registered user, the request fails and the process ends. When the added user matches an unregistered user, its request is allowed. The web service layer specifies the address of the data service layer and a unique data access port for the new user, and then the data service layer allocates storage space for the new user and starts the storage space. The corresponding microservice group enables newly added users to access their own microservice group and storage space through the specified data service layer address and unique data access port when accessing based on the virtual IP.

Embodiment 3

This embodiment provides a method for using the microservice storage platform suitable for cold data in the foregoing embodiment, as shown in FIG. 3 , including steps S301 to S302:

Step S301, access the user's microservice group through the virtual ip and the unique data access port;

Step S302, operating the storage space of the user through the microservice group.

In this embodiment, when the user is not registered or has other illegal rights, when the user accesses his own microservice group through the virtual IP and his own unique data access port, he cannot operate the user's storage space through the microservice group. When a user is registered, he can access the microservice group of the host through the virtual ip and his own unique data access port, and can operate his own independent storage space through the microservice group, so that the user's business operation is independent, the storage resources are isolated, and the storage space is reduced. It avoids the interference of external factors, and can perform hot and cold scheduling of the entire storage resources according to the user's operating habits.

Embodiment 4

This embodiment provides a method for using the microservice storage platform suitable for cold data in the foregoing embodiment, as shown in FIG. 4 , including steps S401 to S403:

Step S401, the web service layer obtains a user's hot and cold scheduling request for data in its storage space;

Step S402, the web service layer sends the hot and cold scheduling request to the address of the user's data service layer;

Step S403 , the data service layer converts the data in the user's storage space as a whole into cold data or hot data according to the cold and hot scheduling request.

This embodiment is the data hot and cold scheduling process of the user's storage space. The web service layer obtains the user's cold and hot scheduling request for the data in its storage space, and sends the cold and hot scheduling request to the address of the user's data service layer. The data service layer The data in the user's storage space is converted into cold data or hot data as a whole according to the hot and cold scheduling request.

Due to the user's exclusivity of storage space, a user has a group of microservices exclusively, and resources between users are independent of each other. Therefore, when the user's storage resources become cold, the computing and storage resources can be completely released. The spatial data is independently scheduled for hot and cold, which reduces the interference of external factors and quickly adjusts the hot and cold of the entire storage space data.

In the several embodiments provided by the embodiments of the present invention, it should be understood that the disclosed system and method may also be implemented in other manners. The system and method embodiments described above are merely illustrative.

It should be noted that, herein, the terms "first", "second" and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. The terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion such that a process, method, article or device comprising a list of elements includes not only those elements, but also other not expressly listed elements, or also include elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

Although the embodiments disclosed in the present invention are as above, the described contents are only the embodiments adopted to facilitate the understanding of the present invention, and are not intended to limit the present invention. Any person skilled in the art to which the present invention belongs, without departing from the spirit and scope disclosed by the present invention, can make any modifications and changes in the form and details of the implementation, but the scope of patent protection of the present invention, The scope as defined by the appended claims shall still prevail.

Claims (10)

1. A microservice storage platform suitable for cold data, comprising: The web service layer is configured with a virtual ip, which is used to specify a data service layer address and a unique data access port for the user, and receives the user's access based on the virtual ip; The data service layer is configured with at least one microservice group and at least one storage space, and the at least one storage space is mapped to the corresponding microservice group by means of file mapping; each microservice group and corresponding storage space are separate for a user All, the user accesses his own microservice group and storage space through the specified data service layer address and unique data access port when accessing based on the virtual IP. 2. The microservice storage platform suitable for cold data according to claim 1, wherein the web service layer is also used for: When receiving a scheduling request for converting the data in the storage space to cold data or hot data from the user's application, send a scheduling request to the data service layer, so that the data service layer converts the data in the user's storage space to cold data data or hot data. 3. The microservice storage platform suitable for cold data according to claim 1, wherein the microservice group comprises a container group. 4. The microservice storage platform suitable for cold data according to claim 1, wherein the storage space comprises a disk array. 5 . The microservice storage platform suitable for cold data according to claim 1 , wherein the web service layer is further configured with a keep-alive group, which is used to implement virtual ip as a unified entry for user access. 6 . 6. The microservice storage platform suitable for cold data according to claim 1, wherein the same virtual ip can be configured by a plurality of the web service layers, and only one virtual ip in the same network at the same time can be configured. Used by Address Resolution Protocol broadcasts. 7. A method for using a microservice storage platform suitable for cold data according to any one of claims 1 to 6, characterized in that, comprising: Access the web service layer through the virtual ip, and issue a request for a new user; The web service layer specifies the address of the data service layer and the unique data access port for new users; The data service layer allocates storage space for new users; The data service layer starts the microservice group corresponding to the storage space for the newly added user. 8. The method for using a microservice storage platform suitable for cold data according to claim 7, wherein after the data service layer starts a microservice group corresponding to the storage space for a new user, the method further comprises: The data service layer updates the relevant information of the new user, including the address of the data service layer and the unique data access port designated for the new user. 9. A method of using a microservice storage platform suitable for cold data according to any one of claims 1 to 6, characterized in that, comprising: Access the user's microservice group through virtual ip and unique data access port; Operate the user's storage space through the microservice group. 10. A method for using a microservice storage platform suitable for cold data according to any one of claims 1 to 6, characterized in that, comprising: The web service layer obtains the user's hot and cold scheduling requests for the data in its storage space; The web service layer sends the hot and cold scheduling request to the address of the user's data service layer; The data service layer converts the data in the user's storage space into cold data or hot data as a whole according to the cold and hot scheduling request.

Images