CN111708656A - Container image pulling method and system based on lazy loading mechanism - Google Patents

Abstract

The invention relates to a container mirror image pulling method and a system based on a lazy loading mechanism, which belong to the technical field of cloud service and calculation, and aim to overcome the defects that a common container in the existing service-free architecture has slow cold starting speed during operation and causes continuous interference on other disk read-write applications during pulling of a mirror image.

Description

Method and system for pulling container image based on lazy loading mechanism

technical field

The invention relates to a method and system for pulling container images based on a lazy loading mechanism, belonging to the technical field of cloud services and computing.

Background technique

With the rapid development of cloud computing, the serverless architecture is being valued and applied by more and more enterprises. According to the concept of serverless architecture and the current common implementation methods, a single application usually runs in a stateless computing container, such as a container , Containerd, etc. These containers are event-triggered and short-lived (the life cycle may only last for one call), which puts forward higher requirements for the elastic scaling of services and the efficiency of cold start during iteration. At present, the most commonly used container runtime technology in the implementation of the serverless architecture is still the container, but the use of the native container to start and stop the container and the management of the container image resources has the following problems: (1) The cloud service application iteration cycle is short, When starting a new version of the application container, the container needs to pull the full data of the container image to the local. When the amount of data is large, the cold start time of the container will become considerable. (2) When N cloud service nodes start the container of the same image, they need to pull N copies of the image data, resulting in a great waste of bandwidth. (3) Cloud servers usually run in the form of multi-tenancy, so when the container image is downloaded at high speed, it will cause continuous read and write pressure on the disk, which will interfere with other IO processes running at the same time.

SUMMARY OF THE INVENTION

In order to overcome the shortcomings of the prior art, the present invention provides a method and system for pulling a container image based on a lazy loading mechanism, which can effectively solve the above problems by using an extended container image download component and a storage drive component. The core idea is to compress the original The image layered data in the form is transferred to the data storage center shared by multiple nodes, and stored in an uncompressed form. When pulling the image data, only the reference of the data is established instead of transmitting the full amount of data, and the transmission of the data is postponed to the container. Since the data required for container startup is relatively small, the cold start efficiency can be greatly improved while the container execution efficiency is guaranteed, and the disk pressure and bandwidth waste of service nodes can be reduced. At the same time, by extending the original container image name format, the present invention enhances the expressive ability of the original container system, and by using the extended container image format, it can ensure that the compatibility of the original system in the container ecological level is not destroyed, and the user can benefit from it. Provides image loading capability based on lazy loading mechanism. The present invention has the characteristics of high expansibility and high efficiency, so as to improve the execution efficiency and stability of the existing serverless architecture.

The technical scheme adopted in the present invention is:

A method for pulling container images based on a lazy loading mechanism, including the following steps:

The extension container daemon of the target electronic device obtains the image name according to the image download request, parses the image name, and selects whether to download the image in a lazy loading form or in a regular form;

If the image is downloaded in the form of lazy loading, the metadata of the image is obtained from the remote container image center, and a soft link with the image layered data in the shared data storage is established locally according to the metadata; link to download the required image layered data from the shared data storage;

If downloading the image in the regular form or in the lazy loading form fails, download the full image data from the remote container image center.

A container image pulling system based on a lazy loading mechanism, including a shared data storage end, a remote container image center end, and an electronic device serving as a local node;

The shared data storage end is responsible for storing uncompressed image layered data and providing shared data storage services for local nodes;

The remote container image center is responsible for storing image metadata and compressed image data;

The electronic device runs the extended container daemon process, which obtains the image name according to the image download request, parses the image name, and selects whether the image should be downloaded in lazy loading form or in regular form; if the image is downloaded in lazy loading form, it will be downloaded from the remote container The image center obtains the metadata of the image, and establishes a local soft link with the image layered data in the shared data storage according to the metadata; when the container starts, it downloads the required image files from the shared data storage through the soft link. Layer data; if downloading the image in the regular form or in the lazy loading form fails, download the full image data from the remote container image center.

Further, the extended container daemon includes an extended download component, which is responsible for obtaining the metadata of the image from the remote container image center according to the image download request, and establishes and shares data storage according to the image layered data ID in the metadata. The soft link of the image layered data in the terminal; when the container is started, the required mirror layered data is downloaded from the shared data storage terminal through the soft link.

Further, according to metadata, the soft link with the mirror layered data in the shared data storage terminal is established locally, which means that the metadata includes the unique DiffID of mirroring all layered data, and the DiffID of each layered data is used to calculate the layered data. The ChainID value is used to search for the corresponding mirror layered data on the shared data storage side by using the ChainID value, and a soft link with the mirror layered data is established locally.

Further, while locally establishing a soft link with the image layered data, the obtained image metadata parameters are set, and the parameters include layer size and CacheID information, so as to ensure the normal operation of the original image storage service of the container.

Further, the extended container daemon calls the default image download component of the container to download the full amount of image data from the remote container image center.

Further, the extended container daemon process includes an extended storage driver component, the storage driver component is called before the container is started, and is responsible for obtaining the layered data required by the image from the local image layered storage according to the image ID used by the container. , and use the layered data to build the root file system of the container in an overlay manner; and is responsible for monitoring the read and write modes of the process in the container to the layered data of the image during the container running phase, obtain the hotspot data of the container through statistical analysis, and call The extension download component downloads the hotspot data to local storage for use when the container runs after the second startup.

The advantages of the present invention compared with the prior art are:

Aiming at the problems that the existing serverless platform uses the native container as the runtime, the cold start time of the container is long, the interference time to other applications is long, and the bandwidth utilization rate is low, the invention stores the container image data in the storage center shared by multiple nodes, and improves the The image download implementation only establishes the reference of the corresponding layered data, thereby delaying the actual data transmission until the container process execution period, which not only improves the operational stability of the serverless architecture platform, but also further improves the cold start efficiency of the container.

Description of drawings

In order to more clearly illustrate the working mode of the embodiments of the present invention and the technical solutions of the prior art, a brief description of the accompanying drawings of the prior art of the present invention will be made below:

1 is a schematic structural diagram of a container image pulling system based on a lazy loading mechanism according to an embodiment;

2 is a schematic flowchart of an extension download component of the container image pulling system based on the lazy loading mechanism of the embodiment;

FIG. 3 is a schematic diagram of an extended storage driver process of a container image pulling system based on a lazy loading mechanism according to an embodiment.

Detailed ways

The technical solutions of the present invention will be further described below with reference to the accompanying drawings of the present invention. The described embodiments are part of the embodiments of the present invention, but not all of the embodiments. For those skilled in the art, some well-known technologies may not be described in detail.

A container image pulling system based on a lazy loading mechanism, including a shared data storage end, a remote container image center end, and an electronic device, the electronic device can be summarized as running a client server and extending a container daemon;

Customer service is an interactive service in the foreground. The main body of the service can be a specific person or an application. When a customer has a demand for a certain service resource, the customer server sends a request containing business logic to the container daemon;

The container daemon is the core part. The extended container daemon includes the optimized image data download component and the storage driver component. The interaction between the daemon and the client server is carried out in the form of remote procedure calls. The extended container daemon can accept requests from the client server to pull image data from the shared data storage, and the request contains the name of the image to be downloaded. , By modifying the original container daemon process, the system supports the extended container image format. If the image is submitted in an extended form, the container will be pulled in the form of lazy loading first, and the container daemon process will first try to establish with the remote container image center. connect, and get metadata related to that image. The optimized image data download component will use metadata to download image data from the shared data storage. If the image is submitted in the default form or the download fails in lazy loading mode, the download component will fall back to the regular download process, that is, from the container image center. download from the terminal to ensure the robustness of the service. Image layered data downloaded using the improved method contains only references to the original layered data. The actual data transfer is deferred until the application startup time inside the container. When the container daemon receives the command to start the container, it will call the extended storage driver component to establish the root file system of the container, and the storage driver component uses the downloaded image data to establish the root directory of the container. The data required by the container during the running process will be loaded into the local node in an on-demand manner, which improves the startup speed of the container. The storage driver component monitors the execution of applications in the container, collects and analyzes container hotspot data, and calls the download component to load the corresponding hotspot data to the local node in advance.

The shared data storage side is responsible for storing uncompressed image layered data and providing shared data storage services for multiple nodes. The extension download component needs to query and pull the image layered data from the shared data storage side.

The remote container image center is responsible for storing image metadata, as well as some compressed image data that does not exist on the shared data storage end.

The extension container daemon process includes an extension download component, and the extension download component is described as follows:

The extended download component accepts the image layered data download request from the container daemon, and communicates with the remote container image center to obtain the image metadata. The image metadata includes the unique DiffID (image layer calibration) of all layered data of an image. Check ID), using the DiffID of each layered data, the layered ChainID value (that is, the index ID adopted by the docker content addressing mechanism) can be calculated, and using this value, the extended download component searches the shared data storage for the corresponding Layer data, and establish a soft link of layered data locally, and set relevant metadata parameters of local layered data, including layer size, CacheID (that is, the universal unique identification code generated by the host) and other information to ensure that the container The normal operation of the original image storage service. The extended download component does not download the full amount of data locally when downloading the image layered data, but makes a reference to the layered data on the local node, and the actual data will not be transmitted until the container is started. After receiving the request to download the image from the shared data storage, the extension download component will look for the required hierarchical data in the data storage, and make a soft link to store it locally.

When the extension download component cannot download the image data from the shared data storage, it will fall back to the container's default image download process, that is, call the container's default image download component to download the image data from the remote container image center. The result of the rollback execution will be returned to the container daemon and the client server.

The extended container daemon includes an extended storage driver component, and the storage driver component is described as follows:

The extended storage driver is called before the container starts. It obtains the layered data required by the image from the local image layered storage according to the image ID used by the container. The image ID here is different from the layer-related ID. Image ID is a unique identifier for a container image, which uniquely refers to an image, and an image contains multiple layers. The extended storage driver then uses the hierarchical data to build the container's root file system in an overlay manner. Since the image data is stored locally in the form of a soft link, the data required during the running of the container will be read from the shared data storage end to the local on an as-needed basis.

The extended storage driver will monitor the read/write mode of each layered data by the process in the container during the container running phase, obtain the hotspot data of the container through statistical analysis, and call the extension download component to pre-download these hotspot data or hotspot layered data to the local Storage, the operation of the container after the second startup will use some of the pre-loaded data to improve the efficiency of program operation.

In this embodiment, a system for pulling container images based on a lazy loading mechanism is shown in FIG. 1 , and the steps are as follows:

Step 101: The client server, which can be a user or an upper-layer application that initiates a request to download an image to the container daemon, and needs to be downloaded from the shared data storage end; the client and server, which can be a user or an upper-layer application that initiates the container daemon to start the container request, and the image used is downloaded from the shared data store.

Step 102: the container daemon uniformly encapsulates the interface provided by the extended download component and the extended storage driver, and provides a service interface similar to other default components for the client server;

Step 103: the extension download component communicates with the remote container image center, and uses the image name and version information and other parameters to obtain the metadata information of the image, and the metadata information includes the layered information of the image;

Step 104: the extended download component uses the image layered information to query the layered data to the shared data storage end, and if the corresponding layered data is found, a soft link of the relevant data is established and stored locally;

Step 105: The extended storage driver monitors the read and write of container data during the running of the container, and calls the extended download component to download hot data.

In this embodiment, a process of extending and downloading components of a container image pulling system based on a lazy loading mechanism is shown in Figure 2, and the specific implementation steps are as follows:

Step 201: the client submits a request for downloading image data, and the container daemon parses the image name to determine whether to download from the shared storage preferentially;

Step 202: If it is necessary to download from the shared storage preferentially, the container daemon will call the extended download component. The extended download component first tries to obtain the image metadata from the remote container image center, and obtains the image score according to the package file verification of the image layer. layerDiffID;

Step 203: Calculate the ChainID of each layer by using the obtained DiffID of the mirror layer, that is, to calculate the DiffID of the current layer and all ancestor layers, and the calculation method is:

ChainID(layerN)=SHA256hex(ChainID(layerN-1)+""+DiffID(layerN));

Step 204: Use the calculated ChainID information to search for the hierarchical data of the same ChainID at the shared data storage end;

Step 205: If the data exists, set a local soft link, and set meta-information such as hierarchical data size, CacheID, etc. CacheID is a Universal Unique Identifier (UUID) generated by the host, which corresponds to the image layer file one-to-one, and is used for Host and index image layer files;

Step 206: If the image name indicates that the default download method is directly adopted or any error occurs during the download process of the extended download component, it will fall back to the default download process, and the default download component will download the full amount of image data from the remote container image center;

Step 207: Return the download result or error information of the download component to the container daemon and then to the client server.

In this embodiment, the process of extending the storage driver of the container image pulling system based on the lazy loading mechanism is shown in FIG. 3 :

Step 301: the extended storage driver component obtains a request from the container daemon, that is, the client server to start the container, and the storage driver component builds a root directory system for the container according to the request;

Step 302: After the container is successfully started, the start of the container monitoring unit will be triggered;

Step 303: The container monitoring unit collects data read information during container operation, including file name, file size, etc.;

Step 304: The container monitoring unit transmits the container operation information to the data analysis unit;

Step 305: the data analysis unit returns the analyzed and processed data to the container monitoring unit, which includes the hotspot data read by the container;

Step 306: the container monitoring unit further returns the hotspot data information to the extended storage drive component;

Step 307: the component returns the system call result of building the container root directory;

Step 308: The extended storage driving component invokes the extended download component to download the corresponding hotspot data according to the returned hotspot data.

The present invention uses some commonly used container images Nginx, Redis, Postgres and Couchbase to compare the traditional deployment system and the system of the present invention, and record the startup time of the two containers. The test results are shown in Table 1 below.

Table 1 Image deployment time

Nginx Redis Postgres Couchbase traditional deployment system 4.622s 4.178s 11.121s 32.754s this system 0.516s 0.757s 0.829s 0.856s

As can be seen from Table 1, compared with the traditional deployment method, the system has an improvement of more than 80% in the average deployment time.

The above descriptions of the specific implementation methods of the embodiments of the present invention are only examples, and the protection scope of the present invention is described by the claims. On the basis of understanding the above description by those skilled in the art, any changes and modifications made in any form fall within the protection scope of the present invention.

Claims (10)

1. A method for pulling a container image based on a lazy loading mechanism, characterized in that, comprising the following steps: The extension container daemon of the target electronic device obtains the image name according to the image download request, parses the image name, and selects whether to download the image in a lazy loading form or in a regular form; If the image is downloaded in the form of lazy loading, the metadata of the image is obtained from the remote container image center, and a soft link with the image layered data in the shared data storage is established locally according to the metadata; link to download the required image layered data from the shared data storage; If downloading the image in the regular form or in the lazy loading form fails, download the full image data from the remote container image center. 2. The method according to claim 1, wherein the extended container daemon process comprises an extended download component, and the extended download component is responsible for obtaining the metadata of the image from the remote container image center according to the image download request, and according to the metadata The image layered data ID in establishes a soft link with the image layered data in the shared data storage end; when the container is started, the required image layered data is downloaded from the shared data storage end through the soft link. 3. The method according to claim 1 or 2, characterized in that, according to the metadata, a soft link with the mirrored layered data in the shared data storage terminal is established locally, which means that the metadata includes a mirror image of all layered data. Unique DiffID, using the DiffID of each layered data to calculate the layered ChainID value, using the ChainID value to search for the corresponding mirror layered data in the shared data storage, and establish a local soft link with the mirror layered data. 4. The method according to claim 1, wherein extending the container daemon process comprises an extended storage driver component, the storage driver component is called before the container is started, and is responsible for extracting files from the local image according to the image ID used by the container. Obtain the layered data required by the image in the layer storage, and use the layered data to build the root file system of the container in an overlay mode; and is responsible for monitoring the read and write modes of the process in the container to the layered data of the image during the container running phase, After statistical analysis, the hotspot data of the container is obtained, and the extension download component is called to download the hotspot data to the local storage for use when the container runs after the second startup. 5. A container image pulling system based on a lazy loading mechanism, characterized in that it comprises a shared data storage end, a remote container image central end, and an electronic device serving as a local node; wherein, The shared data storage end is responsible for storing uncompressed image layered data and providing shared data storage services for local nodes; The remote container image center is responsible for storing image metadata and compressed image data; The electronic device runs the extended container daemon process, which obtains the image name according to the image download request, parses the image name, and selects whether the image should be downloaded in lazy loading form or in regular form; if the image is downloaded in lazy loading form, it will be downloaded from the remote container The image center obtains the metadata of the image, and establishes a local soft link with the image layered data in the shared data storage according to the metadata; when the container starts, it downloads the required image files from the shared data storage through the soft link. Layer data; if downloading the image in the regular form or in the lazy loading form fails, download the full image data from the remote container image center. 6. The system according to claim 5, wherein the extended container daemon process comprises an extended download component, and the extended download component is responsible for obtaining the metadata of the image from the remote container image center according to the image download request, and according to the metadata The image layered data ID in establishes a soft link with the image layered data in the shared data storage end; when the container is started, the required image layered data is downloaded from the shared data storage end through the soft link. 7. The system according to claim 5 or 6, characterized in that, establishing a soft link with the mirrored layered data in the shared data storage terminal locally according to the metadata means that the metadata includes a mirror image of all layered data. Unique DiffID, using the DiffID of each layered data to calculate the layered ChainID value, using the ChainID value to search for the corresponding mirror layered data in the shared data storage, and establish a local soft link with the mirror layered data. 8. system as claimed in claim 7, is characterized in that, while establishing the soft link with this mirror layered data locally, set the mirror metadata parameter that obtains, parameter comprises layer size, CacheID information, to ensure The normal operation of the original image storage service of the container. 9 . The system of claim 5 , wherein the extended container daemon process calls a default image download component of the container to download the full amount of image data from the remote container image center. 10 . 10. The system according to claim 5, wherein the extended container daemon process comprises an extended storage driver component, the storage driver component is called before the container is started, and is responsible for extracting files from the local image according to the image ID used by the container. Obtain the layered data required by the image in the layer storage, and use the layered data to build the root file system of the container in an overlay mode; and is responsible for monitoring the read and write modes of the process in the container to the layered data of the image during the container running phase, After statistical analysis, the hotspot data of the container is obtained, and the extension download component is called to download the hotspot data to the local storage for use when the container runs after the second startup.

Images