Disclosure of Invention
The disclosure provides a self-adaptive management method, device and equipment for PCDN resources and a storage medium, which solve the technical problem of how to accurately identify hot spot resources and perform reasonable cache on the basis of the finite storage of PCDN nodes.
According to a first aspect of the present disclosure, an adaptive management method for PCDN resources is provided.
The method comprises the following steps:
Analyzing CDN bandwidth usage data by taking a CDN bandwidth occupation value of a target resource as a heat sequencing basis, selecting N resources with highest bandwidth occupation as hot spot resources, and generating a resource downloading task containing a hot spot resource caching instruction, wherein N is a positive integer;
And responding to the self storage resource identification list and the current storage space occupancy rate reported by each PCDN node, determining the storage occupancy rate of each node and the global residual storage space of the PCDN network, and determining whether to issue a resource downloading task and whether to generate and issue a resource deleting task based on the storage occupancy rate of each node and the global residual storage space of the PCDN network.
The aspect and any possible implementation manner as described above further provide an implementation manner, where determining whether to issue a resource downloading task and whether to generate and issue a resource deleting task based on a storage occupancy of each node and a global remaining storage space of a PCDN network includes:
When the storage occupancy rate of the target PCDN node is detected not to exceed a preset threshold value, directly issuing the resource downloading task to the node;
When the storage occupancy rate of the target PCDN node is detected to exceed a preset threshold, further judging the global residual storage space of the PCDN network, and determining whether to issue a resource downloading task and whether to generate and issue a resource deleting task.
The aspect and any possible implementation manner as described above further provide an implementation manner, where further determining whether to issue a resource download task and whether to generate and issue a resource delete task further includes:
if the global residual storage space of the PCDN network is not smaller than a preset threshold value, temporarily not issuing the resource downloading task to the target PCDN node;
if the global residual storage space of the PCDN network is smaller than a preset threshold, screening at least one target resource with the lowest heat according to the real-time heat sequencing result of cached resources in the network, determining a PCDN node set cached with the target resource, and synchronously issuing a resource deleting task and a resource downloading task aiming at the target resource to nodes in the node set.
In the foregoing aspect and any possible implementation manner, there is further provided an implementation manner, where the resource deletion task and the resource downloading task are associated execution instruction sets, and after the target node receives the instruction sets, the target node removes the low-heat resource release storage space specified by the resource deletion task, and downloads new resources specified by the resource downloading task.
Aspects and any one of the possible implementations as described above, further providing an implementation, the method further including:
If any PCDN node does not report the self-stored resource identification list and the current storage space occupancy rate beyond the preset time, judging that the node is invalid, and re-adding the cached resource of the node into a resource downloading task queue to be distributed.
Aspects and any one of the possible implementations as described above, further providing an implementation, the method further including:
Setting a resource weight coefficient based on CDN bandwidth saving amount corresponding to the service type, and preferentially issuing a resource downloading task corresponding to the service with a high weight coefficient when the resource downloading task is issued, wherein the weight coefficient corresponding to the service type with the larger bandwidth saving amount is higher.
According to a second aspect of the present disclosure, an adaptive management apparatus for PCDN resources is provided.
The device comprises:
The task generating module is used for analyzing CDN bandwidth usage data by taking a CDN bandwidth occupation value of a target resource as a heat sequencing basis, selecting N resources with highest bandwidth occupation as hot spot resources, and generating a resource downloading task containing a hot spot resource caching instruction, wherein N is a positive integer;
The task distribution module is used for responding to the self storage resource identification list and the current storage space occupancy rate reported by each PCDN node, determining the storage occupancy rate of each node and the global residual storage space of the PCDN network, and determining whether to issue a resource downloading task and whether to generate and issue a resource deleting task based on the storage occupancy rate of each node and the global residual storage space of the PCDN network.
According to a third aspect of the present disclosure, an electronic device is provided. The electronic device comprises a memory and a processor, the memory having stored thereon a computer program, the processor implementing the method as described above when executing the program.
According to a fourth aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon a computer program which when executed by a processor implements a method according to the first and/or second aspects of the present disclosure.
In the method, CDN bandwidth occupation values of target resources are used as heat sequencing basis, CDN bandwidth utilization data are analyzed, N resources with highest bandwidth occupation are selected as hot spot resources, downloading tasks are generated, the hot spot resources can be cached to appropriate nodes in time, access delay of users is reduced, user experience is improved, whether to issue resource downloading tasks and whether to generate and issue resource deleting tasks are flexibly determined according to storage occupation rates of all nodes and global residual storage space of a PCDN network, waste of storage resources is avoided, and effective caching of the hot spot resources is ensured.
It should be understood that what is described in this summary is not intended to limit the critical or essential features of the embodiments of the disclosure nor to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following description.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present disclosure more apparent, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure, and it is apparent that the described embodiments are some embodiments of the present disclosure, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments in this disclosure without inventive faculty, are intended to be within the scope of this disclosure.
In addition, the term "and/or" is merely an association relation describing the association object, and means that three kinds of relations may exist, for example, a and/or B, and that three kinds of cases where a exists alone, while a and B exist alone, exist alone. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship.
In the method, CDN bandwidth occupation values of target resources are used as heat sequencing basis, CDN bandwidth utilization data are analyzed, N resources with highest bandwidth occupation are selected as hot spot resources, downloading tasks are generated, the hot spot resources can be cached to appropriate nodes in time, access delay of users is reduced, user experience is improved, whether to issue resource downloading tasks and whether to generate and issue resource deleting tasks are flexibly determined according to storage occupation rates of all nodes and global residual storage space of a PCDN network, waste of storage resources is avoided, and effective caching of the hot spot resources is ensured.
Fig. 1 shows a flowchart of a method for adaptively managing PCDN resources according to an embodiment of the disclosure, and as shown in fig. 1, a method 100 for adaptively managing PCDN resources may include the following steps:
S110, analyzing CDN bandwidth usage data by taking a CDN bandwidth occupation value of a target resource as a heat sequencing basis, selecting N resources with highest bandwidth occupation as hot spot resources, and generating a resource downloading task containing the hot spot resource caching instruction, wherein N is a positive integer.
In some embodiments, the method further comprises:
If any PCDN node does not report the self-stored resource identification list and the current storage space occupancy rate beyond the preset time, judging that the node is invalid, and re-adding the cached resource of the node into a resource downloading task queue to be distributed.
In some embodiments, the method further comprises:
Setting a resource weight coefficient based on CDN bandwidth saving amount corresponding to the service type, and preferentially issuing a resource downloading task corresponding to the service with a high weight coefficient when the resource downloading task is issued, wherein the weight coefficient corresponding to the service type with the larger bandwidth saving amount is higher.
Specifically, in terms of identification of hot spot resources, the system will periodically analyze CDN bandwidth usage data. The method comprises the steps of performing offline aggregation processing on logs, performing accumulated calculation on bandwidths corresponding to each URL, sequencing the bandwidths from high to low according to the total bandwidth, and selecting N resources before ranking as hot spot resources, wherein N is a positive integer and can be set according to actual requirements and system capacity. For each URL determined to be a hotspot, the system generates a cache instruction, where the cache instruction at least includes key information such as resource URL, expected copy number, priority, expiration time, and verification mode. These instructions are then packaged into resource download tasks and pushed into the task queue awaiting subsequent dispatch processing.
Specifically, during the distribution of the resource download task, the system may consider the weighting coefficient of the service type. The service weight coefficient is preset based on CDN bandwidth saving amount corresponding to the service type, and is stored in a service weight coefficient mapping table in a database. For example, the short video traffic weight may be 1.5, live time shift weight 2.0, package download weight 1.0, etc. The calculation method of the weight coefficient is to count the total bandwidth saved by PCDN of a certain service in the past 30 days compared with pure CDN, divide the total request bandwidth of the service to obtain the saving rate, and then linearly map the saving rate to the weight coefficient range between 1.0 and 3.0. When the scheduler pulls the tasks from the task queue, the tasks are ordered according to the descending order of the weight coefficients, and the resource downloading tasks corresponding to the service with high weight coefficients are preferentially issued. If the tasks with the same weight are met, the tasks are arranged in descending order of the original heat degree so as to ensure the rationality and the high efficiency of the resource allocation.
Specifically, the system is also provided with a node failure judging mechanism. Each PCDN node needs to send JSON data containing a self-stored resource identification list and the current disk occupancy rate to a resource management system through a heartbeat interface according to a set time interval of M minutes. The heartbeat interface returns an HTTP 200 status code to indicate that the report was successful. The resource management system maintains a hash table in memory where the key is the node id and the value is the timestamp of the last heartbeat. The background of the system has a timing task, scans the hash table every M minutes, and checks whether the difference between the current time and the latest heartbeat time stamp exceeds a preset duration. If the preset time period is exceeded, the node is judged to be invalid. Once a node is determined to be dead, the system triggers a resource redistribution process by first querying the metadata base for a list of all resource identifiers cached by the dead node, and then calculating a new copy demand number for each resource. For resources with the demand number greater than 0, the resources are repackaged into resource downloading tasks and put back into a queue to be distributed, and meanwhile, the priority of the tasks is set to be high so as to complement copies as soon as possible, and normal operation of the system is ensured.
S120, responding to the self storage resource identification list and the current storage space occupancy rate reported by each PCDN node, determining the storage occupancy rate of each node and the global residual storage space of the PCDN network, and determining whether to issue a resource downloading task and whether to generate and issue a resource deleting task based on the storage occupancy rate of each node and the global residual storage space of the PCDN network.
In some embodiments, the determining whether to issue the resource download task and whether to generate and issue the resource delete task based on the storage occupancy of each node and the global remaining storage space of the PCDN network includes:
When the storage occupancy rate of the target PCDN node is detected not to exceed a preset threshold value, directly issuing the resource downloading task to the node;
When the storage occupancy rate of the target PCDN node is detected to exceed a preset threshold, further judging the global residual storage space of the PCDN network, and determining whether to issue a resource downloading task and whether to generate and issue a resource deleting task.
In some embodiments, the further determining whether to issue a resource download task and whether to generate and issue a resource delete task includes:
if the global residual storage space of the PCDN network is not smaller than a preset threshold value, temporarily not issuing the resource downloading task to the target PCDN node;
if the global residual storage space of the PCDN network is smaller than a preset threshold, screening at least one target resource with the lowest heat according to the real-time heat sequencing result of cached resources in the network, determining a PCDN node set cached with the target resource, and synchronously issuing a resource deleting task and a resource downloading task aiming at the target resource to nodes in the node set.
In some embodiments, the resource deletion task and the resource downloading task are associated execution instruction sets, and after the target node receives the instruction sets, the target node removes the low-heat resource release storage space specified by the resource deletion task, and downloads new resources specified by the resource downloading task.
Specifically, the system firstly receives a self-storage resource identification list and a current storage space occupancy rate which are periodically reported by each PCDN node through a heartbeat interface, and stores the data in a memory hash table of the resource management system, wherein a key of the hash table is a node ID, and the value comprises reporting time and storage information. The system will calculate the storage occupancy of each node and the global remaining storage space of the entire PCDN network from these data.
When a resource downloading task needs to be issued to a certain target PCDN node, the system firstly checks whether the storage occupancy rate of the node exceeds a preset threshold value. If the total storage space of the PCDN network exceeds the preset threshold, the system further judges whether the total residual storage space of the PCDN network is smaller than the preset threshold. If the global residual storage space is not smaller than the preset threshold value, the system temporarily does not issue a resource downloading task to the target node, marks the task as a pending state, and re-evaluates the task after waiting for the release of the storage space of the subsequent node.
If the global residual storage space is smaller than a preset threshold, the system screens out at least one target resource with the lowest heat according to a real-time heat sequencing result of cached resources in the network, wherein the heat sequencing is performed based on CDN bandwidth occupation values, and the lower the bandwidth occupation is, the lower the heat of the resources is. The system queries the metadata base, determines a PCDN node set which caches the low-heat target resources, and then generates a resource deletion task and a resource downloading task for each target resource to form an associated execution instruction set. In the instruction set, the resource deleting task designates the resource URL to be deleted and the deleting priority, and the resource downloading task designates the new resource URL to be downloaded, the expected copy number, the priority, the expiration time, the verification mode and the like. The system synchronously issues this instruction set to each node in the set of target nodes.
Specifically, after receiving the instruction set, the target node executes a resource deletion task, removes the designated low-heat resource to release the storage space, updates the local storage resource identification list, and reports a new storage occupancy rate to the resource management system. After the deletion of the resources and the release of the storage space are completed, the node starts to execute the resource downloading task, downloads new resources from a designated source and caches the new resources according to the requirements in the task instructions. After the downloading is completed, the node updates the local storage resource identification list again, and reports the new storage occupancy rate to the resource management system.
After completing the task deletion and downloading, the node sends feedback of the task execution result to the resource management system, including success or failure state and error information (if any). And the resource management system updates node state information in the hash table according to the result fed back by the node, and recalculates the global residual storage space. If a node task is found to fail to execute, the system can correspondingly adjust according to the failure reason, for example, re-issue the task or adjust the task priority.
Through the flow, the accurate identification and reasonable caching of the hot spot resources can be realized, meanwhile, the resource allocation strategy can be dynamically adjusted according to the node state and the global storage condition, the overall performance and the resource utilization efficiency of the system are improved, and the stable operation and the efficient service of the PCDN network are ensured.
It should be noted that, for simplicity of description, the foregoing method embodiments are all described as a series of acts, but it should be understood by those skilled in the art that the present disclosure is not limited by the order of acts described, as some steps may be performed in other orders or concurrently in accordance with the present disclosure. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all alternative embodiments, and that the acts and modules referred to are not necessarily required by the present disclosure.
The foregoing is a description of embodiments of the method, and the following further describes embodiments of the present disclosure through examples of apparatus.
Fig. 2 is a block diagram of an adaptive management apparatus for PCDN resources according to an embodiment of the present disclosure, and as shown in fig. 2, an adaptive management apparatus 200 for PCDN resources may include:
the task generating module 210 is configured to analyze CDN bandwidth usage data by using a CDN bandwidth occupation value of a target resource as a hotness ranking basis, select N resources with highest bandwidth occupation as hotspot resources, and generate a resource downloading task including the hotspot resource caching instruction, where N is a positive integer;
The task distribution module 220 is configured to determine, in response to the self storage resource identifier list and the current storage space occupancy reported by each PCDN node, a storage occupancy of each node and a global remaining storage space of the PCDN network, and determine whether to issue a resource downloading task and whether to generate and issue a resource deleting task based on the storage occupancy of each node and the global remaining storage space of the PCDN network.
It should be noted that, for simplicity of description, the foregoing method embodiments are all described as a series of acts, but it should be understood by those skilled in the art that the present disclosure is not limited by the order of acts described, as some steps may be performed in other orders or concurrently in accordance with the present disclosure.
Further, those skilled in the art will also appreciate that the embodiments described in the specification are all alternative embodiments, and that the acts and modules referred to are not necessarily required by the present disclosure.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the described modules may refer to corresponding procedures in the foregoing method embodiments, which are not described herein again.
In the technical scheme of the disclosure, the acquisition, storage, application and the like of the related user personal information all conform to the regulations of related laws and regulations, and the public sequence is not violated.
According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
Fig. 3 illustrates a block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure. Electronic device 300 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic device 300 may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 3, the electronic device 300 may include a computing unit 301 that may perform various suitable actions and processes in accordance with a computer program stored in a Read Only Memory (ROM) 302 or a computer program loaded from a storage unit 308 into a Random Access Memory (RAM) 303. In the RAM303, various programs and data required for the operation of the electronic device 300 may also be stored. The computing unit 301, the ROM302, and the RAM303 are connected to each other by a bus 304. An input/output (I/O) interface 305 is also connected to bus 304.
Various components in the electronic device 300 are connected to the I/O interface 305, including an input unit 306, such as a keyboard, mouse, etc., an output unit 307, such as various types of displays, speakers, etc., a storage unit 308, such as a magnetic disk, optical disk, etc., and a communication unit 309, such as a network card, modem, wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunication networks.
The computing unit 301 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 301 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as method 100. For example, in some embodiments, the method 100 may be implemented as a computer program product, including a computer program, tangibly embodied on a computer-readable medium, such as the storage unit 308. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 300 via the ROM302 and/or the communication unit 309. One or more of the steps of the method 100 described above may be performed when the computer program is loaded into RAM303 and executed by the computing unit 301. Alternatively, in other embodiments, the computing unit 301 may be configured to perform the method 100 by any other suitable means (e.g. by means of firmware).
The various embodiments described above herein may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems-on-a-chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be a special or general purpose programmable processor, operable to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a computer-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a computer-readable storage medium would include one or more wire-based electrical connections, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
It should be noted that the present disclosure further provides a non-transitory computer readable storage medium storing computer instructions, where the computer instructions are configured to cause a computer to perform the method 100 and achieve corresponding technical effects achieved by performing the method according to the embodiments of the present disclosure, which are not described herein for brevity.
In addition, the present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the method 100.
To provide for interaction with a user, the embodiments described above may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user, for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The above-described embodiments may be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a Local Area Network (LAN), a Wide Area Network (WAN), and the Internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel, sequentially, or in a different order, provided that the desired results of the disclosed aspects are achieved, and are not limited herein.
The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.