CN113760638A - A log service method and device based on kubernetes cluster - Google Patents

Abstract

The invention discloses a log service method and device based on kubernetes cluster, and relates to the technical field of computers. A specific implementation of the method includes: acquiring the running status of the application running on the kubernetes cluster; wherein, the running node corresponding to the application locally stores the application log generated by the application during the running process to obtain a local application log set; The state determines the completed application, and stores the locally stored application log corresponding to the completed application in the cloud to obtain the cloud application log collection; in response to the application log query request initiated by the user, according to the application number included in the application log query request , query the local application log collection and the cloud application log collection in turn to determine the target application log, and send the target application log to the user. This embodiment saves storage resources, simplifies the log query process, improves log query efficiency, and improves user experience.

Description

A log service method and device based on kubernetes cluster

technical field

The present invention relates to the field of computer technology, in particular to a method and device for log service based on a kubernetes cluster.

Background technique

Kubernetes is an open source application for managing containerized applications on multiple hosts in a cloud platform. As a resource scheduling framework, Kubernetes clusters are often used to schedule online services. The goal is to make the deployment of containerized applications simple and efficient. (powerful), Kubernetes provides a mechanism for application deployment, planning, update, and maintenance. Traditional distributed log solutions such as ElK (Elasticsearch, Logstash, and Kibana, which are log collection and management systems built by three open source frameworks) are usually used in current Kubernetes clusters.

Spark, a popular big data computing framework, is better suited for algorithms that require iteration such as data mining and machine learning. The existing Spark usually uses the traditional Yarn scheduler for resource scheduling.

In the process of realizing the present invention, the inventor found that there are at least the following problems in the prior art:

1. The existing log service methods for running applications in kubernetes clusters mainly use traditional distributed log solutions, which consume storage resources and have low log query efficiency;

2. When Spark applications run in a kubernetes cluster, logs need to be queried and displayed through specific pages, search, query, display and other components. The log query process is complicated, resulting in poor user experience.

SUMMARY OF THE INVENTION

In view of this, embodiments of the present invention provide a log service method and device based on a kubernetes cluster, which can save storage resources, simplify the log query process, improve log query efficiency, and improve user experience.

In order to achieve the above purpose, according to the first aspect of the embodiments of the present invention, a log service method based on a kubernetes cluster is provided, including:

Obtain the running status of the application running on the kubernetes cluster; wherein, the running node corresponding to the application locally stores the application log generated by the application during the running process to obtain the local application log collection;

Determine the completed application according to the running state, and store the locally stored application log corresponding to the completed application in the cloud to obtain a cloud application log collection;

In response to the application log query request initiated by the user, according to the application number included in the application log query request, the local application log collection and the cloud application log collection are sequentially queried to determine the target application log, and the target application log is sent to the user.

Further, the steps of obtaining the running status of the application running on the kubernetes cluster include:

Obtain a list of application numbers from the kubernetes cluster at regular intervals. The running status corresponding to each application number in the application number list is running on the kubernetes cluster.

Further, the step of determining the completed application according to the running state includes:

Compare the application number list with the local list, and determine the application in the complement of the application number list and the local list as the completed application, where the local list includes the application number corresponding to the application applied for running on the kubernetes cluster.

Further, after performing the cloud storage of the locally stored application logs corresponding to the completed applications to obtain a set of local application logs, the method further includes:

Delete the application log corresponding to the completed application from the local application log collection.

Further, the application is a Spark application, there are multiple running nodes, and each running node stores the application logs generated during the running process of the Spark application respectively, to obtain a local application log set corresponding to each running node.

Further, the application log query request also includes the running node number;

The steps of sequentially querying the local application log collection and the cloud application log collection according to the application number included in the application log query request to determine the target application log further include:

According to the application number and the running node number included in the log query request, the local application log collection and the cloud application log collection corresponding to the running node number are sequentially queried to determine the target application log.

According to a second aspect of the embodiments of the present invention, a log service device based on a kubernetes cluster is provided, including a running state acquisition module, a log storage module, and a log query module; wherein,

The running status obtaining module is used to obtain the running status of the application running on the kubernetes cluster, and send the obtained running status to the log storage module, where the running node corresponding to the application will process the application log generated by the application during the running process. Local storage, get the local application log collection;

The log storage module is used to determine the completed application according to the running state, and store the locally stored application log corresponding to the completed application in the cloud to obtain a cloud application log collection;

The log query module is used to query the local application log collection and the cloud application log collection in turn according to the application number included in the application log query request in response to the application log query request initiated by the user, so as to determine the target application log, and store the target application log. sent to the user.

Further, the application is a Spark application, there are multiple running nodes, and each running node stores the application logs generated during the running process of the Spark application respectively, to obtain a local application log set corresponding to each running node.

According to a third aspect of the embodiments of the present invention, an electronic device is provided, including:

one or more processors;

storage means for storing one or more programs,

When one or more programs are executed by one or more processors, the one or more processors implement any of the above-mentioned log service methods based on a kubernetes cluster.

According to a fourth aspect of the embodiments of the present invention, there is provided a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, implements any of the foregoing Kubernetes cluster-based log service methods.

An embodiment of the above invention has the following advantages or beneficial effects: because the running state of the application running on the kubernetes cluster is obtained; wherein, the running node corresponding to the application locally stores the application log generated by the application during the running process to obtain Local application log collection; determine the completed application according to the running state, and store the locally stored application log corresponding to the completed application in the cloud to obtain the cloud application log collection; in response to the application log query request initiated by the user, according to the application log Query the application number included in the request, query the local application log collection and the cloud application log collection in turn, to determine the target application log, and send the target application log to the user. The log service method mainly adopts the traditional distributed log solution, which consumes storage resources and the technical problem of low log query efficiency, thereby saving storage resources, simplifying the log query process, improving log query efficiency, and improving the technical effect of user experience; Further, the above application is a Spark application, which solves the problem that in the prior art, when the Spark application is running in a kubernetes cluster, the logs need to be queried and displayed through specific pages, search, query, display and other components, and the log query process is complicated, resulting in poor user experience. technical problems, and then achieve the technical effect of improving the user experience.

Further effects of the above non-conventional alternatives will be described below in conjunction with specific embodiments.

Description of drawings

The accompanying drawings are used for better understanding of the present invention and do not constitute an improper limitation of the present invention. in:

1 is a schematic diagram of the main process of a log service method based on a kubernetes cluster provided according to a first embodiment of the present invention;

2 is a schematic diagram of the main process of a log service method based on a kubernetes cluster provided according to a second embodiment of the present invention;

3 is a schematic flowchart of a specific embodiment corresponding to the method described in FIG. 2;

4 is a schematic diagram of main modules of a log service device based on a kubernetes cluster provided according to an embodiment of the present invention;

5 is an exemplary system architecture diagram to which an embodiment of the present invention may be applied;

FIG. 6 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

1 is a schematic diagram of the main process of a log service method based on a kubernetes cluster provided according to a first embodiment of the present invention; the log service method according to the embodiment of the present invention mainly involves collection (storage), query and display of application logs. As shown in FIG. 1, the log service method based on the kubernetes cluster provided by the embodiment of the present invention mainly includes:

Step S101 , obtaining the running status of the application running on the kubernetes cluster; wherein, the running node corresponding to the application locally stores the application log generated during the running process of the application to obtain a local application log set.

In the prior art, the collection of application logs of applications running on the kubernetes cluster is mainly through traditional distributed log collection solutions such as ELK, that is, to rebuild a set of multiple components (including log collection components, log storage management, etc.) Components, log analysis display components, etc., log service systems with complex architecture and high coupling between components, such as log storage, by mounting application logs generated by applications running on kubernetes clusters on network storage For storage, the application startup delay is often caused when the application is running, and the application startup failure rate is high.

In the implementation of the present invention, a local storage device is mounted for the running node corresponding to the application, so that the running node can directly store the application log generated during the running of the application locally. Due to the faster local storage speed, the log storage efficiency is improved, and the storage failure problem caused by the simultaneous operation of multiple applications in the kubernetes cluster is solved.

Further, according to an embodiment of the present invention, the above step of acquiring the running status of an application running on a kubernetes cluster includes:

Obtain a list of application numbers from the kubernetes cluster at regular intervals. The running status corresponding to each application number in the application number list is running on the kubernetes cluster.

In the actual running process, the running status includes "running" (running, running), "running retrying", "running failure", "submitting", "running successfully" (wherein, the last four statuses all indicate At a certain stage or all running completed), and other intermediate states, not just two states of "running" and "running complete", through the above settings, the list of application numbers whose running state is "running" can be accurately obtained. Get running apps, so you can determine which apps aren't running. It is beneficial to collect application logs corresponding to applications that are "not running", and is beneficial to avoid the loss of application logs.

Step S102 , determining the completed application according to the running state, and storing the locally stored application log corresponding to the completed application in the cloud to obtain a cloud application log set.

Specifically, according to an embodiment of the present invention, the above step of determining an application that has completed execution according to the running state includes:

Compare the application number list with the local list, and determine the application in the complement of the application number list and the local list as the completed application, where the local list includes the application number corresponding to the application applied for running on the kubernetes cluster.

The local list records all application numbers submitted for running in the kubernetes cluster. By comparing the application number list corresponding to the running application with the local list, the complement of the two is the application that is "not running". That is, for an application that has been run, the application log corresponding to this type of application is transferred from the local storage device to the cloud storage device, so as to facilitate subsequent log query.

Further, according to a specific implementation of the embodiment of the present invention, a first application number can also be determined from the local list, and it is checked whether the first application number exists in the received "application number list corresponding to the running application". , if it exists, it means that the application corresponding to the first application number is in the "running" state; if it does not exist, the first application number can be cloud-stored in the application log corresponding to the local storage device.

Further, according to the embodiment of the present invention, after the step of storing the locally stored application log corresponding to the completed application in the cloud to obtain the local application log set, the method further includes: from the local application log set, storing the running application log in the cloud. The corresponding application log is deleted.

Through the above settings, after the corresponding application logs in the local storage device are stored in the cloud, the corresponding application logs in the local storage device are deleted, which can free up the storage space of the local storage device, reduce the burden of running nodes, and avoid the need for local storage space. Insufficient, affecting the operating efficiency of the running node.

Step S103, in response to the application log query request initiated by the user, according to the application number included in the application log query request, sequentially query the local application log collection and the cloud application log collection to determine the target application log, and send the target application log to the user. .

Specifically, according to the embodiment of the present invention, the application log query request may further include the application running time (for the application log named with the running time as the storage file, the efficiency and accuracy of querying the target application log can be improved).

Further, according to the embodiment of the present invention, the above application is a Spark application, there are multiple running nodes, and each running node stores application logs generated during the running process of the application respectively, to obtain a local application log set corresponding to each running node.

According to the embodiment of the present invention, the above application is a distributed computing program, preferably a Spark application. Distributed computing programs involve a large amount of computation and require the synergy of multiple running nodes, and are more suitable for running on kubernetes clusters.

Spark application is a popular big data computing framework with a wide range of applications. In the existing technology, Spark applications run in Yarn (another scheduling framework) cluster, but due to Yarn's insufficient scalability and low flexibility , the resource utilization rate is not high, and the hybrid deployment of offline and online cannot be well realized. Through the combination of Spark application and kubernetes cluster (a containerized application that manages multiple hosts in the cloud platform), it can not only unify the scheduling framework, optimize the computing process, but also realize the hybrid deployment of online and offline services.

Preferably, according to the embodiment of the present invention, the application log query request further includes the running node number; the above-mentioned according to the application number included in the application log query request, the local application log collection and the cloud application log collection are sequentially queried to determine the target application log. Steps also include:

According to the application number and the running node number included in the log query request, the local application log collection and the cloud application log collection corresponding to the running node number are sequentially queried to determine the target application log.

Specifically, after the application is started, it will be assigned tasks (tasks) for multiple running nodes, and the multiple running nodes will execute part of the running tasks respectively, and the running node number can be further specified in the application log query request, and the running node can be obtained. The corresponding target application log, and then the running status and resource usage status of the application on the running node can be determined according to the target application log, so as to monitor and check the running health status of the application on each running node.

According to the technical solution of the embodiment of the present invention, because the running state of the application running on the kubernetes cluster is obtained by adopting; wherein, the running node corresponding to the application locally stores the application log generated by the application during the running process to obtain a local application log collection; Determine the completed application according to the running state, and store the locally stored application log corresponding to the completed application in the cloud to obtain the cloud application log collection; in response to the application log query request initiated by the user, according to the application log query request included Application number, query the local application log collection and the cloud application log collection in turn, to determine the target application log, and send the target application log to the user. The traditional distributed log solution is mainly used, which consumes storage resources and the technical problem of low log query efficiency, thereby saving storage resources, simplifying the log query process, improving log query efficiency, and improving the technical effect of user experience; further, the above application For Spark applications, it solves the technical problems in the prior art that when Spark applications run in a kubernetes cluster, logs need to be queried and displayed through specific pages, search, query, display and other components, and the log query process is complicated, resulting in poor user experience. To achieve the technical effect of improving the user experience.

FIG. 2 is a schematic diagram of the main process of a log service method based on a kubernetes cluster provided according to a second embodiment of the present invention; the application scenario of the embodiment of the present invention is a log service method of Spark running on a kubernetes cluster, as shown in FIG. 2 , mainly include:

Step S201 , obtain a list of application numbers from the kubernetes cluster at regular intervals, wherein the running states corresponding to each application number in the application number list are all running on the kubernetes cluster.

Specifically, the Spark application runs on the kubernetes cluster, and the Spark application runs on multiple running nodes. Each running node stores the application logs generated during the running process locally to obtain a local application log collection. Specifically, according to a specific implementation of the embodiment of the present invention, as shown in FIG. 3 , after Spark submits a running application to the kubernetes cluster through submit (submission program), it uses the driver (driver) for multiple running nodes (for Spark: Computing nodes, executors, computing programs) allocate multiple tasks (tasks) respectively, each running node executes the task processing (that is, running the application), and stores the application logs generated during the running process to the local disk (local volume), that is, local storage. The kubernetes cluster uses SparkTaskSvc (running status acquisition module) to query the running status of the application (which can be specific to each running node).

Since Spark has multiple intermediate states during the running process, not only two states of "running" and "running completed", through the above settings, the list of application numbers whose running state is "running" can be obtained accurately. Applications that are running, which in turn can determine applications that are not running.

Further, according to the embodiments of the present invention, the above applications are not only applicable to Spark applications, but also to distributed computing programs, which require a large amount of computation involved in the running process and require the synergy of multiple running nodes, so they are all suitable for running on kubernetes clusters.

Spark application is a popular big data computing framework with a wide range of applications. In the prior art, Spark applications run in Yarn (another scheduling framework) cluster, but due to Yarn's lack of scalability, low flexibility, and low resource utilization, it cannot well achieve a mix of offline and online. deploy. Through the combination of Spark application and kubernetes cluster (a containerized application that manages multiple hosts in the cloud platform), it can not only unify the scheduling framework, optimize the computing process, but also realize the hybrid deployment of online and offline services.

Step S202 , according to the list of application numbers and the first application number, determine whether the application corresponding to the first application number is finished running. If yes, that is, the application corresponding to the first application number is completed, step S203 is executed; if not, that is, the running state corresponding to the first application number is "running", and the process goes to step S201.

Further, according to the embodiment of the present invention, the local list can also be directly compared with the above-mentioned application number list, and the application in the intersection of the application number list and the local list is determined to be an application that has not been completed; The application in the complement of the local list is determined as the completed application. Since the local list records all application numbers submitted for running in the kubernetes cluster, by comparing the application number list corresponding to the running application with the local list, the complement of the two is the application "not running" , for the application log corresponding to this type of application, transfer it from the local storage device to the cloud storage device, so as to facilitate subsequent log query.

Step S203: Determine the application log corresponding to the first application number in the local storage device, perform cloud storage for it, and delete the corresponding application log in the local storage device.

Through the above settings, after the corresponding application logs in the local storage device are stored in the cloud, the corresponding application logs in the local storage device are deleted, which can release the storage space of the local storage device, reduce the burden on the operating node, and avoid the need for local storage space. Insufficient, affecting the operating efficiency of the running node. Specifically, according to a specific implementation of the embodiment of the present invention, by setting a cleaner (cleaning module, which is equivalent to the log storage module in the following, it can be set in each node of the kubernetes cluster), cleaner (log storage module) Obtain the running status from SparkTaskSvc (running status acquisition module), and store the application log corresponding to the "running completed" application to the cloud (store to CS, cloud storage, cloud storage), and at the same time, store the application log in the local disk (localvolume) The corresponding application log is deleted.

Step S204, in response to the application log query request initiated by the user, according to the application number and the running node number included in the log query request, sequentially query the local application log set and the cloud application log set corresponding to the running node number to determine the target application log. .

Specifically, according to the embodiment of the present invention, the application log query request may further include the application running time (for the application log named with the running time as the storage file, the efficiency and accuracy of querying the target application log can be improved).

Through the above settings, the target application log corresponding to the running node can be obtained, and then the running status and resource usage status of the application on the running node can be determined according to the target application log, so as to realize the running health status of the application on each running node. Monitoring and troubleshooting. Specifically, as shown in FIG. 3, according to a specific implementation of the embodiment of the present invention, by setting the log service (log query module, providing log query service), after receiving the log query request initiated by the user, the log query request The information indicated in (including the application number and the running node number), query the local disk and cloud storage device in turn to determine the target application log.

Step S205, sending the target application log to the user.

Specifically, as shown in FIG. 3, according to a specific implementation of the embodiment of the present invention, a user can initiate a log query request through the SparkUI or HistoryServer (history service program) of the Spark application (HistoryServer and SparkUI pages provide a set of The web ui user interface used (this interface provides the log query link of each running node) is used to monitor the running status of the Spark application and the resource consumption of the kubernetes cluster. Completed application tasks use HistoryServer). The user can initiate a log query request including the application running time, application number, and running node number to the log service (log query module) through the page, receive the target application log returned by the log service, and finally display the target application to the user on the SparkUI or HistoryServer page. log. Through the above settings, the query process of application logs is hidden from users, which retains the user's habit of viewing application logs through SparkUI or HistoryServer page under scheduling frameworks such as Yarn, which improves user experience.

According to the technical solution of the embodiment of the present invention, because the running state of the application running on the kubernetes cluster is obtained by adopting; wherein, the running node corresponding to the application locally stores the application log generated by the application during the running process to obtain a local application log collection; Determine the completed application according to the running state, and store the locally stored application log corresponding to the completed application in the cloud to obtain the cloud application log collection; in response to the application log query request initiated by the user, according to the application log query request included Application number, query the local application log collection and the cloud application log collection in turn, to determine the target application log, and send the target application log to the user. The traditional distributed log solution is mainly used, which consumes storage resources and the technical problem of low log query efficiency, thereby saving storage resources, simplifying the log query process, improving log query efficiency, and improving the technical effect of user experience; further, the above application For Spark applications, it solves the technical problems in the prior art that when Spark applications run in a kubernetes cluster, logs need to be queried and displayed through specific pages, search, query, display and other components, and the log query process is complicated, resulting in poor user experience. To achieve the technical effect of improving the user experience.

Fig. 4 is a schematic diagram of main modules of a kubernetes cluster-based log service device provided according to an embodiment of the present invention; as shown in Fig. 4 , the kubernetes cluster-based log service device 400 provided by an embodiment of the present invention mainly includes:

The running state obtaining module 401 is used to obtain the running state of the application running on the kubernetes cluster; wherein, the running node corresponding to the application locally stores the application log generated by the application during the running process to obtain a local application log set.

In the implementation of the present invention, a local storage device is mounted for the running node corresponding to the application, so that the running node can directly store the application log generated during the running of the application locally. Due to the faster local storage speed, the log storage efficiency is improved, and the storage failure problem caused by the simultaneous operation of multiple applications in the Kubernetes cluster is solved.

Further, according to the embodiment of the present invention, the above-mentioned operating state obtaining module 401 is further configured to: periodically obtain a list of application numbers from the kubernetes cluster, wherein the operating states corresponding to each application number in the application number list are all running on the kubernetes cluster.

In the actual running process, the running status is not only two statuses of "running" and "running successfully". Through the above settings, the number list of applications whose running status is "running" can be obtained, and the running applications can be accurately obtained. In turn, it is possible to determine which applications are not running. It is beneficial to collect application logs corresponding to applications that are "not running", and is beneficial to avoid the loss of application logs.

The log storage module 402 is configured to determine the completed application according to the running state, and store the locally stored application log corresponding to the completed application in the cloud to obtain a cloud application log set.

Specifically, according to the embodiment of the present invention, the above log storage module 402 is further configured to:

Compare the application number list with the local list, and determine the application in the complement of the application number list and the local list as the completed application, where the local list includes the application number corresponding to the application applied for running on the kubernetes cluster.

The local list records all application numbers submitted for running in the kubernetes cluster. By comparing the application number list corresponding to the running application with the local list, the complement of the two is the application that is "not running". For the application log corresponding to this type of application, it is transferred from the local storage device to the cloud storage device, so as to facilitate subsequent log query.

Further, according to the embodiment of the present invention, after the step of cloud storage of the locally stored application logs corresponding to the completed applications to obtain the local application log set, the above log storage module 402 is further configured to:

Delete the application log corresponding to the completed application from the local application log collection.

Through the above settings, after the corresponding application logs in the local storage device are stored in the cloud, the corresponding application logs in the local storage device are deleted, which can free up the storage space of the local storage device, reduce the burden of running nodes, and avoid the need for local storage space. Insufficient, affecting the operating efficiency of the running node.

The log query module 403 is configured to, in response to the application log query request initiated by the user, sequentially query the local application log collection and the cloud application log collection according to the application number included in the application log query request, to determine the target application log, and use the target application log. The log is sent to the user.

Specifically, according to the embodiment of the present invention, the application log query request may further include the application running time (for the application log named with the running time as the storage file, the efficiency and accuracy of querying the target application log can be improved).

Further, according to the embodiment of the present invention, the above application is a Spark application, there are multiple running nodes, and each running node stores application logs generated during the running process of the application respectively, to obtain a local application log set corresponding to each running node.

According to the embodiment of the present invention, the above application is a distributed computing program, preferably a Spark application. Distributed computing programs are designed with a large amount of computation and require the synergy of multiple running nodes, and are more suitable for running on Kubernetes clusters.

Spark application, as a popular big data computing framework, has a wide range of applications. In the prior art, Spark applications run in Yarn (another scheduling framework) cluster. Due to Yarn's insufficient scalability, low flexibility, and resources The utilization rate is not high, and the hybrid deployment of offline and online cannot be well realized. Through the combination of Spark application and kubernetes cluster (a containerized application that manages multiple hosts in the cloud platform), it can not only unify the scheduling framework, optimize the computing process, but also realize the hybrid deployment of online and offline services.

Preferably, according to the embodiment of the present invention, the application log query request further includes the running node number; the above log query module 403 is further configured to:

According to the application number and the running node number included in the log query request, the local application log collection and the cloud application log collection corresponding to the running node number are sequentially queried to determine the target application log.

Specifically, after the application is started, it will be assigned tasks (tasks) for multiple running nodes, and the multiple running nodes will execute part of the running tasks respectively, and the running node number can be further specified in the application log query request, and the running node can be obtained. The corresponding target application log, and then the running status and resource usage status of the application on the running node can be determined according to the target application log, so as to monitor and check the running health status of the application on each running node.

According to the technical solution of the embodiment of the present invention, because the running state of the application running on the kubernetes cluster is obtained by adopting; wherein, the running node corresponding to the application locally stores the application log generated by the application during the running process to obtain a local application log collection; Determine the completed application according to the running state, and store the locally stored application log corresponding to the completed application in the cloud to obtain the cloud application log collection; in response to the application log query request initiated by the user, according to the application log query request included Application number, query the local application log collection and the cloud application log collection in turn, to determine the target application log, and send the target application log to the user. The traditional distributed log solution is mainly used, which consumes storage resources and the technical problem of low log query efficiency, thereby saving storage resources, simplifying the log query process, improving log query efficiency, and improving the technical effect of user experience; further, the above application For Spark applications, it solves the technical problems in the prior art that when Spark applications run in a kubernetes cluster, logs need to be queried and displayed through specific pages, search, query, display and other components, and the log query process is complicated, resulting in poor user experience. To achieve the technical effect of improving the user experience.

FIG. 5 shows an exemplary system architecture 500 of a kubernetes cluster-based log service method or a kubernetes cluster-based log service apparatus to which embodiments of the present invention can be applied.

As shown in FIG. 5, the system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505 (this architecture is only an example, and the components included in the specific architecture can be adjusted according to the specific application). The network 504 is a medium used to provide a communication link between the terminal devices 501 , 502 , 503 and the server 505 . Network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user can use the terminal devices 501, 502, 503 to interact with the server 505 through the network 504 to receive or send messages and the like. Various communication client applications may be installed on the terminal devices 501 , 502 and 503 , such as big data computing applications, data processing applications, log service applications, etc. (only examples).

The terminal devices 501, 502, 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

The server 505 may be a server that provides various services, for example, a background management server that provides support for shopping websites browsed by the terminal devices 501 , 502 , and 503 (just an example). The background management server can analyze and process the received application log query request and other data, and feed back the processing result (for example, the target application log—just an example) to the terminal device.

It should be noted that the log service method based on the kubernetes cluster provided by the embodiment of the present invention is generally executed by the server 505 , and accordingly, the log service device based on the kubernetes cluster is generally set in the server 505 .

It should be understood that the numbers of terminal devices, networks and servers in FIG. 5 are only illustrative. There can be any number of terminal devices, networks and servers according to implementation needs.

Referring to FIG. 6 below, it shows a schematic structural diagram of a computer system 600 suitable for implementing a terminal device or a server according to an embodiment of the present invention. The terminal device or server shown in FIG. 6 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present invention.

As shown in FIG. 6, a computer system 600 includes a central processing unit (CPU) 601, which can be loaded into a random access memory (RAM) 603 according to a program stored in a read only memory (ROM) 602 or a program from a storage section 608 Instead, various appropriate actions and processes are performed. In the RAM 603, various programs and data necessary for the operation of the system 600 are also stored. The CPU 601 , the ROM 602 , and the RAM 603 are connected to each other through a bus 604 . An input/output (I/O) interface 605 is also connected to bus 604 .

The following components are connected to the I/O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc. ; and a communication section 609 including a network interface card such as a LAN card, a modem, and the like. The communication section 609 performs communication processing via a network such as the Internet. Drivers 610 are also connected to I/O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.

In particular, the processes described above with reference to the flowcharts may be implemented as computer software programs in accordance with the disclosed embodiments of the present invention. For example, embodiments disclosed herein include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods illustrated in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 609 and/or installed from the removable medium 611 . When the computer program is executed by the central processing unit (CPU) 601, the above-described functions defined in the system of the present invention are performed.

It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more functions for implementing the specified logical function(s) executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented in special purpose hardware-based systems that perform the specified functions or operations, or can be implemented using A combination of dedicated hardware and computer instructions is implemented.

The modules involved in the embodiments of the present invention may be implemented in a software manner, and may also be implemented in a hardware manner. The described module can also be set in the processor, for example, it can be described as: a processor includes a running state acquisition module, a log storage module and a log query module. Among them, the name of these modules does not constitute a limitation of the module itself under certain circumstances. For example, the running status acquisition module can also be described as "used to obtain the running status of the application running on the kubernetes cluster, and will obtain The obtained running status is sent to the log storage module, wherein the running node corresponding to the application locally stores the application log generated by the application during the running process, and obtains the module of the local application log collection.

As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or may exist alone without being assembled into the device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by a device, the device includes: obtaining the running state of the application running on the kubernetes cluster; wherein, the corresponding running state of the application The node stores the application logs generated by the application during the running process locally, and obtains the local application log collection; determines the completed application according to the running status, and stores the locally stored application log corresponding to the completed application in the cloud to obtain the cloud application Log collection; in response to the application log query request initiated by the user, according to the application number included in the application log query request, sequentially query the local application log collection and the cloud application log collection to determine the target application log, and send the target application log to the user. .

According to the technical solution of the embodiment of the present invention, because the running state of the application running on the kubernetes cluster is obtained by adopting; wherein, the running node corresponding to the application locally stores the application log generated by the application during the running process to obtain a local application log collection; Determine the completed application according to the running state, and store the locally stored application log corresponding to the completed application in the cloud to obtain the cloud application log collection; in response to the application log query request initiated by the user, according to the application log query request included Application number, query the local application log collection and the cloud application log collection in turn, to determine the target application log, and send the target application log to the user. The traditional distributed log solution is mainly used, which consumes storage resources and the technical problem of low log query efficiency, thereby saving storage resources, simplifying the log query process, improving log query efficiency, and improving the technical effect of user experience; further, the above application For Spark applications, it solves the technical problems in the prior art that when Spark applications run in a kubernetes cluster, logs need to be queried and displayed through specific pages, search, query, display and other components, and the log query process is complicated, resulting in poor user experience. To achieve the technical effect of improving the user experience.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. A journal service method based on kubernets cluster is characterized by comprising the following steps:

acquiring the running state of the application running on the kubernets cluster; the running node corresponding to the application locally stores an application log generated in the running process of the application to obtain a local application log set;

determining an application which is finished running according to the running state, and performing cloud storage on a locally stored application log corresponding to the application which is finished running to obtain a cloud application log set;

responding to an application log query request initiated by a user, sequentially querying the local application log set and the cloud application log set according to an application number included in the application log query request to determine a target application log, and sending the target application log to the user.

2. The kubernets cluster-based log service method of claim 1, wherein the step of obtaining the running state of the applications running on the kubernets cluster includes:

and acquiring an application number list from the kubernets cluster at regular time, wherein the running state corresponding to each application number in the application number list is running on the kubernets cluster.

3. The kubernets cluster-based log service method of claim 2, wherein the step of determining the running-completed application according to the running state includes:

and comparing the application number list with a local list, and determining the applications in the complementary set of the application number list and the local list as the applications which are completely run, wherein the local list comprises application numbers corresponding to the applications which are applied for running on the kubernets cluster.

4. The kubernets cluster-based log service method of claim 1, wherein after the step of performing cloud storage on locally stored application logs corresponding to the applications that are completely run to obtain a local application log set, the method further comprises:

and deleting the application log corresponding to the application which is finished in operation from the local application log set.

5. The kubernetes cluster-based log service method of claim 1, wherein the application is a Spark application, the number of the operating nodes is multiple, and each operating node stores an application log generated by the Spark application in an operating process locally to obtain a local application log set corresponding to each operating node.

6. The kubernets cluster-based log service method of claim 5, wherein the application log query request further includes a running node number;

the step of sequentially querying the local application log set and the cloud application log set according to the application number included in the application log query request to determine the target application log further includes:

and sequentially inquiring a local application log set and the cloud application log set corresponding to the running node number according to the application number and the running node number included in the log inquiry request so as to determine a target application log.

7. A journal service device based on a kubernets cluster is characterized by comprising an operation state acquisition module, a journal storage module and a journal query module; wherein,

the running state acquisition module is used for acquiring the running state of the application running on the kubernets cluster and sending the acquired running state to the log storage module, wherein the running node corresponding to the application locally stores the application log generated in the running process of the application to obtain a local application log set;

the log storage module is used for determining an application which is finished running according to the running state, and performing cloud storage on a locally stored application log corresponding to the application which is finished running to obtain a cloud application log set;

the log query module is used for responding to an application log query request initiated by a user, sequentially querying the local application log set and the cloud application log set according to an application number included in the application log query request to determine a target application log, and sending the target application log to the user.

8. The kubernetes cluster-based log service device of claim 7, wherein the application is a Spark application, the number of the operating nodes is multiple, and each operating node stores an application log generated by the Spark application in an operating process locally, so as to obtain a local application log set corresponding to each operating node.

9. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-6.

10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-6.

Images