CN114443046A - A graph computing method and system based on dynamic code generation - Google Patents

Abstract

The invention discloses a graph calculation method and a graph calculation system based on dynamic code generation, and particularly relates to the field of computers. The invention eliminates the overhead of data exchange by directly injecting the generated code into the local code space; constructing a secondary-compiling intermediate graph structure, so that the access code of graph data can be compiled and optimized; meanwhile, the intermediate graph structure cache is added, and the preprocessing overhead of a graph algorithm is avoided.

Description

A graph computing method and system based on dynamic code generation

technical field

The present invention relates to the field of computer technology, and more particularly, the present invention relates to a graph computing method and system based on dynamic code generation.

Background technique

Graph computing is an important technology for analyzing and mining associations in data. Due to the nonlinear characteristics of graph structure, the development of graph algorithms is difficult, the performance is poor, and the code is not easy to optimize. In order to support the needs of efficient graph computing, the corresponding computing framework should be used. Andsheng.Graph computing framework encapsulates graphs into a unified data structure, and provides basic operators to assist the development of graph algorithms.

The existing graph computing frameworks are mainly divided into two categories: the first category is to provide a fixed graph operator library and use an interpreted language to combine operators to implement general graph computing operations, such as graphkit; this kind of framework algorithm has low development cost, Flexible use, but poor performance; the second type is to use concurrent libraries such as OpenMP or Cilk to implement compiled graph computing APIs to support graph algorithm development, such as Ligra; this type of framework has better performance and reduces the complexity of graph algorithms to a certain extent. The development threshold is high, but the algorithm implementation is still very difficult, and the code modification is difficult. With the explosive growth of graph data, implementing high-performance graph algorithms in a flexible manner has become a hot demand in the current graph computing field. Therefore, the use of dynamic code generation techniques to develop graph computing frameworks has become a major trend. Dynamic code generation is a classic way to combine the advantages of both interpreted and compiled languages. The user submits the algorithm description through the interactive interface, and the corresponding graph computing system compiles and runs in real time, which greatly improves the flexibility of graph algorithm development without losing operational efficiency.

However, the existing graph framework for dynamic code generation still has some defects. As shown in Figure 1, the existing graph computing framework based on code generation is developed using interpreted code, which forms an external code space, and only calculates the core graph code. For dynamic generation, some data needs to be passed to the local code space through unboxing and returned through boxing, which increases the overhead of data packing and unboxing.

SUMMARY OF THE INVENTION

In order to overcome the above-mentioned defects of the prior art, the embodiments of the present invention provide a graph computing method and system based on dynamic code generation, so as to solve the problems raised in the above-mentioned background art.

To achieve the above object, the present invention provides the following technical solutions:

A graph computing method and system based on dynamic code generation, comprising a graph computing framework main body, which is set in a local code space, the graph computing framework main body including a code generator connected to an external editing desk, and the code generator is used for receiving The graph algorithm code output by the external editing platform is directly injected into the local code space to generate the graph algorithm code, which realizes the main body of the graph computing system using the local language.

In a preferred embodiment, the graph computing framework body further includes an intermediate graph compiler, a graph building module, and a graph buffer;

The graph building module is used to convert the data source into graph data, and output the graph data in combination with the basic operator code to the intermediate graph compiler;

The intermediate graph compiler is used to obtain the code construction object sent by the graph construction module, assemble the object into a compilable local code structure using code generation primitives, and output the structure to the graph buffer for storage;

The graph buffer is used to receive and save the native code structure output by the intermediate graph compiler.

In a preferred embodiment, the graph computing framework main body is provided with a general graph algorithm development API based on the intermediate graph structure.

In a preferred embodiment, the editing platform provides a graph algorithm template, including a graph algorithm signature, a basic graph operation operator API and a graph data access API.

In a preferred embodiment, the graph computing framework main body further includes a code executor,

The code executor is responsible for receiving the user's connection request and generating an execution platform;

The execution platform is used for parsing the graph algorithm name, reading the corresponding native code from the code executor and injecting it, searching for the corresponding intermediate graph code from the graph cache, and generating the final code in combination with the graph algorithm native code and executing it.

Technical effects and advantages of the present invention:

The invention eliminates the overhead of data exchange by directly injecting the generated code into the local code space; constructs an intermediate graph structure that can be compiled twice, so that the access code of the graph data can be compiled and optimized; at the same time, the intermediate graph structure cache is added, The preprocessing overhead of graph algorithms is avoided.

Description of drawings

1 is a schematic diagram of a graph computing framework structure generated by an existing dynamic code;

2 is a schematic diagram of a graph computing framework structure generated by dynamic code of the present invention;

3 is a schematic diagram of the internal structure of the graph computing framework generated by the dynamic code of the present invention;

Fig. 4 is a schematic diagram of the pre-generated code of the graph computing framework of the present invention;

FIG. 5 is a schematic diagram of the graph algorithm of the present invention taking the same graph data.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Example 1

A graph computing method and system based on dynamic code generation of the present invention, as shown in Figures 2 and 3, includes a local code space and an external editing platform; the local code space includes a graph computing framework body, and the graph computing framework body is The external code space in the prior art is transferred to the local code space, and the generated code is injected to solve the data exchange overhead; the external editing desk is provided with a graph algorithm template, and the graph computing framework body includes the code connected to the external editing desk The generator is used to receive external editing desk code and monitor code changes, and directly inject the generated graph algorithm code into the local code space. It realizes the main body of the graph computing system using local language, stores the graph data in its original structure, and transfers it to the dynamically generated code space through the memory sharing mechanism, realizing zero copy of data and avoiding data exchange overhead.

Example 2

As a further optimization improvement, in the above embodiment 1, we solve the data exchange overhead by transferring the main body of the graph computing framework from the external code space in the prior art to the local code space, and injecting the generated code; Provide more sufficient compilation optimization and solve the function call overhead. As shown in Figure 3, the main body of the graph computing framework also includes an intermediate graph compiler, a graph building module and a graph buffer; the graph building module is used to convert the data source Convert the graph data into graph data, and output the graph data in combination with the basic operator code to the intermediate graph compiler; the intermediate graph compiler obtains the code construction object sent by the graph construction module, and assembles the object into a compilable code using the code generation primitives the native code structure, and output the structure to the graph buffer for saving; the graph buffer is used to receive and save the native code structure output by the intermediate graph compiler.

As shown in Figure 4, the basic operation code of the graph is precompiled in this way, forming an intermediate graph structure that can be recompiled, and combined with the graph algorithm for recompilation, eliminating the overhead of external calls.

As shown in FIG. 5 , this patent also enables different graph algorithms to reuse the same graph data through the graph buffer, which reduces the preprocessing time of the algorithm.

Example 3

As a further optimization improvement, in order to improve the usability of the graph computing framework main body and improve the user's development efficiency, the graph computing framework main body designs a general graph algorithm development API based on the intermediate graph structure. In the present invention, the user connects to the editing platform for graph algorithm development, the editing platform provides the graph algorithm template, including the graph algorithm signature, the basic graph operation operator API and the graph data access API, and the editing platform submits the graph algorithm to the code generator.

Further, in order to effectively utilize the API interface, as shown in FIG. 3 , the main body of the graph computing framework also includes a code executor, and the code executor is responsible for receiving the connection request from the user, generating an execution table, and the user submits the graph to the execution table. The algorithm execution request, the execution station parses the name of the graph algorithm, reads the corresponding local code from the code executor and injects it, the execution station searches for the corresponding intermediate graph code from the graph cache, and generates the final code combined with the graph algorithm local code and executes it. Output the result.

The last points to be noted are: First of all, in the description of this application, it should be noted that, unless otherwise specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, and may be mechanical connection. or electrical connection, or internal communication between two components, or direct connection, "up", "down", "left", "right", etc. are only used to indicate relative positional relationship, when the absolute position of the object being described changes, the relative positional relationship may change;

Secondly: in the drawings of the disclosed embodiments of the present invention, only the structures involved in the embodiments of the present disclosure are involved, other structures may refer to the general design, and the same embodiment and different embodiments of the present invention can be combined with each other under the condition of no conflict;

Finally: the above is only the preferred embodiment of the present invention, and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the present invention. within the scope of protection.

Claims (5)

1. a graph computation method and system based on dynamic code generation, it is characterized in that: comprise graph computation framework main body, it is arranged in local code space, and described graph computation framework main body comprises the code generator that is connected with external editing desk, so The code generator described above is used to receive the graph algorithm code output by the external editing platform and directly inject the generated graph algorithm code into the local code space, which realizes the main body of the graph computing system using the local language. 2. a kind of graph computing method and system based on dynamic code generation according to claim 1, is characterized in that: described graph computation framework main body also comprises intermediate graph compiler, graph building module and graph buffer; The graph building module is used to convert the data source into graph data, and output the graph data in combination with the basic operator code to the intermediate graph compiler; The intermediate graph compiler is used to obtain the code construction object sent by the graph construction module, assemble the object into a compilable local code structure using code generation primitives, and output the structure to the graph buffer for storage; The graph buffer is used to receive and save the native code structure output by the intermediate graph compiler. 3 . The graph computing method and system based on dynamic code generation according to claim 1 , wherein the graph computing framework main body is provided with a general graph algorithm development API based on the intermediate graph structure. 4 . 4. a kind of graph computing method and system based on dynamic code generation according to claim 1, is characterized in that: described editing platform provides graph algorithm template, comprises graph algorithm signature, basic graph operation operator API and graph data access API. 5. a kind of graph computation method and system based on dynamic code generation according to claim 2, is characterized in that: described graph computation framework main body also comprises code executor inside, The code executor is responsible for receiving the user's connection request and generating an execution platform; The execution platform is used for parsing the graph algorithm name, reading the corresponding native code from the code executor and injecting it, searching for the corresponding intermediate graph code from the graph cache, and generating the final code in combination with the graph algorithm native code and executing it.

Images