[go: up one dir, main page]

CN107168681B - Real-time task processing method and device - Google Patents

Real-time task processing method and device Download PDF

Info

Publication number
CN107168681B
CN107168681B CN201710210712.XA CN201710210712A CN107168681B CN 107168681 B CN107168681 B CN 107168681B CN 201710210712 A CN201710210712 A CN 201710210712A CN 107168681 B CN107168681 B CN 107168681B
Authority
CN
China
Prior art keywords
aggregation function
operation object
time
processing task
streaming processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201710210712.XA
Other languages
Chinese (zh)
Other versions
CN107168681A (en
Inventor
孙琦
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing QIYI Century Science and Technology Co Ltd
Original Assignee
Beijing QIYI Century Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing QIYI Century Science and Technology Co Ltd filed Critical Beijing QIYI Century Science and Technology Co Ltd
Priority to CN201710210712.XA priority Critical patent/CN107168681B/en
Publication of CN107168681A publication Critical patent/CN107168681A/en
Application granted granted Critical
Publication of CN107168681B publication Critical patent/CN107168681B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
    • G06F9/30003—Arrangements for executing specific machine instructions
    • G06F9/3005—Arrangements for executing specific machine instructions to perform operations for flow control
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
    • G06F9/30098—Register arrangements
    • G06F9/30101—Special purpose registers

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The embodiment of the invention discloses a real-time task processing method and a real-time task processing device, wherein the real-time task processing method comprises the following steps: when a streaming processing task currently executes an aggregation function, loading a stored encapsulation data packet of the aggregation function executed by the streaming processing task, wherein the encapsulation data packet comprises: an operation object of the aggregation function of the streaming processing task, a functional relationship of the operation object; and executing the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the aggregation function and the encapsulation data packet and the functional relation of the operation object. The embodiment of the invention realizes the state management of the real-time data processing process, conveniently performs the updating operation, reduces the system processing time and the workload of developers, and improves the development efficiency of real-time tasks.

Description

Real-time task processing method and device
Technical Field
The invention relates to the technical field of big data processing, in particular to a real-time task processing method and device.
Background
With the prosperity of the internet industry, especially the continuous rise of the mobile internet, more and more data are generated by machines and users, and various big data processing systems such as Hadoop, Spark, Storm and the like are also promoted. Some of these systems are batch processing engines for batch processing tasks, such as Hadoop MapReduce, some are real-time processing engines for streaming data processing, such as Storm, and some are capable of performing both batch processing tasks and streaming data processing real-time processing tasks, such as Spark.
When a batch big data processing system starts to calculate, all data are ready to wait for input calculation; in the streaming data processing engine, data is not completely prepared from the beginning, but gradually appears over time, like a behavior log of a user, and is characterized in that the data is not sent once but is 'streamed' little by little like running water. The data from the "stream" is also processed by the real-time processing system little by little. The main differences between the two types of tasks are: the batch processing task has global data, so that the data aggregation operation is convenient to carry out; and the real-time processing of the streaming data does not have global data and is in an aggregation intermediate state. When a streaming task is processed in real time, a real-time processing engine for processing streaming data needs to be rewritten by a developer in real time for processing processes and functional logic relationships of the same operation at different times or in different states. For example, the number of errors in a log stream is an accumulated number over a period of time, and needs to be temporarily stored in an external storage database, such as MySQL, Redis, and the like. When counting error number in a period of time for the first time, a developer writes a logic function of related operation and a program connected with a database, the program is used for counting the error number in the current time stored in an external storage system, when counting the error number in a period of time for the next time, the developer needs to rewrite the logic function of related operation and the program connected with the database, and the program is used for counting the error number in the current time stored in the external storage system again.
The existing method for processing streaming tasks in real time increases the system processing time and the workload of developers for processing processes of different time or different states operated by the same or similar logic relations, and has low overall development efficiency.
Disclosure of Invention
Embodiments of the present invention provide a method and an apparatus for processing a real-time task, so as to implement state management of a real-time data processing process, perform update operations conveniently, reduce system processing time and workload of a developer, and improve development efficiency of the real-time task. The specific technical scheme is as follows:
in order to achieve the above object, an embodiment of the present invention discloses a real-time task processing method, including:
when a streaming processing task currently executes an aggregation function, loading a stored encapsulation data packet of the aggregation function executed by the streaming processing task, wherein the encapsulation data packet comprises: an operation object of the aggregation function of the streaming processing task, a functional relationship of the operation object;
and executing the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the aggregation function and the encapsulation data packet and the functional relation of the operation object.
Optionally, before the loading, when the streaming processing task currently executes the aggregation function, the encapsulated packet of the aggregation function executed by the streaming processing task, the method further includes:
when the streaming processing task executes the aggregation function for the first time, executing the aggregation function operation of the streaming processing task according to the functional relationship of the operation objects on the numerical values of the operation objects of the aggregation function;
when the streaming processing task finishes executing the aggregation function for the first time, encapsulating the operation object of the aggregation function executed by the streaming processing task and the functional relationship of the operation object into the encapsulation data packet;
and storing the encapsulated data packet.
Optionally, after the aggregation function operation of the streaming processing task is performed on the numerical value of the operation object of the aggregation function according to the functional relationship of the operation object, the method further includes:
and executing the aggregation function operation of the streaming processing task on the numerical value of the operation object of the aggregation function according to the functional relation of the operation object to obtain a result of the numerical value of the operation object, and storing the result in a database.
Optionally, before the performing, according to the aggregation function and the encapsulation data packet, the aggregation function operation on the current value of the operation object of the aggregation function according to the functional relationship of the operation object, the method further includes:
accumulating the update data of the operation object from the time when the streaming processing task executes the aggregation function for the first time to the time when the streaming processing task executes the aggregation function at present, and accumulating the update data to the value of the operation object from the time when the streaming processing task executes the aggregation function for the first time to obtain the current value of the operation object from which the streaming processing task executes the aggregation function at present;
loading the encapsulation data packet of the aggregation function of the streaming processing task and the current value of the operand;
correspondingly, the executing, according to the operation object of the aggregation function in the encapsulated data packet and the functional relationship of the operation object, a corresponding operation on the current value of the operation object of the aggregation function according to the functional relationship of the operation object includes:
acquiring an operation object of the aggregation function in the encapsulated data packet and a function relation of the operation object through the encapsulated data packet;
and executing the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the functional relation of the operation object.
Optionally, the executing, according to the aggregation function and the encapsulation data packet, the aggregation function operation on the current value of the operation object of the aggregation function according to the functional relationship of the operation object includes:
acquiring an operation object of the aggregation function and a function relation of the operation object in the encapsulated data packet through the encapsulated data packet, and acquiring a current numerical value of the operation object of the aggregation function in a database;
and executing the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the functional relation of the operation object.
In order to achieve the above object, an embodiment of the present invention further discloses a real-time task processing device, including:
a state loading module, configured to load a stored encapsulated packet of an aggregation function executed by a streaming processing task when the streaming processing task currently executes the aggregation function, where the encapsulated packet includes: an operation object of the aggregation function of the streaming processing task, a functional relationship of the operation object;
and the execution module is used for executing the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the aggregation function and the encapsulation data packet and the functional relation of the operation object.
Optionally, the apparatus further comprises:
a first execution module, configured to execute, when the streaming processing task executes the aggregation function for the first time, the aggregation function operation of the streaming processing task according to a functional relationship of the operation object on the numerical value of the operation object of the aggregation function;
an encapsulation module, configured to encapsulate, when the streaming processing task finishes executing the aggregation function for the first time, the operation object of the aggregation function executed by the streaming processing task and the functional relationship of the operation object in the encapsulation data packet;
and the first storage module is used for storing the encapsulated data packet.
Optionally, the apparatus further comprises:
and the second storage module is used for executing the aggregation function operation of the streaming processing task on the numerical value of the operation object of the aggregation function according to the functional relation of the operation object to obtain a result of the numerical value of the operation object, and storing the result in a database.
Optionally, the apparatus further comprises:
an accumulation module, configured to accumulate update data of the operation object from when the streaming processing task executes the aggregation function for the first time to when the streaming processing task executes the aggregation function for the current time, and accumulate the update data to the value of the operation object from when the streaming processing task executes the aggregation function for the first time, so as to obtain a current value of the operation object from which the streaming processing task executes the aggregation function for the current time;
a data packet loading module, configured to load the encapsulated data packet of the aggregation function of the streaming processing task and the current value of the operation object;
correspondingly, the execution module includes:
the obtaining submodule is used for obtaining the operation object of the aggregation function in the encapsulated data packet and the functional relation of the operation object through the encapsulated data packet;
and the execution submodule is used for executing the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the functional relation of the operation object.
Optionally, the executing module is further configured to obtain, through the encapsulated data packet, an operation object of the aggregation function in the encapsulated data packet, a functional relationship of the operation object, and obtain, in a database, a current numerical value of the operation object of the aggregation function; and executing the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the functional relation of the operation object.
The embodiment of the invention discloses a real-time task processing method and a device, which pack, encapsulate and store an operation object of an aggregation function of a streaming processing task and a function relation of the operation object, directly acquire the stored operation object of the aggregation function of the streaming processing task and the function relation of the operation object when executing the same aggregation function of the streaming processing task in real time, and provide a method for encapsulating and caching the operation object of the same operation function and the logic relation of the function relation of the operation object. The data read-write interface does not need to be repeatedly written by a developer for the function interface with the same problem every time, the time for processing the real-time data with the same operation is reduced, and the overall development efficiency is improved. In addition, the aggregation function operation of the streaming processing task is executed according to the functional relation of the operation object, the result of the numerical value of the operation object is obtained and stored in the database, and the storage method of various data states of real-time task processing is realized. The embodiment of the invention finally realizes the state management of the real-time data processing process, conveniently performs the updating operation and improves the development efficiency of the real-time task. Of course, it is not necessary for any product or method of practicing the invention to achieve all of the above-described advantages at the same time.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flowchart of a real-time task processing method according to an embodiment of the present invention;
FIG. 2 is a flow chart of an implementation of a package data packet of a real-time task processing method according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating an operation of executing an aggregation function in a real-time task processing method according to an embodiment of the present invention;
FIG. 4 is a diagram of a real-time task processing device according to an embodiment of the present invention;
FIG. 5 is a diagram of another real-time task processing device according to an embodiment of the present invention;
fig. 6 is a diagram of an embodiment of a real-time task processing device.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
To achieve the above object, an embodiment of the present invention discloses a real-time task processing method, as shown in fig. 1. Fig. 1 is a flowchart of a real-time task processing method according to an embodiment of the present invention, including:
s101, when the stream processing task currently executes the aggregation function, loading an encapsulation data packet of the aggregation function executed by the stored stream processing task, wherein the encapsulation data packet comprises: the operation object of the aggregation function of the streaming processing task and the function relation of the operation object;
specifically, when a large data processing streaming task, different times of processing the same task and tasks in different states are required. The embodiment of the invention discloses that the aggregation function operations of the big data processing streaming task, which execute the same type and the same function, are stored in a database, and the stored logic relation is loaded from the database when the aggregation function operations of the same type and the same function are executed next time.
An aggregation function is a function that performs a computation on a set of values and returns a single value, often used in conjunction with selecting a group by clause of a select statement. The aggregation function includes: returning the average value in the specified group, and neglecting null values; returning the number of items in the specified group; returning the maximum value of the specified data; returning the minimum value of the specified data; the sum of the specified data is returned, only for the digit column, and null values are ignored; returning the number of items in the specified group; generating an additional column, the output value being 1 when a row is added by the tile cube operator or the curlolup operator, and 0 when the added row is not generated by the tile cube or the curlolup; returning binary check values calculated for the rows or expression lists in the table for detecting the change of the rows in the table; returning a check value of the specified data, and neglecting a null value; returning check values calculated on rows of the table or on the expression list for generating a hash index; returning the statistical standard deviation of all values in the given expression; returning the filling statistical standard deviation of all values in the given expression; returning the statistical variance of all values in the given expression; the filled statistical variance of all values in a given expression is returned.
Specifically, when the streaming processing task currently executes the aggregation function, an encapsulated data packet of the aggregation function, which is stored for the first time, is loaded from the database, and the aggregation function operation object and the function relationship of the operation object are encapsulated in the encapsulated data packet. And decapsulating the encapsulated data packet to obtain the aggregation function operation object and the function relation of the operation object.
And S102, executing the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the functional relation of the operation object according to the aggregation function and the packaging data packet.
Specifically, the current value of the operation object of the aggregation function that is currently executed is obtained in the database, the obtained functional relationship between the operation object of the aggregation function and the operation object is obtained after the obtained encapsulated data packet is decapsulated, and a specific aggregation function operation is executed on the current value of the operation object of the aggregation function according to the functional relationship between the operation object of the aggregation function. The aggregation function operation may be one or more of the aggregation functions listed above.
The embodiment of the invention discloses a real-time task processing method, which packages, encapsulates and stores an operation object of an aggregation function of a streaming processing task and a function relation of the operation object, directly obtains the stored operation object of the aggregation function of the streaming processing task and the function relation of the operation object when executing the same aggregation function of the streaming processing task in real time, and provides a method for encapsulating and caching the operation object of the same operation function and the logic relation of the function relation of the operation object. The data read-write interface does not need to be repeatedly written by a developer for the function interface with the same problem every time, the time for processing the real-time data with the same operation is reduced, and the overall development efficiency is improved. The embodiment of the invention finally realizes the state management of the real-time data processing process, conveniently performs the updating operation and improves the development efficiency of the real-time task.
Optionally, in an embodiment of the real-time task processing method according to the embodiment of the present invention, before loading the encapsulated packet of the aggregation function executed by the streaming processing task when the streaming processing task currently executes the aggregation function, the method further includes an encapsulated packet implementation process of the real-time task processing method, as shown in fig. 2. Fig. 2 is a flowchart of an implementation of an encapsulated data packet of a real-time task processing method according to an embodiment of the present invention, where the flowchart includes:
s201, when the stream processing task executes the aggregation function for the first time, executing the aggregation function operation of the stream processing task according to the functional relation of the operation objects on the numerical values of the operation objects of the aggregation function;
specifically, when the streaming processing task executes the aggregation function for the first time, according to the written logical relationship of the aggregation function, the operation object for executing the aggregation function for the first time and the functional relationship of the operation object of the aggregation function are obtained, the numerical value of the operation object is obtained in the database, and the operation of the aggregation function is executed on the numerical value of the operation object according to the functional relationship of the operation object of the aggregation function.
For example, when the streaming processing task executes the average value in a website service designation group for the first time, the first written logical relationship function is used to obtain the value of the website service in the database, and the averaging operation is performed on the value of the website service according to the logical relationship function.
S202, when the stream processing task finishes the first execution of the aggregation function, encapsulating the operation object of the aggregation function executed by the stream processing task and the functional relation of the operation object in an encapsulation data packet;
specifically, when the streaming processing task is finished when the aggregation function is executed for the first time, the operation object and the function relationship of the operation object of the aggregation function executed by the streaming processing task form an organic whole, the data and the source code of the operation data are organically combined to form a class, and finally the function relationship of the operation object and the operation object of the aggregation function executed by the streaming processing task is encapsulated into an encapsulation data packet.
For example, when the streaming processing task finishes executing the average value in a specified group of the website service for the first time, the operation object of the written logic relation function for executing the average value on the numerical value of the website service and the function relation of the operation object are encapsulated in the encapsulation data packet.
S203, storing the encapsulated data packet.
Specifically, the formed encapsulated data packet of the aggregation function is stored in a database, where the database may be a user-defined database storing the logical relationship of the aggregation function, and may also be commonly used MySQL, Redis, and the like, which may be determined according to the implementation situation.
It can be seen that, in an embodiment of the real-time task processing method according to the embodiment of the present invention, some common aggregation function operations are encapsulated, so that when a streaming processing task executes the same aggregation function in real time, external data does not need to be manually controlled, and when the same aggregation function operation is used next time, the encapsulated data packet is directly obtained, and a corresponding aggregation function operation is executed. The method for packaging and caching the operation objects of the same operation function and the logic relation of the function relation of the operation objects is provided, so that each streaming processing task has a corresponding state associated with the same aggregation function when executing the same aggregation function in real time, and the packaged data packet and the result storage of the streaming processing task executing the same aggregation function in real time are provided, so that the real-time streaming processing task can be conveniently searched and all states of the streaming processing task can be conveniently maintained.
Optionally, in an embodiment of the real-time task processing method in the embodiment of the present invention, after performing, according to a functional relationship of an operation object, an aggregation function operation of a streaming processing task on a numerical value of the operation object of the aggregation function, the method further includes:
and executing the aggregation function operation of the streaming processing task on the numerical value of the operation object of the aggregation function according to the functional relation of the operation object to obtain the result of the numerical value of the operation object, and storing the result in the database.
Specifically, when the streaming processing task is finished when the aggregation function is executed for the first time, the aggregation function operation of the streaming processing task is executed according to the functional relationship of the operation objects on the operation object numerical values obtained in the database, and the operated result is stored in the database.
For example, when a streaming processing task executes a maximum value of specified data of a website service for the first time, the maximum value of the website service at a time period or a time point is acquired in the database by using a first written logical relation function, the acquired maximum value is input, and the acquired maximum value is saved in the database again. When the maximum value of the website service designated data is executed at the next time period or time point, the maximum value of the website service designated data at the total time period or time point can be obtained only by searching the maximum value of the time period from the end of the maximum value of the website service designated data executed for the first time by the streaming processing task to the current time point, and then comparing the stored maximum values of the website service designated data executed for the first time.
It can be seen that, in an embodiment of the real-time task processing method according to the embodiment of the present invention, the result of obtaining the value of the operation object is stored in the database by executing the aggregation function operation of the streaming processing task according to the functional relationship of the operation object on the value of the aggregation function, when the streaming processing task executes the aggregation function again, the result of obtaining the value of the operation object, which is executed by the streaming processing task for the first time, can be directly obtained, and the streaming processing task can end as a time point when the aggregation function is executed for the first time, and calculate data of a time period from the time point to the time point when the streaming processing task executes the aggregation function again, so as to reduce the time for processing the data and improve the overall efficiency of the real-time task processing method. In addition, the aggregation function operation of the streaming processing task is executed according to the functional relation of the operation object, the result of the numerical value of the operation object is obtained and stored in the database, and the storage method of various data states of real-time task processing is realized.
Optionally, in an embodiment of the real-time task processing method according to the embodiment of the present invention, before performing, according to the aggregation function and the encapsulation data packet, an aggregation function operation on a current numerical value of an operation object of the aggregation function according to a functional relationship of the operation object, the method further includes:
accumulating the updating data of the operation object from the time period from the time when the streaming processing task executes the aggregation function for the first time to the time when the streaming processing task executes the aggregation function for the current time, and accumulating the updating data to the value of the operation object from the time when the streaming processing task executes the aggregation function for the first time to obtain the current value of the operation object of the aggregation function currently executed by the streaming processing task;
specifically, a time period is obtained by taking the aggregation function executed by the streaming processing task for the first time as a starting point and taking the aggregation function currently executed by the streaming processing task as an ending point. And accumulating the update data of the operation object of the attrition processing task in the time period in the database to obtain the update data volume of the operation object in the time period. If the data of the aggregation function executed by the streaming processing task in the time period is in an increasing form, the numerical value of the operation object of the streaming processing task for executing the aggregation function for the first time is taken as a base number, and the obtained updated data quantity of the operation object of the current time period for executing the aggregation function is accumulated to the numerical value of the operation object of the streaming processing task for executing the aggregation function for the first time, so that the current numerical value of the operation object of the current time period for executing the aggregation function by the streaming processing task is obtained. If the data of the aggregation function executed by the streaming processing task in the time period is in a descending form, the numerical value of the operation object of the aggregation function executed by the streaming processing task for the first time is subtracted from the obtained updated data quantity of the operation object of the current aggregation function execution time period, and the current numerical value of the operation object of the aggregation function executed by the streaming processing task at present is obtained.
For example, when some index needs to be aggregated in the streaming processing task, for example, the error keywords in the text data stream are scanned and summed to count the sum of the error keywords occurring in the system in real time. And taking the sum of the error keywords generated in the statistical system for the first execution of the streaming processing task as a starting point, and taking the sum of the error keywords generated in the statistical system for the current execution of the streaming processing task as an ending point to obtain a time period. And accumulating the updating data of the error keywords generated in the system in the time period to obtain the updating data volume of the error keywords generated in the system in the time period. And accumulating the updated data quantity of the error keywords generated in the system of the streaming processing task in the time period to the sum of the error keywords generated in the system of the streaming processing task for the first time to obtain the current numerical value of the error keywords generated in the system of the streaming processing task for the current time.
Loading an encapsulation data packet of an aggregation function of a streaming processing task and a current numerical value of an operation object;
specifically, an encapsulated data packet of the aggregation function stored in the database and the current value of the operation object are loaded into the real-time task executor.
For example, when the error keyword sum occurring in the real-time statistical system is counted, the package data packet of the error keyword sum occurring in the statistical system is executed for the first time by the streaming processing task, and the obtained current value of the error keyword occurring in the statistical system currently executed by the streaming processing task is loaded into the real-time task executor.
Correspondingly, according to the operation object of the aggregation function in the encapsulated data packet and the functional relationship of the operation object, executing corresponding operation on the current numerical value of the operation object of the aggregation function according to the functional relationship of the operation object, including:
acquiring an operation object of an aggregation function in a packaged data packet and a functional relation of the operation object through the packaged data packet;
specifically, the loaded encapsulated data packet is decapsulated, and a function relationship between an operation object and an operation object of the aggregation function in the encapsulated data packet is obtained.
For example, when the error keyword sum occurring in the system is counted in real time, the loaded streaming processing task first executes decapsulation of the encapsulated packet of the error keyword sum occurring in the statistical system, obtains an operation object of the error keyword occurring in the statistical system within the encapsulated packet of the error keyword sum occurring in the execution statistical system, and obtains a statistical function of the operation object of the error keyword occurring in the statistical system.
And executing the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the functional relation of the operation object.
Specifically, the operation object of the streaming processing task aggregation function and the functional relationship of the operation object are obtained, and the aggregation function operation is executed on the current numerical value of the operation object of the aggregation function according to the functional relationship of the operation object of the streaming processing task aggregation function.
For example, when the sum of error keywords occurring in the real-time statistics system is obtained, after the operation object of the error keyword occurring in the statistics system, the statistical function for the operation object of the error keyword occurring in the statistics system, and the current value of the error keyword occurring in the streaming processing task currently executing the statistics system are obtained, the current value of the error keyword occurring in the currently executing statistics system is calculated according to the statistical function for the operation object of the error keyword occurring in the statistics system. And executing statistical operation to obtain the sum of error keywords generated in the current system.
Therefore, the updating data of the operation object from the time period from the time when the streaming processing task executes the aggregation function for the first time to the time when the streaming processing task executes the aggregation function for the current time are accumulated, the updating data are accumulated to the numerical value of the operation object from the time when the streaming processing task executes the aggregation function for the first time, the current numerical value of the operation object from the time when the streaming processing task executes the aggregation function for the current time is obtained, the data updating amount of the whole time period is convenient to obtain, only the updating data of the operation object from the time period from the time when the streaming processing task executes the aggregation function for the first time to the time when the streaming processing task executes the aggregation function for the current time are needed to be calculated, the calculation.
Optionally, in an embodiment of the real-time task processing method according to the embodiment of the present invention, according to the aggregation function and the encapsulation data packet, for the current numerical value of the operation object of the aggregation function, the aggregation function operation is executed according to the functional relationship of the operation object, as shown in fig. 3. Fig. 3 is a flowchart of an operation of executing an aggregation function of a real-time task processing method according to an embodiment of the present invention, including:
s301, acquiring an operation object of an aggregation function in a packaged data packet and a functional relation of the operation object through the packaged data packet, and acquiring a current numerical value of the operation object of the aggregation function in a database;
specifically, the operation object of the aggregation function in the encapsulated packet and the functional relationship of the operation object are obtained by decapsulating the encapsulated packet loaded into the executor. And obtaining the current value of the operation object of the aggregation function currently executed by the streaming processing task in the database.
For example, when the error keyword sum occurring in the real-time statistics system is counted, the encapsulated data packet of the error keyword sum occurring in the statistics system executed for the first time by the streaming processing task is loaded into the real-time task executor, the encapsulated data packet of the error keyword sum occurring in the statistics system executed for the first time by the streaming processing task is unpacked in the real-time task executor, the operation object of the error keyword occurring in the statistics system in the encapsulated data packet is obtained, and the statistical function of the operation object of the error keyword occurring in the statistics system is obtained. And acquiring the current numerical value of the operation object of the error keyword generated in the statistical system in the database.
S302, the current numerical value of the operation object of the aggregation function is operated according to the functional relation of the operation object.
Specifically, after obtaining the operation object of the aggregation function, the functional relationship of the operation object, and the current value of the operation object of the aggregation function in the encapsulated data packet, the aggregation function operation is performed on the current value of the operation object of the aggregation function according to the functional relationship of the operation object.
For example, when the sum of error keywords occurring in the system is counted in real time, the above-obtained statistical function of the operation object of the error keyword occurring in the statistical system, the statistical function of the operation object of the error keyword occurring in the statistical system, and the current value of the operation object of the error keyword occurring in the statistical system are counted, and then the sum of error keywords occurring in the system in the time period is executed on the current value of the operation object of the error keyword occurring in the statistical system according to the statistical functional relationship of the operation object of the error keyword occurring in the statistical system.
Therefore, by acquiring the aggregation function logical relationship of the encapsulated data packet, acquiring the current numerical value of the operation object of the current aggregation function in the database, and executing the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the functional relationship of the operation object, a user does not need to write a program for processing the streaming task aggregation function in real time, and only needs to acquire the current updated data in the database and call the aggregation function logical relationship to execute the aggregation function, so that the development efficiency of real-time computing operation is improved.
In an embodiment of the real-time task processing method according to the embodiment of the present invention, the real-time task processing method according to the present invention is described in detail by taking an example of scanning error keywords in a text data stream, summing the error keywords, and counting the sum of error keywords occurring in a certain system in real time. The method comprises the following concrete steps:
step one, when the stream type processing task executes the aggregation function for the first time, the aggregation function operation of the stream type processing task is executed according to the functional relation of the operation objects for the numerical value of the operation object of the aggregation function.
When the sum of the error keywords occurring in a certain system is counted in real time corresponding to the first execution of the streaming processing task, the following steps are specifically performed:
when the streaming processing task executes the error keyword sum occurring in the statistical system for the first time, connecting an external storage database Redis, and acquiring a numerical value of an index in the database according to the name of an aggregation function executing the error keyword sum occurring in the system, for example, a counter ("error _ num") is the index of an operation "error _ num". The value of the operand of the error key occurring in the system in the database is operated according to the statistical function of the operand of the error key occurring in the statistical system, for example: , + means an add operation on the value of the operand of the error key occurred in the system, -means a subtract operation on the value of the operand of the error key occurred in the system. And obtaining the result of the sum of the error keywords generated in the system when the streaming processing task executes for the first time.
And step two, when the stream type processing task finishes the first execution of the aggregation function, encapsulating the operation object of the aggregation function executed by the stream type processing task and the functional relation of the operation object in an encapsulation data packet.
When the sum of the error keywords occurring in a certain system is counted in real time corresponding to the first execution of the streaming processing task, the following steps are specifically performed:
when the sum of the error keywords generated in the statistical system is executed for the first time by the streaming processing task, the operation object of the error keywords generated in the statistical system of the sum of the error keywords generated in the statistical system is executed for the first time by the streaming processing task, and the statistical function of the operation object of the error keywords generated in the statistical system is packaged to form an error keyword sum packaged data packet generated in the statistical system.
And step three, storing the encapsulated data packet.
When the sum of the error keywords occurring in a certain system is counted in real time corresponding to the first execution of the streaming processing task, the following steps are specifically performed:
and performing the sum of error keywords generated in the statistical system for the first time by the streaming processing task, and packaging and storing the formed sum of error keywords generated in the statistical system into a data packet.
And step four, executing the aggregation function operation of the streaming processing task on the numerical value of the operation object of the aggregation function according to the functional relation of the operation object to obtain the result of the numerical value of the operation object, and storing the result in the database.
When the sum of the error keywords occurring in a certain system is counted in real time corresponding to the first execution of the streaming processing task, the following steps are specifically performed:
and storing the result of the sum of the error keywords generated in the system for the first execution of the streaming processing task in a database.
Step five, when the stream type processing task currently executes the aggregation function, loading the stored encapsulation data packet of the aggregation function executed by the stream type processing task, wherein the encapsulation data packet comprises: the operation object of the aggregation function of the streaming processing task and the functional relation of the operation object.
When the sum of the error keywords generated in a certain system is counted in real time corresponding to the current execution of the streaming processing task, the following steps are specifically performed:
and when the streaming processing task currently executes the error keyword sum generated in the system, loading the stored error keyword sum generated in the statistical system to encapsulate the data packet.
And step six, executing the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the function relation of the operation object according to the aggregation function and the packaging data packet.
When the sum of the error keywords generated in a certain system is counted in real time corresponding to the current execution of the streaming processing task, the following steps are specifically performed:
and encapsulating a data packet according to the function of the error keyword sum generated in the execution statistical system and the error keyword sum generated in the statistical system, and executing the error keyword sum operation currently generated in the statistical system according to the statistical function of the operation object of the error keyword generated in the statistical system in the data packet for the current value of the operation object of the error keyword generated in the system in the database.
In order to achieve the above object, an embodiment of the present invention further discloses a real-time task processing device, as shown in fig. 4. Fig. 4 is a diagram of a real-time task processing device according to an embodiment of the present invention, including:
a state loading module 401, configured to load, when the streaming processing task currently executes the aggregation function, a stored encapsulated data packet of the aggregation function executed by the streaming processing task, where the encapsulated data packet includes: the operation object of the aggregation function of the streaming processing task and the function relation of the operation object;
and the execution module 402 is configured to execute the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the functional relationship of the operation object, according to the aggregation function and the encapsulation data packet.
The embodiment of the invention discloses a real-time task processing device, which packages, encapsulates and stores an operation object of an aggregation function of a streaming processing task and a function relation of the operation object, directly acquires the stored operation object of the aggregation function of the streaming processing task and the function relation of the operation object when executing the same aggregation function of the streaming processing task in real time, and provides a method for encapsulating and caching the operation object of the same operation function and the logic relation of the function relation of the operation object. The data read-write interface does not need to be repeatedly written by a developer for the function interface with the same problem every time, the time for processing the real-time data with the same operation is reduced, and the overall development efficiency is improved. The embodiment of the invention finally realizes the state management of the real-time data processing process, conveniently performs the updating operation and improves the development efficiency of the real-time task.
It should be noted that, the apparatus according to the embodiment of the present invention is an apparatus applying the real-time task processing method, and all embodiments of the real-time task processing method are applicable to the apparatus and can achieve the same or similar beneficial effects.
Optionally, in an embodiment of the real-time task processing device in the embodiment of the present invention, the real-time task processing device further includes:
the first execution module is used for executing the aggregation function operation of the streaming processing task according to the functional relation of the operation objects to the numerical value of the operation object of the aggregation function when the streaming processing task executes the aggregation function for the first time;
the encapsulation module is used for encapsulating the operation object of the aggregation function executed by the streaming processing task and the functional relation of the operation object into an encapsulation data packet when the streaming processing task finishes executing the aggregation function for the first time;
and the first storage module is used for storing the encapsulated data packet.
Optionally, in an embodiment of the real-time task processing device in the embodiment of the present invention, the real-time task processing device further includes:
and the second storage module is used for executing the aggregation function operation of the streaming processing task on the numerical value of the operation object of the aggregation function according to the functional relation of the operation object to obtain the result of the numerical value of the operation object, and storing the result in the database.
Optionally, in an embodiment of the real-time task processing device in the embodiment of the present invention, the real-time task processing device further includes:
the accumulation module is used for accumulating the update data of the operation object from the time period from the time when the streaming processing task executes the aggregation function for the first time to the time when the streaming processing task executes the aggregation function for the current time, and accumulating the update data to the value of the operation object from the time when the streaming processing task executes the aggregation function for the first time to obtain the current value of the operation object of the aggregation function currently executed by the streaming processing task;
the data packet loading module is used for loading an encapsulated data packet of an aggregation function of the streaming processing task and a current numerical value of an operation object;
accordingly, module 402 is executed, comprising:
the obtaining submodule is used for obtaining the operation object of the aggregation function in the encapsulated data packet and the functional relation of the operation object through the encapsulated data packet;
and the execution submodule is used for executing the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the functional relation of the operation object.
Optionally, in an embodiment of the real-time task processing device according to the embodiment of the present invention, the executing module 402 is further configured to obtain, by encapsulating the data packet, an operation object of the aggregation function in the encapsulated data packet and a function relationship of the operation object, and obtain, in the database, a current numerical value of the operation object of the aggregation function; and executing the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the functional relation of the operation object.
In one embodiment of the real-time task processing device of the present invention, as shown in fig. 5. Fig. 5 is a diagram of another real-time task processing device according to an embodiment of the present invention, including:
the first executing module 501 is configured to, when the streaming processing task executes the aggregation function for the first time, execute the aggregation function operation of the streaming processing task on the numerical value of the operation object of the aggregation function according to the functional relationship of the operation object.
The encapsulating module 502 is configured to encapsulate, when the streaming processing task is finished when the aggregation function is executed for the first time, the operation object of the aggregation function executed by the streaming processing task and the functional relationship of the operation object in an encapsulation data packet.
The first storage module 503 is configured to store the encapsulated data packet.
The second storage module 504 is configured to execute the aggregation function operation of the streaming processing task on the numerical value of the operation object of the aggregation function according to the functional relationship of the operation object, obtain a result of the numerical value of the operation object, and store the result in the database.
A state loading module 505, configured to load, when the streaming processing task currently executes the aggregation function, a stored encapsulation data packet of the aggregation function executed by the streaming processing task, where the encapsulation data packet includes: the operation object of the aggregation function of the streaming processing task and the functional relation of the operation object.
And an executing module 506, configured to execute, according to the aggregation function and the encapsulation data packet, an aggregation function operation on the current numerical value of the operation object of the aggregation function according to the functional relationship of the operation object.
In an embodiment of the real-time task processing device according to the embodiment of the present invention, the real-time task processing device according to the present invention is described in detail by taking an example of scanning error keywords in a text data stream, summing the error keywords, and counting the sum of error keywords occurring in a certain system in real time, as shown in fig. 6. Fig. 6 is a diagram of an embodiment of a real-time task processing device.
In fig. 5, a first executing module 501 is configured to, when the streaming processing task executes the aggregation function for the first time, execute the aggregation function operation of the streaming processing task according to the functional relationship of the operation objects on the numerical value of the operation object of the aggregation function.
And setting a real-time task executor to execute when the sum of error keywords generated in a certain system is counted in real time corresponding to the first execution of the streaming processing task. The method specifically comprises the following steps:
and the real-time task executor is used for connecting an external storage database Redis when the streaming processing task executes the error keyword sum generated in the statistical system for the first time, and acquiring the value of the index in the database according to the name of an aggregation function for executing the error keyword sum generated in the system, for example, the counter (error _ num) is the index of operation 'error _ num'. The value of the operand of the error key occurring in the system in the database is operated according to the statistical function of the operand of the error key occurring in the statistical system, for example: , + means an add operation on the value of the operand of the error key occurred in the system, -means a subtract operation on the value of the operand of the error key occurred in the system. And obtaining the result of the sum of the error keywords generated in the system when the streaming processing task executes for the first time.
The encapsulating module 502 is configured to encapsulate, when the streaming processing task is finished when the aggregation function is executed for the first time, the operation object of the aggregation function executed by the streaming processing task and the functional relationship of the operation object in an encapsulation data packet.
When the sum of the error keywords occurring in a certain system is counted in real time corresponding to the first execution of the streaming processing task, the following steps are specifically performed:
and the real-time task executor is used for encapsulating the operation object of the error keyword generated in the statistical system of the error keyword sum generated in the statistical system for the first execution of the streaming processing task and the statistical function of the operation object of the error keyword generated in the statistical system to form an error keyword sum encapsulated data packet generated in the statistical system when the error keyword sum generated in the statistical system for the first execution of the streaming processing task is finished.
The first storage module 503 is configured to store the encapsulated data packet.
When the sum of the error keywords occurring in a certain system is counted in real time corresponding to the first execution of the streaming processing task, the following steps are specifically performed:
and the state metadata information storage database is used for carrying out the error keyword sum generated in the statistical system for the first time by the streaming processing task and forming an error keyword sum generated in the statistical system to encapsulate the data packet for storage.
The second storage module 504 is configured to execute the aggregation function operation of the streaming processing task on the numerical value of the operation object of the aggregation function according to the functional relationship of the operation object, obtain a result of the numerical value of the operation object, and store the result in the database.
When the sum of the error keywords occurring in a certain system is counted in real time corresponding to the first execution of the streaming processing task, the following steps are specifically performed:
and the external storage database is used for storing the result of the sum of the error keywords generated in the system for executing the streaming processing task for the first time in the database.
A state loading module 505, configured to load, when the streaming processing task currently executes the aggregation function, a stored encapsulation data packet of the aggregation function executed by the streaming processing task, where the encapsulation data packet includes: the operation object of the aggregation function of the streaming processing task and the functional relation of the operation object.
When the sum of the error keywords generated in a certain system is counted in real time corresponding to the current execution of the streaming processing task, the following steps are specifically performed:
and the state manager is used for loading the stored error key sum packaging data packet generated in the statistical system when the streaming processing task executes the error key sum generated in the system again.
And an executing module 506, configured to execute, according to the aggregation function and the encapsulation data packet, an aggregation function operation on the current numerical value of the operation object of the aggregation function according to the functional relationship of the operation object.
When the sum of the error keywords generated in a certain system is counted in real time corresponding to the current execution of the streaming processing task, the following steps are specifically performed:
and the task executor is used for encapsulating a data packet according to the function of the error keyword sum generated in the execution statistical system and the error keyword sum generated in the statistical system, and executing the current error keyword sum operation in the statistical system according to the statistical function of the operation object of the error keyword generated in the statistical system in the data packet for the current value of the operation object of the error keyword generated in the system in the database.
It should be noted that an implementation diagram of a real-time task processing device according to an embodiment of the present invention is only one of the real-time task processing devices disclosed in the embodiments of the present invention, and any executor or execution module having the same functions as the first execution module 501, the encapsulation module 502, the first storage module 503, the second storage module 504, the status loading module 505, and the execution module 506 or having the idea of the real-time task processing method according to the embodiments of the present invention belongs to the protection scope of the present invention.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims (10)

1. A real-time task processing method, comprising:
when a streaming processing task currently executes an aggregation function, loading a stored encapsulation data packet of the aggregation function executed by the streaming processing task, wherein the encapsulation data packet comprises: an operation object of the aggregation function of the streaming processing task, a functional relationship of the operation object;
and executing the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the aggregation function and the encapsulation data packet and the functional relation of the operation object.
2. The real-time task processing method according to claim 1, wherein before loading the stored encapsulation packet of the aggregation function executed by the streaming processing task when the streaming processing task currently executes the aggregation function, the method further comprises:
when the streaming processing task executes the aggregation function for the first time, executing the aggregation function operation of the streaming processing task according to the functional relationship of the operation objects on the numerical values of the operation objects of the aggregation function;
when the streaming processing task finishes executing the aggregation function for the first time, encapsulating the operation object of the aggregation function executed by the streaming processing task and the functional relationship of the operation object into the encapsulation data packet;
and storing the encapsulated data packet.
3. A real-time task processing method according to claim 2, wherein after the aggregation function operation of the streaming processing task is performed on the numerical value of the operand of the aggregation function according to the functional relationship of the operand, the method further comprises:
and executing the aggregation function operation of the streaming processing task on the numerical value of the operation object of the aggregation function according to the functional relation of the operation object to obtain a result of the numerical value of the operation object, and storing the result in a database.
4. A real-time task processing method according to claim 1 or 2, wherein before the performing the aggregation function operation on the current value of the operand of the aggregation function according to the functional relationship of the operand according to the aggregation function and the encapsulation packet, the method further comprises:
accumulating the update data of the operation object from the time when the streaming processing task executes the aggregation function for the first time to the time when the streaming processing task executes the aggregation function at present, and accumulating the update data to the value of the operation object from the time when the streaming processing task executes the aggregation function for the first time to obtain the current value of the operation object from which the streaming processing task executes the aggregation function at present;
loading the encapsulation data packet of the aggregation function of the streaming processing task and the current value of the operand;
correspondingly, the executing, according to the operation object of the aggregation function in the encapsulated data packet and the functional relationship of the operation object, a corresponding operation on the current value of the operation object of the aggregation function according to the functional relationship of the operation object includes:
acquiring an operation object of the aggregation function in the encapsulated data packet and a function relation of the operation object through the encapsulated data packet;
and executing the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the functional relation of the operation object.
5. The real-time task processing method according to claim 1, wherein the performing, according to the aggregation function and the encapsulation packet, the aggregation function operation on the current value of the operation object of the aggregation function according to the functional relationship of the operation object includes:
acquiring an operation object of the aggregation function and a function relation of the operation object in the encapsulated data packet through the encapsulated data packet, and acquiring a current numerical value of the operation object of the aggregation function in a database;
and executing the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the functional relation of the operation object.
6. A real-time task processing apparatus, comprising:
a state loading module, configured to load a stored encapsulated packet of an aggregation function executed by a streaming processing task when the streaming processing task currently executes the aggregation function, where the encapsulated packet includes: an operation object of the aggregation function of the streaming processing task, a functional relationship of the operation object;
and the execution module is used for executing the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the aggregation function and the encapsulation data packet and the functional relation of the operation object.
7. The real-time task processing device according to claim 6, wherein the device further comprises:
a first execution module, configured to execute, when the streaming processing task executes the aggregation function for the first time, the aggregation function operation of the streaming processing task according to a functional relationship of the operation object on the numerical value of the operation object of the aggregation function;
an encapsulation module, configured to encapsulate, when the streaming processing task finishes executing the aggregation function for the first time, the operation object of the aggregation function executed by the streaming processing task and the functional relationship of the operation object in the encapsulation data packet;
and the first storage module is used for storing the encapsulated data packet.
8. The real-time task processing device of claim 7, further comprising:
and the second storage module is used for executing the aggregation function operation of the streaming processing task on the numerical value of the operation object of the aggregation function according to the functional relation of the operation object to obtain a result of the numerical value of the operation object, and storing the result in a database.
9. A real-time task processing device according to claim 6 or 7, characterized in that said device further comprises:
an accumulation module, configured to accumulate update data of the operation object from when the streaming processing task executes the aggregation function for the first time to when the streaming processing task executes the aggregation function for the current time, and accumulate the update data to the value of the operation object from when the streaming processing task executes the aggregation function for the first time, so as to obtain a current value of the operation object from which the streaming processing task executes the aggregation function for the current time;
a data packet loading module, configured to load the encapsulated data packet of the aggregation function of the streaming processing task and the current value of the operation object;
correspondingly, the execution module includes:
the obtaining submodule is used for obtaining the operation object of the aggregation function in the encapsulated data packet and the functional relation of the operation object through the encapsulated data packet;
and the execution submodule is used for executing the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the functional relation of the operation object.
10. The real-time task processing device according to claim 6, wherein the execution module is further configured to obtain, through the encapsulated data packet, the operation object of the aggregation function in the encapsulated data packet, the functional relationship of the operation object, and obtain, in a database, the current value of the operation object of the aggregation function; and executing the aggregation function operation on the current numerical value of the operation object of the aggregation function according to the functional relation of the operation object.
CN201710210712.XA 2017-03-31 2017-03-31 Real-time task processing method and device Active CN107168681B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710210712.XA CN107168681B (en) 2017-03-31 2017-03-31 Real-time task processing method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710210712.XA CN107168681B (en) 2017-03-31 2017-03-31 Real-time task processing method and device

Publications (2)

Publication Number Publication Date
CN107168681A CN107168681A (en) 2017-09-15
CN107168681B true CN107168681B (en) 2020-11-06

Family

ID=59849803

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710210712.XA Active CN107168681B (en) 2017-03-31 2017-03-31 Real-time task processing method and device

Country Status (1)

Country Link
CN (1) CN107168681B (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103152126A (en) * 2012-04-06 2013-06-12 中国科学技术大学 Data packaging method and device based on forward error correction protection codes
CN104615777A (en) * 2015-02-27 2015-05-13 浪潮集团有限公司 Method and device for real-time data processing based on stream-oriented calculation engine
CN105608219A (en) * 2016-01-07 2016-05-25 上海通创信息技术有限公司 Stream-oriented recommended engine, recommendation system and recommendation method based on clustering

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020029207A1 (en) * 2000-02-28 2002-03-07 Hyperroll, Inc. Data aggregation server for managing a multi-dimensional database and database management system having data aggregation server integrated therein
US8204938B2 (en) * 2003-02-14 2012-06-19 Devereux Research Ab Llc System and method for immediate and delayed real-time communication activities using availability data from and communications through an external instant messaging system
CN102799423B (en) * 2011-05-27 2015-07-29 深圳市金蝶中间件有限公司 Method and the device of dynamic approach is performed in JSF
CN106708487A (en) * 2015-11-16 2017-05-24 北京国双科技有限公司 Data presenting method and device
CN106506266B (en) * 2016-11-01 2019-05-14 中国人民解放军91655部队 Network flow analysis method based on GPU, Hadoop/Spark mixing Computational frame

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103152126A (en) * 2012-04-06 2013-06-12 中国科学技术大学 Data packaging method and device based on forward error correction protection codes
CN104615777A (en) * 2015-02-27 2015-05-13 浪潮集团有限公司 Method and device for real-time data processing based on stream-oriented calculation engine
CN105608219A (en) * 2016-01-07 2016-05-25 上海通创信息技术有限公司 Stream-oriented recommended engine, recommendation system and recommendation method based on clustering

Also Published As

Publication number Publication date
CN107168681A (en) 2017-09-15

Similar Documents

Publication Publication Date Title
US11216302B2 (en) Modifying task dependencies at worker nodes using precompiled libraries
US11182204B2 (en) System and method for batch evaluation programs
CN110795455A (en) Dependency relationship analysis method, electronic device, computer device and readable storage medium
CN111666296A (en) SQL data real-time processing method and device based on Flink, computer equipment and medium
US20200250182A1 (en) Per-statement monitoring in a database environment
US20160012110A1 (en) General and automatic approach to incrementally computing sliding window aggregates in streaming applications
CN108037967A (en) A kind of menu loading method and electronic equipment based on more parent-child structures
EP3384385B1 (en) Methods and systems for mapping object oriented/functional languages to database languages
CN113986424A (en) Page display method and device
US20180260260A1 (en) In-memory data analytic system that provides an integrated tracking mechanism for explicit memory resources
US10534774B2 (en) Query performance degradation analysis timing
CN117668050A (en) Cross-data-source hybrid engine query method, system, equipment and medium
CN107168681B (en) Real-time task processing method and device
CN102685590B (en) Method and device for managing set top box memory
CN112052035A (en) Version packaging method and device based on bank back-line system
CN116501744B (en) Method, device and computer equipment for automatic table creation and storage of simulation data
CN111767252A (en) Log query method, device, computer equipment and storage medium
CN103678485A (en) Virtual experiment process node driving and movable packaging system
CN116028108B (en) Method, device, equipment and storage medium for analyzing dependent package installation time
CN111814041A (en) A recommended method, apparatus, storage medium and computer equipment for an NPM package
CN107506299B (en) Code analysis method and terminal equipment
CN115269495B (en) Business scheme metadata processing method and system based on aPaaS platform
CN115905571A (en) Generation method and device of purchasing knowledge graph
CN109542893B (en) Composite data structure setting method based on programming language and electronic equipment
Ventocilla Big data programming with Apache spark

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant