WO2020073687A1 - Columnar storage method and apparatus for streaming data, device, and storage medium - Google Patents

Abstract

The present application relates to the field of streaming data storage, in particular, to a columnar storage method and apparatus for streaming data, a device, and a storage medium. The columnar storage method for streaming data comprises: reading data from a real-time message system to obtain data to be processed; parsing the data to be processed to obtain structured data; converting the structured data into data in Row format, and every time a group of structured data is converted into data in Row format, storing the data in Row format into a memory; and forming multiple rows of data in Row format stored in the memory into data in Dataset<Row> format, and writing same to a file system in a columnar storage format. According to the present application, streaming data in a real-time message system is processed by means of Spark Streaming, thereby solving the problem that streaming data in a real-time message system cannot be stored in a columnar storage format at present, greatly improving the speed of subsequent processing of a large amount of data, and reducing the time for converting a row storage structure into a columnar storage structure.

Description

Streaming data column storage method, device, equipment and storage medium

This application requires the priority of the Chinese patent application submitted to the China Patent Office on October 11, 2018, with the application number 201811182661.5 and the invention titled "Streaming Data Column Storage Method, Apparatus, Equipment, and Storage Media", all of which are approved by The reference is incorporated in the application.

Technical field

The present application relates to the field of streaming data storage, and in particular to a method, device, device, and storage medium for streaming data column storage.

Background technique

In recent years, with the rapid development of the Internet, the rapid growth of data has become a common opportunity and challenge faced by many industries. In today's network environment, a large number of data sources are real-time and uninterrupted, and the response time to users is also required to be real-time. These data are collected, calculated and queried in a streaming form, such as a real-time messaging system, and all incoming data are processed in a streaming manner. It has various and massive network data inflows at all times, with different inflow speeds, and complex and diverse data structures, including binary files, text files, and compressed files. For such systems, the underlying storage system needs to support: store incoming data in a unified format, provide a unified interface for upper-layer applications, facilitate retrieval, and have certain requirements for real-time performance.

In response to today's big data trends, a number of big data processing platforms have emerged, such as kafka and flume. Specifically, the front-end application streams the messages into the message queue, and then the message queue writes these data to the disk in some form, such as hdfs, or the local disk.

The inventor realized that due to the streaming form of the real-time messaging system, the messages are eventually written to disk in the form of line storage, such as json, or ordinary text. In big data processing, in many cases, the data needs to be stored in the form of column storage. At this time, traditional tools such as flume cannot meet the needs.

Summary of the invention

In view of this, it is necessary to provide a method and device for streaming data column storage for the data in the existing real-time messaging system is written to the file system in the form of row storage, not in the file system , Equipment and storage media.

A streaming data column storage method includes the following steps:

Read the data from the real-time messaging system to get the data to be processed;

Analyze the data to be processed to obtain structured data;

Convert the structured data to Row format data, each time a set of the structured data is converted to

After the Row format data is stored in the memory;

Combine multiple rows of Row format data stored in the memory into a Dataset <Row> format data,

Write to the file system in a column storage format.

In one of the embodiments, the reading data from the real-time messaging system to obtain the data to be processed includes:

Obtain the access authority of the real-time messaging system and connect to the real-time messaging system;

Set an execution cycle, and read data from the real-time messaging system according to the execution cycle.

In one of the embodiments, the parsing the data to be processed to obtain structured data includes judging the format of the data to be processed, and using different methods for parsing according to the judgment result, specifically including:

If the data to be processed is in json format, FastJSON is called to parse the data to be processed in json format into the structured data;

If the to-be-processed data is in the csv format, then according to the content of the to-be-processed data, and using the DataFrame () method to add structured information to the to-be-processed data in the csv format to obtain the structured data.

In one embodiment, the rows of the Row format data stored in the memory form a Dataset <Row> format data, and are written into the file system in a column storage format, including:

Form the Dataset <Row> format data in multiple rows of the Row format data through the data frame method;

Convert the Dataset <Row> format data to parquet format data through parquet (), and write the parquet format data to the file system using spark.read ().

In one of the embodiments, the setting an execution cycle, and reading data from the real-time messaging system according to the execution cycle includes:

Start reading from the location of the first piece of data in the real-time messaging system;

Receive the completed reading instruction, stop reading, and record the completed reading position;

Obtain the position where the last reading was completed, start reading from the position where the last reading was completed, until the instruction to complete the reading is received, stop reading, and record the position where the reading was completed.

In one of the embodiments, if the to-be-processed data is in json format, calling FastJSON to parse the to-be-processed data in json format into the structured data includes:

Extract the field information of the data to be processed in the json format;

Sort the data to be processed in the json format according to the field information to obtain the structured data.

In one of the embodiments, the rows of the Row format data stored in the memory form a Dataset <Row> format data, and after being written to the file system in a column storage format, the method further includes:

Divide the storage path according to the column information of the data to be processed;

The partitionBy () function is called, and the columns with the same column names in the data to be processed are stored in different directories according to different values in the columns.

A streaming data column storage device includes the following modules:

The data acquisition module is set to read data from the real-time messaging system to obtain data to be processed;

A data analysis module configured to analyze the data to be processed to obtain structured data;

The data conversion module is configured to convert the structured data into Row format data, and each time a group of the structured data is converted into Row format data, it is stored in the memory;

The data storage module is configured to compose multiple rows of Row format data stored in the memory into a Dataset <Row> format data, and write the data into a file system in a column storage format.

A computer device includes a memory and a processor, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by one or more of the processors, the one or more of the processors are executed The steps of the above streaming data column storage method.

A storage medium storing computer-readable instructions, when the computer-readable instructions are executed by one or more processors, causes the one or more processors to perform the steps of the above streaming data column storage method.

This application uses Sparking to process the streaming data in the real-time messaging system, which solves the problem that the streaming data in the real-time messaging system cannot be saved as a column storage format, which greatly improves the speed of subsequent processing of large amounts of data. It also saves the time to convert the row storage structure to the column storage structure. Using Spark Streaming as the computing framework, it greatly uses distributed computing to improve the conversion and storage performance.

BRIEF DESCRIPTION

The drawings are only for the purpose of showing the preferred embodiments, and are not considered to limit the present application.

FIG. 1 is an overall flowchart of a streaming data column storage method of this application;

2 is a schematic diagram of a data acquisition process in a streaming data column storage method of this application;

3 is a schematic diagram of a data parsing process in a streaming data column storage method of this application;

4 is a schematic diagram of a data storage process in a streaming data column storage method of this application;

FIG. 5 is a structural diagram of a streaming data column storage device of the present application.

detailed description

Those skilled in the art can understand that unless specifically stated, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the word "comprising" used in the specification of this application refers to the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, Integers, steps, operations, elements, components, and / or their groups.

FIG. 1 is an overall flowchart of a streaming data column storage method of the present application. As shown in FIG. 1, a streaming data column storage method includes the following steps:

Step S1: Read data from the real-time messaging system to obtain data to be processed.

Among them, a streaming data column storage device of the present application mainly includes the Spark Streaming program. The present application mainly relies on Spark Streaming to process the streaming data of the real-time messaging system, thereby realizing the conversion of the streaming data in the real-time messaging system into columns The storage format is written to the file system. The data in the real-time messaging system is streaming data, and the real-time messaging system is also a processing component of streaming data.

Among them, Spark Streaming includes data acquisition module, data analysis module, data conversion module and data storage module.

When the above steps are executed, one of the sub-modules of the data acquisition module issues a data acquisition instruction at regular intervals, and the other sub-module receives the data acquisition instruction. After receiving the data acquisition instruction, it executes the instruction and reads it from the real-time messaging system Data to get the data to be processed.

Step S2: Analyze the data to be processed to obtain structured data.

When the above steps are executed, the data acquisition module sends the obtained data to be processed to the data parsing module. The data parsing module parses the data to be processed, and uses different methods to parse according to different formats of the data to be processed. The data obtained from the real-time messaging system has a complex and diverse data structure, including binary files, text files, compressed files and other data in various formats. After receiving the data to be processed in different formats, the data parsing module adopts different methods for parsing, and finally parses the data to be processed into structured data, and then sends it to the data conversion module.

Step S3, the structured data is converted into Row format data, and each time a group of the structured data is converted into Row format data, it is stored in the memory.

When the above steps are performed, the data conversion module converts the structured data sent from the data analysis module to Row format data, and temporarily stores it in the data storage module.

In one of the preferred embodiments, the parsed structured data is converted into Row format data through spark.createRow ().

Among them, the Row format is a format that comes with Spark Streaming, and the Row format is a data structure with column information, which is essentially a row of data.

In step S4, a plurality of rows of Row format data stored in the memory are combined into a Dataset <Row> format data, and written into the file system in a column storage format.

When the above steps are executed, the Row format data converted by the data conversion module is temporarily stored in the data storage module, and every once in a while, the accumulated multi-row Row format data is composed into a Dataset <Row> format data, which is stored in a column format at a time Write to the file system.

Among them, the Dataset <Row> format is a format that comes with Spark Streaming. The Dataset <Row> format is a matrix composed of many row Row formats. It is an ordered collection of Row format data. Dataset <Row> is a column information structure. Converting Dataset <Row> format data to column storage format means that the data exists in the form of columns.

In this embodiment, the streaming data in the real-time messaging system is parsed through Spark Streaming, and each piece of data is converted into Row format data in Spark Streaming, and the multiple rows of Row format data are combined and temporarily placed in the data storage module , And then form the Dataset <Row> format one-time write file system, which solves the problem that current streaming data cannot be stored in columns. Using Spark Streaming as a computing framework, improves the data conversion and storage performance.

In one embodiment, FIG. 2 is a schematic diagram of a data acquisition process in a streaming data column storage method of the present application. As shown in FIG. 2, a data acquisition process of a streaming data column storage method includes the following steps :

Step S101: Acquire the access right of the real-time messaging system and connect to the real-time messaging system.

When the above steps are performed, the access rights of the real-time messaging system are obtained by using the user name and password of the remote connection authority, and connected to the real-time messaging system through the Hibernate object relationship mapping framework.

Step S102: Set an execution cycle, and read data from the real-time messaging system according to the execution cycle.

When the above steps are executed, the execution cycle of the Spark Streaming program is set, and the execution cycle is passed into the Spark Streaming program as the parameter value.

In one of the preferred embodiments, the execution cycle may also be written as a fixed value in the program and set in the configuration parameters of the Spark Streaming program.

In one of the preferred embodiments, the execution cycle may be set to be the same for each read time interval, or may be set to be different for the read time interval according to the speed of the data inflow rate of the real-time messaging system.

In this embodiment, the execution cycle is passed into the Spark Streaming program as a parameter value, which is more flexible. Writing the execution cycle as a fixed value in the program can ensure high value security, and the execution cycle can be based on the real-time messaging system data inflow The speed can be set flexibly.

In one embodiment, FIG. 3 is a schematic diagram of a data parsing process in a streaming data column storage method of the present application. As shown in FIG. 3, a data parsing process of a streaming data column storage method includes the following steps :

Step S201, if the data to be processed is in json format, FastJSON is called to parse the data to be processed in json format into the structured data.

When the above steps are performed, if the data obtained from the real-time messaging system is in json format, it is parsed using a related library. In one of the preferred embodiments, it is parsed using FastJSON.

Specifically, if the data obtained from the real-time messaging system is {"id": 0, "name": "Alice", "age": 21} json format data, its structure includes 3 fields, namely id, name and age represent id, name and age, respectively. After using Fastjson to parse it, it will be parsed into a structured data containing id, name, and age, and then the parsed data will be converted to Row format data that comes with Spark Streaming.

Step S202, if the data to be processed is in the csv format, then according to the content of the data to be processed, and using the DataFrame () method to add structured information to the data to be processed in the csv format to obtain the structured data.

Among them, unlike data in json and avro formats, data in csv format generally contains only data information, not structure information. As mentioned in the above step S201, {"id": 0, "name": "Alice", "age": 21} json format data, if it is in csv format, its data content is only 0, Alice, 21. In this format of data, the meaning of each column cannot be determined by the content of the data. It is necessary to set the first column as id, the second column as the name, and the third column as the age according to the user's perception of the data. The content adds structured information by itself, parses the data into structured data, and then converts the data to Row format data.

When the above steps are executed, the spark.createDataFrame (RowJavaRDD, type) method is used to add structured information to the data. Among them, RowJavaRDD refers to data information, and type is structural information.

In this embodiment, different analysis methods are used for data in different formats, so that the data is uniformly parsed into structured data, and then the data is converted into Row format data, which saves data processing time and improves the accuracy of data processing.

In one embodiment, FIG. 4 is a schematic diagram of a data storage process in a streaming data column storage method of the present application. As shown in FIG. 4, a data storage process of a streaming data column storage method includes the following steps :

In step S301, multiple rows of the Row format data are combined into the Dataset <Row> format data by a data frame method.

In one of the preferred embodiments, spark.createDataFrame (RowJavaRDD, type) is used to compose multiple rows of Row format data into Dataset <Row> format data, where RowJavaRDD represents data information and type represents structure information.

Step S302: Convert the Dataset <Row> format data to parquet format data through parquet (), and write the parquet format data to the file system using spark.read ().

When the above steps are executed, use spark.read (). Parquet (filename) to write Dataset <Row> format data to the file system in parquet format.

When the above steps are performed, if you want to store columns in parquet format, use parquet () to convert Dataset <Row> format data to parquet format. Specifically, use parquet (filename) to convert Dataset <Row> format data to parquet format. Parquet is a file format that supports columnar storage.

When the above steps are executed, use spark.read () to write the converted parquet format data to the file system.

In this step, data can also be written to the file system through other column storage formats.

The file system includes local files (file: //) and HDFS (hdfs: //), and can also include other file systems supported by other sparks, such as Amazon S3 (s3: //). It is generally formulated by the file name. For example, if you want to write to the data folder in the root directory of hdfs, you can set it to hdfs: /// data /.

In this embodiment, by using spark.createDataFrame (RowJavaRDD, type), multiple rows of Row format data are combined into a Dataset <Row> format data, so that the streaming data is converted into a column data structure, which lays a solid foundation for subsequent data column storage. Use spark.read () to write the data in the format of Dataset <Row> to the file system, and realize that the streaming data is written to the file system in the format of column storage.

In one embodiment, reading data from the real-time messaging system according to the execution cycle includes the following specific steps:

Start reading from the location of the first piece of data in the real-time messaging system.

Receive the read completed command, stop reading, and record the completed position.

Obtain the position where the last reading was completed, start reading from the position where the last reading was completed, until the instruction to complete the reading is received, stop reading, and record the position where the reading was completed.

When the above steps are executed, when the program is to read data for the first time, it will start reading from the location of the first piece of data in the real-time messaging system until the latest data generated during reading is read. At this time, the read will be received After the completed instruction, the reading is stopped, and Spark Streaming automatically records the position where the reading is completed.

Among them, the first reading data refers to the reading when the program is started for the first time. The Spark Streaming program runs permanently. If it is not suspended, it can continue to run. Because the data of the real-time messaging system is written continuously, every time the data is read, Spark Streaming records the location of each read for the next reading.

Each time the data is read in the future, the position where the last reading was completed is obtained, and the reading is started from the position where the last reading was completed, until the instruction to read the completion is received, the reading is stopped, and the position where the reading is completed is recorded.

In this embodiment, each time the reading is completed, the position where the reading is completed is recorded, which is convenient for the next reading, and is not easy to make mistakes, which improves the speed and quality of data acquisition.

In one embodiment, calling FastJSON to parse the to-be-processed data in the json format into the structured data includes the following specific steps:

Extract the field information of the data to be processed in the json format;

Sort the data to be processed in the json format according to the field information to obtain the structured data.

The data obtained from the real-time messaging system is {"age": 21, "id": 0, "name": "Alice",} json format data, using FastJSON to extract the field information of the data, respectively, age, id, and name represent age, id, and name, respectively. Then sort the data to be processed according to the field information. For example, if the sorted data structure is {"id", "name", "age"}, then {"id", "name", "age"} is the structure化 数据。 Data.

In one embodiment, the data conversion module decides whether to modify the parsed structured data as needed. If the storage needs to be stored according to the year, month and day, if the data obtained from the real-time messaging system contains a time stamp, such as "2017-09-21 08: 16: 05.011", and the storage needs to be stored according to the year, month and day, then The year, month, and day information in the time stamp needs to be extracted.

In one embodiment, the storage path may be divided according to the column information of the data to be processed, and the column with the same column name in the data to be processed may be stored in different columns according to different values in the column through the partitionBy () function table of Contents. When the above steps are executed, the storage path is divided by spark.read (). PartitionBy (), for example, fill in the parameters as newDf.write (). Mode (SaveMode.Append) .partitionBy ("stream", "year", "month "," day "," hour "). orc (" orc ") refers to the path segmentation based on the stream, year, month, and day fields.

PartitionBy is part of the analytical function. It differs from the aggregate function groupBy in that it can return multiple records in a group. The aggregate function generally has only one record that reflects the statistical value. partitionBy is used to group the result set. Specify that it treats the entire result set as a group, and partitionBy returns each piece of data in the group, and can sort the grouped data.

In this embodiment, the partitioning of the storage path is realized by using the partitionBy function, which facilitates the subsequent processing of a large amount of data.

A streaming data column storage device, as shown in FIG. 5, includes the following modules:

The data acquisition module is set to read data from the real-time messaging system to obtain data to be processed;

A data analysis module configured to analyze the data to be processed to obtain structured data;

The data conversion module is configured to convert the structured data into Row format data, and each time a group of the structured data is converted into Row format data, it is stored in the memory;

The data storage module is configured to compose multiple rows of Row format data stored in the memory into a Dataset <Row> format data, and write the data into a file system in a column storage format.

In one embodiment, a computer device is proposed, which includes a memory and a processor. The memory stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors are executed. The computer-readable instructions implement the steps of the streaming data column storage method described in the foregoing embodiments.

In one embodiment, a storage medium storing computer-readable instructions is provided. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the above-mentioned embodiments. Steps of streaming data column storage method. Wherein, the storage medium may be a non-volatile storage medium.

Those of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing relevant hardware through a program. The program may be stored in a computer-readable storage medium, and the storage medium may include: Read only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, etc.

The technical features of the above-mentioned embodiments can be arbitrarily combined. To simplify the description, all possible combinations of the technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, All should be considered within the scope of this description.

The above-mentioned embodiments only express some exemplary embodiments of the present application, and their descriptions are more specific and detailed, but they should not be construed as limiting the patent scope of the present application. It should be noted that, for those of ordinary skill in the art, without departing from the concept of the present application, a number of modifications and improvements can also be made, which all fall within the protection scope of the present application. Therefore, the protection scope of the patent of this application shall be subject to the appended claims.

Claims (20)

A streaming data column storage method includes the following steps: Read the data from the real-time messaging system to get the data to be processed; Analyze the data to be processed to obtain structured data; Convert the structured data into Row format data, and each time a set of the structured data is converted into Row format data, it is stored in the memory; A plurality of rows of the Row format data stored in the memory are combined into a Dataset <Row> format data, which is written into the file system in a column storage format. The streaming data column storage method according to claim 1, said reading data from a real-time messaging system to obtain data to be processed includes: Obtain the access authority of the real-time messaging system and connect to the real-time messaging system; Set an execution cycle, and read data from the real-time messaging system according to the execution cycle. According to the streaming data column storage method according to claim 1, the parsing the data to be processed to obtain structured data includes determining the format of the data to be processed and using different methods according to the judgment result The analysis includes: If the data to be processed is in json format, FastJSON is called to parse the data to be processed in json format into the structured data; If the to-be-processed data is in the csv format, then according to the content of the to-be-processed data, and using the DataFrame () method to add structured information to the to-be-processed data in the csv format to obtain the structured data. The streaming data column storage method according to claim 1, wherein the rows of the Row format data stored in the memory form a Dataset <Row> format data, and are written into a file system in a column storage format, including : Form the Dataset <Row> format data in multiple rows of the Row format data through the data frame method; Convert the Dataset <Row> format data to parquet format data through parquet (), and write the parquet format data to the file system using spark.read (). The streaming data column storage method according to claim 2, wherein the set execution cycle, and reading data from the real-time messaging system according to the execution cycle includes: Start reading from the location of the first piece of data in the real-time messaging system; Receive the completed reading instruction, stop reading, and record the completed reading position; Obtain the position where the last reading was completed, start reading from the position where the last reading was completed, until the instruction to complete the reading is received, stop reading, and record the position where the reading was completed. According to the streaming data column storage method of claim 3, if the to-be-processed data is in json format, calling FastJSON to parse the to-be-processed data in json format into the structured data includes: Extract the field information of the data to be processed in the json format; Sort the data to be processed in the json format according to the field information to obtain the structured data. According to the streaming data column storage method according to claim 1, the rows of the Row format data stored in the memory form a Dataset <Row> format data, and are written into the file system in a column storage format, Also includes: Divide the storage path according to the column information of the data to be processed; The partitionBy () function is called, and the columns with the same column names in the data to be processed are stored in different directories according to different values in the columns. A streaming data column storage device includes the following modules: The data acquisition module is set to read data from the real-time messaging system to obtain data to be processed; A data analysis module configured to analyze the data to be processed to obtain structured data; The data conversion module is configured to convert the structured data into Row format data, and each time a group of the structured data is converted into Row format data, it is stored in the memory; The data storage module is configured to compose multiple rows of Row format data stored in the memory into a Dataset <Row> format data, and write the data into a file system in a column storage format. A computer device includes a memory and a processor, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by one or more of the processors, the one or more of the processors are executed The following steps: Read the data from the real-time messaging system to get the data to be processed; Analyze the data to be processed to obtain structured data; Convert the structured data into Row format data, and each time a set of the structured data is converted into Row format data, it is stored in the memory; A plurality of rows of the Row format data stored in the memory are combined into a Dataset <Row> format data, which is written into the file system in a column storage format. The computer device of claim 9, when the computer-readable instructions are executed by one or more of the processors, causing the one or more of the processors to further perform the following steps: Obtain the access authority of the real-time messaging system and connect to the real-time messaging system; Set an execution cycle, and read data from the real-time messaging system according to the execution cycle. The computer device of claim 9, when the computer-readable instructions are executed by one or more of the processors, causing the one or more of the processors to further perform the following steps: If the data to be processed is in json format, FastJSON is called to parse the data to be processed in json format into the structured data; If the to-be-processed data is in the csv format, then according to the content of the to-be-processed data, and using the DataFrame () method to add structured information to the to-be-processed data in the csv format to obtain the structured data. The computer device of claim 9, when the computer-readable instructions are executed by one or more of the processors, causing the one or more of the processors to further perform the following steps: Form the Dataset <Row> format data in multiple rows of the Row format data through the data frame method; Convert the Dataset <Row> format data to parquet format data through parquet (), and write the parquet format data to the file system using spark.read (). The computer device of claim 10, when the computer-readable instructions are executed by one or more of the processors, causing the one or more of the processors to further perform the following steps: Start reading from the location of the first piece of data in the real-time messaging system; Receive the completed reading instruction, stop reading, and record the completed reading position; Obtain the position where the last reading was completed, start reading from the position where the last reading was completed, until the instruction to complete the reading is received, stop reading, and record the position where the reading was completed. The computer device of claim 11, when the computer-readable instructions are executed by one or more of the processors, causing the one or more of the processors to further perform the following steps: Extract the field information of the data to be processed in the json format; Sort the data to be processed in the json format according to the field information to obtain the structured data. The computer device of claim 9, when the computer-readable instructions are executed by one or more of the processors, causing the one or more of the processors to further perform the following steps: Divide the storage path according to the column information of the data to be processed; The partitionBy () function is called, and the columns with the same column names in the data to be processed are stored in different directories according to different values in the columns. A storage medium storing computer-readable instructions, which when executed by one or more processors, causes the one or more processors to perform the following steps: Read the data from the real-time messaging system to get the data to be processed; Analyze the data to be processed to obtain structured data; Convert the structured data into Row format data, and each time a set of the structured data is converted into Row format data, it is stored in the memory; A plurality of rows of the Row format data stored in the memory are combined into a Dataset <Row> format data, which is written into the file system in a column storage format. The storage medium of claim 16, when the computer-readable instructions are executed by one or more processors, causing the one or more processors to execute the following steps: Obtain the access authority of the real-time messaging system and connect to the real-time messaging system; Set an execution cycle, and read data from the real-time messaging system according to the execution cycle. The storage medium of claim 16, when the computer-readable instructions are executed by one or more processors, causing the one or more processors to perform further steps as follows: If the data to be processed is in json format, FastJSON is called to parse the data to be processed in json format into the structured data; If the to-be-processed data is in the csv format, then according to the content of the to-be-processed data, and using the DataFrame () method to add structured information to the to-be-processed data in the csv format to obtain the structured data. The storage medium of claim 16, when the computer-readable instructions are executed by one or more processors, causing the one or more processors to execute the following steps: Form the Dataset <Row> format data in multiple rows of the Row format data through the data frame method; Convert the Dataset <Row> format data to parquet format data through parquet (), and write the parquet format data to the file system using spark.read (). The storage medium of claim 17, when the computer-readable instructions are executed by one or more processors, causing the one or more processors to execute the following steps: Start reading from the location of the first piece of data in the real-time messaging system; Receive the completed reading instruction, stop reading, and record the completed reading position; Obtain the position where the last reading was completed, start reading from the position where the last reading was completed, until the instruction to complete the reading is received, stop reading, and record the position where the reading was completed.

Images