CN117609112B - Data stream processing method, device and multi-core processor - Google Patents
Data stream processing method, device and multi-core processor Download PDFInfo
- Publication number
- CN117609112B CN117609112B CN202311629442.8A CN202311629442A CN117609112B CN 117609112 B CN117609112 B CN 117609112B CN 202311629442 A CN202311629442 A CN 202311629442A CN 117609112 B CN117609112 B CN 117609112B
- Authority
- CN
- China
- Prior art keywords
- data stream
- high frequency
- target data
- identifier
- lookup table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 37
- 238000012545 processing Methods 0.000 claims abstract description 82
- 238000013507 mapping Methods 0.000 claims abstract description 76
- 238000000034 method Methods 0.000 claims abstract description 15
- 238000012217 deletion Methods 0.000 claims abstract description 11
- 230000037430 deletion Effects 0.000 claims abstract description 11
- 230000008569 process Effects 0.000 claims description 7
- 238000013500 data storage Methods 0.000 abstract description 6
- 230000004048 modification Effects 0.000 abstract description 6
- 238000012986 modification Methods 0.000 abstract description 6
- 238000007792 addition Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 5
- 238000013461 design Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0866—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches for peripheral storage systems, e.g. disk cache
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0866—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches for peripheral storage systems, e.g. disk cache
- G06F12/0871—Allocation or management of cache space
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Hardware Redundancy (AREA)
Abstract
本申请提供数据流处理方法、装置及多核处理器，在多核处理器中执行的方法包括：接收针对目标数据流的包含有目标数据流的唯一标识和操作类型的处理请求；若唯一标识属于非高频标识，则基于非高频标识地址映射表判断唯一标识是否存储在非高频标识查找表中，若否，则根据操作类型处理预存储在DDR中的目标数据流对应的系统数据对象或者更新非高频标识地址映射表及非高频标识查找表，非高频标识查找表用于存储预设在DDR中的各个系统数据对象中的一部分。本申请能够有效提高多核处理器的数据存储的扩展性，并以系统数据对象进行数据增删改查等处理，能够有效提高数据流处理的有效性及可靠性，并能够提高多核处理器的应用范围以及数据流处理性能。
The present application provides a data stream processing method, device and multi-core processor, the method executed in the multi-core processor includes: receiving a processing request for a target data stream including a unique identifier of the target data stream and an operation type; if the unique identifier belongs to a non-high-frequency identifier, judging whether the unique identifier is stored in a non-high-frequency identifier lookup table based on a non-high-frequency identifier address mapping table, if not, processing the system data object corresponding to the target data stream pre-stored in the DDR according to the operation type or updating the non-high-frequency identifier address mapping table and the non-high-frequency identifier lookup table, the non-high-frequency identifier lookup table is used to store a part of each system data object preset in the DDR. The present application can effectively improve the scalability of data storage of the multi-core processor, and perform data addition, deletion, modification and query processing with the system data object, which can effectively improve the effectiveness and reliability of data stream processing, and can improve the application scope of the multi-core processor and the data stream processing performance.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data stream processing method and apparatus, and a multicore processor.
Background
In the face of processing massive data, multi-core computing can complete a plurality of large-scale computing tasks, and data reading in a memory and an auxiliary memory often becomes a bottleneck for improving the running speed of an application program. The multi-processing core system faces limited internal storage resources in terms of access and data interaction and cannot be expanded infinitely.
Specifically, in a multi-core processor such as a DPU, different processing cores with different functions store and access processed data streams, and now, most of the different functional cores in the processor establish an independent mapping table to record storage information, so that the design has fixed storage performance and no space expansibility for the multi-core processor. In addition, the application scenes facing the core processor are richer, each processing core processes data streams with different lengths aiming at different scenes, and the working pressures are different. If reasonable data object management is lacking, the data flow of the processing core is limited to a certain extent, the application range of the multi-core processor is reduced, and the processing performance of the multi-core is also reduced.
Disclosure of Invention
In view of this, embodiments of the present application provide a data stream processing method, apparatus, and multi-core processor, so as to obviate or improve one or more disadvantages in the related art.
One aspect of the present application provides a data stream processing method, executed in a multi-core processor, the data stream processing including:
receiving a processing request aiming at a target data stream, wherein the processing request comprises a unique identifier and an operation type of the target data stream;
If the unique identifier of the target data stream belongs to a non-high frequency identifier, judging whether the unique identifier of the target data stream is stored in a local non-high frequency identifier lookup table corresponding to the non-high frequency identifier address mapping table based on a local preset non-high frequency identifier address mapping table, and if not, correspondingly processing a system data object corresponding to the target data stream prestored in the DDR according to the operation type, or correspondingly updating the non-high frequency identifier address mapping table and the non-high frequency identifier lookup table, wherein the non-high frequency identifier lookup table is used for storing part of each system data object preset in the DDR.
In some embodiments of the application, further comprising:
if the unique identification of the target data stream is judged to be stored in a local non-high frequency identification lookup table corresponding to the non-high frequency identification address mapping table based on the local preset non-high frequency identification address mapping table, correspondingly processing the system data object corresponding to the target data stream in the non-high frequency identification lookup table according to the operation type.
In some embodiments of the application, further comprising:
And if the unique identifier of the target data stream belongs to the high-frequency identifier, correspondingly processing a system data object corresponding to the target data stream in a local preset high-frequency identifier linked list according to the operation type, wherein the target data stream is used for storing the system data object corresponding to each unique identifier belonging to the high-frequency identifier.
In some embodiments of the present application, the non-high frequency identification address mapping table is used for storing the correspondence among the unique identification, the state identification and the storage address of each system data object in the non-high frequency identification lookup table;
Correspondingly, the determining, based on the local preset non-high frequency identification address mapping table, whether the unique identification of the target data stream is stored in the local non-high frequency identification lookup table corresponding to the non-high frequency identification address mapping table, if not, correspondingly processing a system data object corresponding to the target data stream pre-stored in the DDR according to the operation type, or correspondingly updating the non-high frequency identification address mapping table and the non-high frequency identification lookup table, including:
Judging whether each unique identifier in the non-high frequency identifier address mapping table contains a unique identifier of the target data stream, and the state identifier corresponding to the unique identifier of the target data stream is currently displayed as a valid state, if not, correspondingly processing a system data object corresponding to the target data stream prestored in the DDR according to the operation type, or correspondingly updating the non-high frequency identifier address mapping table and the non-high frequency identifier lookup table.
In some embodiments of the present application, the processing the system data object corresponding to the target data stream pre-stored in the DDR according to the operation type, or updating the non-high frequency identification address mapping table and the non-high frequency identification lookup table correspondingly, includes:
if the operation type is object writing, judging whether the non-high frequency identification lookup table contains residual storage bits or not;
if yes, writing the system data object corresponding to the target data stream into the forefront storage bit in the current residual storage bits of the non-high frequency identification lookup table, and correspondingly updating the non-high frequency identification address mapping table;
If not, occupying the last storage bit in the non-high frequency identification lookup table to write the system data object corresponding to the target data stream, and correspondingly updating the non-high frequency identification address mapping table.
In some embodiments of the present application, the processing the system data object corresponding to the target data stream pre-stored in the DDR according to the operation type, or updating the non-high frequency identification address mapping table and the non-high frequency identification lookup table correspondingly, includes:
and if the operation type is object access or object deletion, correspondingly extracting or deleting the system data object corresponding to the target data stream prestored in the DDR.
In some embodiments of the present application, the processing the system data object corresponding to the target data stream in the non-high frequency identification lookup table according to the operation type correspondence includes:
If the operation type is object writing, correspondingly updating a system data object corresponding to the target data stream in the non-high frequency identification lookup table, and replacing the system data object to the forefront storage bit in the non-high frequency identification lookup table;
If the operation type is object access, correspondingly extracting a system data object corresponding to the target data stream in the non-high frequency identification lookup table;
and if the operation type is object deletion, correspondingly deleting the system data object corresponding to the target data stream in the non-high frequency identification lookup table, releasing an address, and correspondingly updating the non-high frequency identification address mapping table.
In some embodiments of the present application, the system data object is configured to store a head-of-chain address, a tail-of-chain address, and ID status information corresponding to a unique identifier of a data stream.
Another aspect of the present application provides a data stream processing apparatus provided in a multi-core processor, the data stream processing apparatus:
The system comprises a request receiving module, a processing module and a processing module, wherein the request receiving module is used for receiving a processing request aiming at a target data stream, and the processing request comprises a unique identifier and an operation type of the target data stream;
And the fast and slow table query module is used for judging whether the unique identifier of the target data stream is stored in a local non-high frequency identifier lookup table corresponding to the non-high frequency identifier address mapping table based on a local preset non-high frequency identifier address mapping table if the unique identifier of the target data stream belongs to the non-high frequency identifier, and correspondingly processing a system data object corresponding to the target data stream prestored in the DDR according to the operation type if the unique identifier of the target data stream is not stored in the local non-high frequency identifier lookup table corresponding to the non-high frequency identifier address mapping table, or correspondingly updating the non-high frequency identifier address mapping table and the non-high frequency identifier lookup table, wherein the non-high frequency identifier lookup table is used for storing a part of each system data object preset in the DDR.
A third aspect of the present application provides a multi-core processor, the type of the multi-core processor including a DPU;
the multi-core processor is internally provided with a data stream processing device which is used for executing the data stream processing method.
The data stream processing method is executed in the multi-core processor, and the processing request aiming at the target data stream is received, wherein the processing request comprises the unique identifier and the operation type of the target data stream, if the unique identifier of the target data stream belongs to a non-high frequency identifier, the unique identifier of the target data stream is judged whether to be stored in a local non-high frequency identifier lookup table corresponding to the non-high frequency identifier address mapping table or not based on a local preset non-high frequency identifier address mapping table, if not, the system data object corresponding to the target data stream prestored in the DDR is correspondingly processed according to the operation type, or the non-high frequency identifier address mapping table and the non-high frequency identifier lookup table are correspondingly updated, wherein the non-high frequency identifier lookup table is used for storing part of the system data objects preset in the DDR, so that the data storage of the multi-core processor can be effectively improved, and the processing efficiency, the reliability of the data stream processing of the multi-core processor can be effectively improved, the multi-core processor can be improved, the application range of the multi-core processor can be effectively improved, the performance of the multi-core processor can be guaranteed, the multi-core processor can be further improved, and the performance of the multi-core processor can be guaranteed, and the multi-core processor can be further improved.
Additional advantages, objects, and features of the application will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and drawings.
It will be appreciated by those skilled in the art that the objects and advantages that can be achieved with the present application are not limited to the above-described specific ones, and that the above and other objects that can be achieved with the present application will be more clearly understood from the following detailed description.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate and together with the description serve to explain the application. The components in the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the application. Corresponding parts in the drawings may be exaggerated, i.e. made larger relative to other parts in an exemplary device actually manufactured according to the present application, for convenience in showing and describing some parts of the present application. In the drawings:
fig. 1 is a schematic flow chart of a data stream processing method according to an embodiment of the application.
Fig. 2 is a second flowchart of a data stream processing method according to an embodiment of the application.
FIG. 3 is a schematic diagram illustrating the storage of a non-high frequency identification address mapping table, a non-high frequency identification lookup table, and a high frequency identification linked list according to an embodiment of the present application.
FIG. 4 is a diagram illustrating a data structure of a non-high frequency identification address mapping table according to an embodiment of the present application.
FIG. 5 is a diagram illustrating a data structure of a sdo chain according to another example of the present application.
Fig. 6 is a third flowchart of a data stream processing method according to an embodiment of the application.
FIG. 7 is a diagram illustrating a data writing process according to an embodiment of the present application.
FIG. 8 is a diagram illustrating a data query execution process according to an embodiment of the present application.
FIG. 9 is a diagram illustrating a data deletion process according to an embodiment of the present application.
Fig. 10 is a schematic structural diagram of a data stream processing apparatus according to an embodiment of the application.
FIG. 11 is a schematic diagram of a multi-core processor according to an embodiment of the application.
Detailed Description
The present application will be described in further detail with reference to the following embodiments and the accompanying drawings, in order to make the objects, technical solutions and advantages of the present application more apparent. The exemplary embodiments of the present application and the descriptions thereof are used herein to explain the present application, but are not intended to limit the application.
It should be noted here that, in order to avoid obscuring the present application due to unnecessary details, only structures and/or processing steps closely related to the solution according to the present application are shown in the drawings, while other details not greatly related to the present application are omitted.
It should be emphasized that the term "comprises/comprising" when used herein is taken to specify the presence of stated features, elements, steps or components, but does not preclude the presence or addition of one or more other features, elements, steps or components.
It is also noted herein that the term "coupled" may refer to not only a direct connection, but also an indirect connection in which an intermediate is present, unless otherwise specified.
Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.
In order to improve the expansibility of data storage of a multi-core processor and ensure the high performance of operation of the multi-core processor, the embodiment of the application respectively provides a data stream processing method, a data stream processing device for executing the data stream processing method and the multi-core processor such as a DPU provided with the data stream processing device, and the high performance and completeness inside a DPU chip are ensured by adopting a fast-slow table combination mode.
The following examples are provided to illustrate the invention in more detail.
Based on this, an embodiment of the present application provides a data stream processing method that may be implemented by a functional unit or an apparatus in a multi-core processor, referring to fig. 1, where the data stream processing method specifically includes the following contents:
Step 100, receiving a processing request aiming at a target data stream, wherein the processing request comprises a unique identifier and an operation type of the target data stream.
It will be appreciated that the operation type may include at least writing (creating or updating), deleting or accessing a target data stream local to the multi-core processor.
In one or more embodiments of the present application, the unique identifier of the target data stream may be an ID or an identifier generated by using other user-defined coding methods.
Step 200, if the unique identifier of the target data stream belongs to a non-high frequency identifier, judging whether the unique identifier of the target data stream is stored in a local non-high frequency identifier lookup table corresponding to the non-high frequency identifier address mapping table based on a local preset non-high frequency identifier address mapping table, and if not, correspondingly processing a system data object corresponding to the target data stream prestored in the DDR according to the operation type, or correspondingly updating the non-high frequency identifier address mapping table and the non-high frequency identifier lookup table, wherein the non-high frequency identifier lookup table is used for storing a part of each system data object preset in the DDR.
In one or more embodiments of the present application, the non-high frequency identification address mapping table and the non-high frequency identification lookup table are in a one-to-one correspondence relationship, that is, the non-high frequency identification address mapping table may change in real time according to the change of the non-high frequency identification lookup table.
It is understood that the non-high frequency identification lookup table is used for storing a part of each system data object preset in the DDR, that is, the number of the system data objects stored in the non-high frequency identification lookup table is limited and is only used for storing a part, but not all, of each system data object preset in the DDR, and the specific number can be set according to actual requirements, for example, storage bits (positions for short) for storing the system data objects one to one in the non-high frequency identification lookup table can be set to 8, and correspondingly, the positions in the non-high frequency identification lookup table are also set to 8.
That is, the non-high frequency identification lookup table stored in the local storage area of the multi-core processor is provided with a plurality of storage bits for storing the system data objects corresponding to the non-high frequency identifications one by one, and the non-high frequency identification address mapping table is provided with a plurality of storage bits for storing the data such as the mapping addresses corresponding to the non-high frequency identifications of the non-high frequency identification lookup table one by one.
It should be noted that, the multi-core processor is locally further provided with a plurality of high-frequency identification linked lists for storing the storage bits of the system data objects corresponding to the high-frequency identifications one by one.
Based on this, the non-high frequency identification lookup table and the high frequency identification linked list form a slow table and a fast table respectively, so as to realize a fast and slow table query mechanism applicable to the step 200.
For convenience of explanation, the non-high frequency identification address mapping table in the embodiment of the present application may be abbreviated as a non-high frequency ID mapping table, the non-high frequency identification lookup table may be abbreviated as a non-high frequency ID LUT table, and the high frequency identification linked list may be abbreviated as a high frequency ID linked list. LUT means a look-up table.
In order to further improve the application range and the data stream processing performance of the multi-core processor, in one or more embodiments of the present application, the system data object is configured to store the link head address, the link tail address, and the ID status information corresponding to the unique identifier of the data stream. The system data object may be abbreviated as Sdo or Sdo (SYSTEM DATA object), and the system data object stored in the non-high frequency identification lookup table and the high frequency identification linked list may be written as a Sdo chain. Correspondingly, the operation type corresponding to the processing request of the target data stream can be abbreviated as sdo operation.
As can be seen from the above description, the data stream processing method provided by the embodiment of the present application can effectively improve the expansibility of data storage of the multi-core processor, and perform processing such as data deletion, modification and check by using the system data object, so that the effectiveness and reliability of data stream processing can be effectively improved, the application range and the data stream processing performance of the multi-core processor can be improved, and further, the high performance and completeness inside the multi-core processor such as the DPU can be ensured, and under the condition of limited design resources of the multi-core processor, the high performance can be ensured, and the storage expansibility of the multi-core processor can be increased.
In order to further improve the expandability of data storage and the data stream processing efficiency of the multi-core processor, in the data stream processing method provided by the embodiment of the present application, referring to fig. 2, step 200 in the data stream processing method specifically includes the following contents:
Step 210, judging whether the unique identifier of the target data stream belongs to a non-high frequency identifier, if yes, executing step 220, and if not, executing step 250.
In step 210, one implementation method for determining that the unique identifier of the target data stream belongs to the non-high frequency identifier may include querying an identifier frequency table pre-stored locally, where the identifier frequency table may include frequencies or times of occurrence of each preset unique identifier in a preset time period, and then determining whether the frequency or times of occurrence of the unique identifier of the target data stream in the preset time period, which is found from the identifier frequency table, exceeds a preset threshold, if yes, determining the unique identifier of the target data stream as the high frequency identifier, and if not, determining the unique identifier of the target data stream as the non-high frequency identifier. In addition, during or after each execution of step 200, the frequency or number of occurrences of the unique identifier of the target data stream in the identifier frequency table within the preset time period needs to be updated correspondingly.
And another implementation way of judging that the unique identifier of the target data stream belongs to the non-high frequency identifier may include judging whether the unique identifier of the target data stream is a high frequency identifier according to a value or a mark in a frequency identifier bit preset in the unique identifier of the target data stream, for example, the frequency identifier bit may include a high frequency mark or a non-high frequency mark, and if the frequency identifier bit in the unique identifier of the target data stream is a high frequency mark, determining that the unique identifier of the target data stream is a high frequency identifier.
Step 220, judging whether the unique identifier of the target data stream is stored in a local non-high frequency identifier lookup table corresponding to the non-high frequency identifier address mapping table based on the local preset non-high frequency identifier address mapping table, if so, executing step 230, and if not, executing step 240.
Step 230, if it is determined based on the local preset non-high frequency identification address mapping table that the unique identification of the target data stream is stored in the local non-high frequency identification lookup table corresponding to the non-high frequency identification address mapping table, correspondingly processing the system data object corresponding to the target data stream in the non-high frequency identification lookup table according to the operation type.
Step 240, if it is determined based on the local preset non-high frequency identification address mapping table that the unique identification of the target data stream is not stored in the local non-high frequency identification lookup table corresponding to the non-high frequency identification address mapping table, correspondingly processing the system data object corresponding to the target data stream pre-stored in the DDR according to the operation type, or correspondingly updating the non-high frequency identification address mapping table and the non-high frequency identification lookup table.
Step 250, if the unique identifier of the target data stream belongs to a high-frequency identifier, correspondingly processing a system data object corresponding to the target data stream in a local preset high-frequency identifier linked list according to the operation type, wherein the target data stream is used for storing the system data object corresponding to each unique identifier belonging to the high-frequency identifier.
In one or more embodiments of the present application, in order to further improve the application validity and reliability of the non-high frequency identification address mapping table, the non-high frequency identification address mapping table is provided with a plurality of storage bits, and each storage bit is used for a correspondence among a unique identifier, a status identifier and a storage address of each system data object stored in the non-high frequency identification lookup table in a one-to-one manner. Wherein, referring to fig. 3, the status identifier may be abbreviated as mapbit, and the storage address may be abbreviated as lut _addr.
On this basis, in order to further describe the above non-high frequency identification address mapping table, the non-high frequency identification lookup table and the high frequency identification linked list, in a specific application example of the fast and slow lookup table of the present application, the specific application example includes the following contents:
(1) The ID number of the data stream is divided into a high frequency ID and a non-high frequency ID.
(2) In order to improve efficiency, a high-frequency ID linked list is designed, and a part of head address, tail address and ID state related information of high-frequency ID with high frequency are stored.
(3) The non-high frequency ID stores information about the head address, tail address, length, and ID status of the ID linked list in the auxiliary information area (SDO area) of the DDR. Meanwhile, a non-high frequency ID LUT table is built for non-high frequency ID to quickly inquire and update, after 8 IDs in the non-high frequency ID (address) mapping table corresponding to the non-high frequency ID LUT table are occupied by new IDs, the original 8 IDs move backwards, and the last ID is extruded out of the LUT table and only exists in DDR. The query system data object Sdo (SYSTEM DATA object) command of the non-high frequency ID first queries whether the non-high frequency ID to be queried is in the non-high frequency ID LUT table, and the DDR is queried only if it is not queried.
(4) The non-high frequency ID (address) mapping table supports at most 8 non-high frequency ID indexes and addresses LUT _addr, mapbit of a random access memory ram storing sdo chains, which are used for indicating which data in the 8 positions are valid, when the chain is fetched, all valid information and the non-high frequency ID with inquiry are simultaneously compared, if the non-high frequency ID LUT table is not compared, the non-high frequency ID LUT table is proved to have no information chain of the non-high frequency ID to be inquired, the information chain of the query ID in the DDR needs to be fetched, and if the comparison is finished (only one comparison is finished), the address corresponding to the non-high frequency ID LUT table is removed to read the sdo chains.
The data structure of the non-high frequency ID (address) mapping table is shown in fig. 4, and the data structures of the non-high frequency identification lookup table (non-high frequency ID LUT table) and the system data object (sdo chain) in the high frequency identification linked list (high frequency ID linked list) are shown in fig. 5.
The meaning of the data structure of the LUT table in fig. 4 is shown in table 1 below:
TABLE 1
The meaning of the data structure of the linked list in fig. 5 is shown in table 2 below:
TABLE 2
In order to further improve the reliability and effectiveness of determining whether the unique identifier of the target data stream is stored in the non-high frequency identifier lookup table, in the data stream processing method provided by the embodiment of the present application, referring to fig. 6, step 220 in the data stream processing method specifically includes the following contents:
Step 221, judging whether each unique identifier in the non-high frequency identifier address mapping table contains a unique identifier of the target data stream, and the state identifier corresponding to the unique identifier of the target data stream is currently displayed as a valid state, if yes, executing step 230, and if no, executing step 240.
In order to further improve the writing efficiency and reliability of the system data object corresponding to the non-high frequency identifier, in the data stream processing method provided by the embodiment of the present application, referring to fig. 6 to fig. 9, step 240 in the data stream processing method specifically includes the following:
Step 241, if the operation type is object writing, judging whether the non-high frequency identification lookup table contains the rest position, if yes, executing step 242, and if not, executing step 243.
And step 242, writing the system data object corresponding to the target data stream into the forefront position in the current residual position of the non-high frequency identification lookup table, and correspondingly updating the non-high frequency identification address mapping table.
Step 243, occupying the last position in the non-high frequency identification lookup table to write the system data object corresponding to the target data stream, and correspondingly updating the non-high frequency identification address mapping table.
In order to further improve the access or deletion efficiency and reliability of the system data object corresponding to the non-high frequency identifier, in the data stream processing method provided by the embodiment of the present application, referring to fig. 6 to fig. 9, step 240 in the data stream processing method further specifically includes the following contents:
Step 244, if the operation type is object access or object deletion, correspondingly extracting or deleting the system data object corresponding to the target data stream pre-stored in the DDR.
In order to further improve the writing, accessing or deleting efficiency and reliability of the system data object corresponding to the high-frequency identifier, in the data stream processing method provided by the embodiment of the present application, referring to fig. 6 to fig. 9, step 250 in the data stream processing method further specifically includes the following contents:
Step 251, if the operation type is object writing, correspondingly updating the system data object corresponding to the target data stream in the non-high frequency identification lookup table, and replacing the system data object to the forefront storage bit in the non-high frequency identification lookup table.
Step 252, if the operation type is object access, correspondingly extracting a system data object corresponding to the target data stream in the non-high frequency identification lookup table.
Step 253, if the operation type is object deletion, correspondingly deleting the system data object corresponding to the target data stream in the non-high frequency identification lookup table, releasing an address, and correspondingly updating the non-high frequency identification address mapping table.
Specifically, in fig. 7, the multi-core processor receives the sdo writing operation, and if the ID to be queried is a high-frequency ID, the sdo chain is directly written to the address corresponding to the high-frequency identification linked list (i.e., the high-frequency ID ram). If the ID to be queried is not a high-frequency ID, reading an information chain of the mapped address on the film, if the ID to be queried is compared, updating a corresponding sdo chain in ram and improving the corresponding sdo chain to the forefront, if the ID to be queried is not compared, applying a ram address to an application module, if the ID to be queried is not applied, occupying the position of the first written ID and updating the priority, if the ID to be queried is applied, detecting status bits (namely storage bits), if 8 are all full, occupying the position of the first written ID and updating the priority, and if the residual position is left, occupying the forefront position in the residual position.
In fig. 8, the multi-core processor receives the inquiry sdo operation, and if the ID to be inquired is a high-frequency ID, inquires a high-frequency identification linked list (i.e. a high-frequency ID ram) and returns a result. If the ID to be queried is not the high-frequency ID, reading the information chain of the mapped address on the chip, if the ID to be queried is compared, reading ram through the corresponding address and returning a result, and if the ID to be queried is not compared, returning the LUT, and if the ID to be queried is not compared, returning to query DDR.
In fig. 9, the multicore processor receives the operation of deleting sdo, and if the ID to be deleted is a high frequency ID, sets 0 to the state position of the corresponding address of the high frequency identification linked list (i.e. the high frequency ID ram). If the ID to be deleted is not the high-frequency ID, reading the information chain of the mapped address on the film, deleting the corresponding ID if the ID to be deleted is compared, releasing the address, reconstructing the information chain, and inquiring the DDR and deleting the ID if the ID to be deleted is not compared.
The present application also provides a data stream processing device for executing all or part of the data stream processing method, which is disposed in the multi-core processor, referring to fig. 10, and specifically includes the following contents:
A request receiving module 10, configured to receive a processing request for a target data stream, where the processing request includes a unique identifier and an operation type of the target data stream;
And the fast and slow table query module 20 is configured to determine, based on a local preset non-high frequency identification address mapping table, whether the unique identification of the target data stream is stored in a local non-high frequency identification lookup table corresponding to the non-high frequency identification address mapping table, if not, correspondingly processing a system data object corresponding to the target data stream pre-stored in the DDR according to the operation type, or correspondingly updating the non-high frequency identification address mapping table and the non-high frequency identification lookup table, where the non-high frequency identification lookup table is used to store a part of each system data object preset in the DDR.
The embodiment of the data stream processing device provided by the present application may be specifically used for executing the processing flow of the embodiment of the data stream processing method in the above embodiment, and the functions thereof are not described herein in detail, and reference may be made to the detailed description of the embodiment of the data stream processing method.
The part of the data stream processing device for performing data stream processing can be completed in the multi-core processor. The selection may be specifically performed according to the processing capability of the multi-core processor, and restrictions of the use scenario of the user. The application is not limited in this regard.
The multi-core processor may have a communication module (i.e. a communication unit) and may be connected to a remote server in a communication manner, so as to implement data transmission with the server. The server may include a server on the side of the task scheduling center, and in other implementations may include a server of an intermediate platform, such as a server of a third party server platform having a communication link with the task scheduling center server. The server may include a single computer device, a server cluster formed by a plurality of servers, or a server structure of a distributed device.
Any suitable network protocol may be used between the servers and the multi-core processor, including those not yet developed at the filing date of the present application. The network protocols may include, for example, TCP/IP protocol, UDP/IP protocol, HTTP protocol, HTTPS protocol, etc. Of course, the network protocol may also include, for example, RPC protocol (Remote Procedure Call Protocol ), REST protocol (Representational STATE TRANSFER) or the like used above the above-described protocol.
As can be seen from the above description, the data stream processing device provided by the embodiment of the present application can effectively improve the expansibility of data storage of a multi-core processor, and perform processing such as data deletion, modification and check by using a system data object, so that the effectiveness and reliability of data stream processing can be effectively improved, the application range and the data stream processing performance of the multi-core processor can be improved, and further, the high performance and completeness inside the multi-core processor such as a DPU can be ensured, and under the condition of limited design resources of the multi-core processor, the high performance can be ensured, and the storage expansibility of the multi-core processor can be increased.
The embodiment of the application also provides a multi-core processor, which can be a DPU specifically, and the multi-core processor is provided with a data stream processing device, and the data stream processing device is used for executing all or part of the content in the data stream processing method described in the previous embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative components, systems, and methods described in connection with the embodiments disclosed herein can be implemented as hardware, software, or a combination of both. The particular implementation is hardware or software dependent on the specific application of the solution and the design constraints. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, a plug-in, a function card, or the like. When implemented in software, the elements of the application are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine readable medium or transmitted over transmission media or communication links by a data signal carried in a carrier wave.
It should be understood that the application is not limited to the particular arrangements and instrumentality described above and shown in the drawings. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and shown as examples. The method processes of the present application are not limited to the specific steps described and shown, but various changes, modifications and additions, or the order between steps may be made by those skilled in the art after appreciating the spirit of the present application.
In this disclosure, features that are described and/or illustrated with respect to one embodiment may be used in the same way or in a similar way in one or more other embodiments and/or in combination with or instead of the features of the other embodiments.
The above description is only of the preferred embodiments of the present application and is not intended to limit the present application, and various modifications and variations can be made to the embodiments of the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.
Claims (8)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311629442.8A CN117609112B (en) | 2023-11-30 | 2023-11-30 | Data stream processing method, device and multi-core processor |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311629442.8A CN117609112B (en) | 2023-11-30 | 2023-11-30 | Data stream processing method, device and multi-core processor |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117609112A CN117609112A (en) | 2024-02-27 |
| CN117609112B true CN117609112B (en) | 2025-02-11 |
Family
ID=89955881
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311629442.8A Active CN117609112B (en) | 2023-11-30 | 2023-11-30 | Data stream processing method, device and multi-core processor |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117609112B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111443875A (en) * | 2020-04-13 | 2020-07-24 | Oppo广东移动通信有限公司 | Data processing method, device and device based on address relation table |
| CN115461727A (en) * | 2020-04-30 | 2022-12-09 | 美光科技公司 | Asynchronous power down handling in sequentially programmed memory subsystems |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4091604B2 (en) * | 2002-10-03 | 2008-05-28 | 株式会社インフォーエス | Bit string matching method and apparatus |
| US9086973B2 (en) * | 2009-06-09 | 2015-07-21 | Hyperion Core, Inc. | System and method for a cache in a multi-core processor |
| US8627189B2 (en) * | 2009-12-03 | 2014-01-07 | Microsoft Corporation | High performance digital signal processing in software radios |
| CN109344094B (en) * | 2018-09-26 | 2021-10-29 | 郑州云海信息技术有限公司 | Address mapping relationship feedback method, apparatus, device and readable storage medium |
| CN112364347A (en) * | 2020-11-19 | 2021-02-12 | 全知科技(杭州)有限责任公司 | High-performance computing method for identifying high-frequency data access and operation |
-
2023
- 2023-11-30 CN CN202311629442.8A patent/CN117609112B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111443875A (en) * | 2020-04-13 | 2020-07-24 | Oppo广东移动通信有限公司 | Data processing method, device and device based on address relation table |
| CN115461727A (en) * | 2020-04-30 | 2022-12-09 | 美光科技公司 | Asynchronous power down handling in sequentially programmed memory subsystems |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117609112A (en) | 2024-02-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111340470B (en) | Block chain transaction processing method, node and contract container | |
| CN105933376B (en) | A data manipulation method, server and storage system | |
| JP4792505B2 (en) | Data synchronization processing method, client, server, and data synchronization system between client and server | |
| US6754799B2 (en) | System and method for indexing and retrieving cached objects | |
| CN103064960B (en) | Data base query method and equipment | |
| CN105653198A (en) | Data processing method and device | |
| CN114143261B (en) | Method and system for dynamically routing back-end address by API gateway | |
| CN110083307A (en) | Date storage method, memory and server | |
| CN103036948B (en) | Namely network file processing method, XM, software serve SaaS platform | |
| CN109947718A (en) | A data storage method, storage platform and storage device | |
| CN106934066B (en) | Metadata processing method and device and storage equipment | |
| CN111796772A (en) | Cache management method, cache node and distributed storage system | |
| CN117609112B (en) | Data stream processing method, device and multi-core processor | |
| CN114218317A (en) | Progress information management and control method, microservice device, electronic device and storage medium | |
| CN117749762B (en) | DNS access request processing method, device, equipment and storage medium | |
| WO2021237431A1 (en) | Data processing method and apparatus, processing device, and data storage system | |
| US10402373B1 (en) | Filesystem redirection | |
| CN110737635A (en) | data blocking method | |
| CN115599711B (en) | Cache data processing method, system, device, equipment and computer storage medium | |
| CN116028677B (en) | Data operation method and system | |
| CN117874061A (en) | System and method for realizing remote storage based on clusters | |
| CN107357853B (en) | Method and device for operating REDIS console and computer system | |
| CN117056246A (en) | Data caching method and system | |
| CN102118256B (en) | Method for acquiring simple network management protocol (SNMP) data and equipment-side device | |
| US7653070B2 (en) | Method and system for supporting efficient and cache-friendly TCP session lookup operations based on canonicalization tags |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |