CN111865326A - Data compression method, device, equipment and storage medium - Google Patents
Data compression method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111865326A CN111865326A CN202010674720.1A CN202010674720A CN111865326A CN 111865326 A CN111865326 A CN 111865326A CN 202010674720 A CN202010674720 A CN 202010674720A CN 111865326 A CN111865326 A CN 111865326A
- Authority
- CN
- China
- Prior art keywords
- transmission
- cost
- data packet
- compression
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000013144 data compression Methods 0.000 title claims abstract description 39
- 230000005540 biological transmission Effects 0.000 claims abstract description 317
- 238000007906 compression Methods 0.000 claims abstract description 154
- 230000006835 compression Effects 0.000 claims abstract description 153
- 238000004891 communication Methods 0.000 claims abstract description 109
- 238000005457 optimization Methods 0.000 abstract description 6
- 238000010586 diagram Methods 0.000 description 6
- 230000006837 decompression Effects 0.000 description 5
- 238000010606 normalization Methods 0.000 description 5
- 238000005265 energy consumption Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000020169 heat generation Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
- H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
- H03M7/60—General implementation details not specific to a particular type of compression
- H03M7/6047—Power optimization with respect to the encoder, decoder, storage or transmission
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The embodiment of the disclosure discloses a data compression method, a data compression device, data compression equipment and a storage medium. The method comprises the following steps: acquiring a starting place and a destination of data packet transmission; determining the transmission cost of data packet transmission according to the starting place and the destination; and determining a compression mode of the data packet according to the transmission cost, compressing the data packet according to the compression mode, and storing the compression mode in the compressed packet after the data packet is compressed. The technical scheme of the embodiment of the disclosure solves the problems that the transmission cost is high in the long-distance transmission of the compression mode with low compression ratio caused by adopting the unified compression mode for compression in the traditional data communication, and the operation amount is large and the consumed time is long when the compression mode with high compression ratio is adopted for compressing and decompressing the data packet, reduces the operation time of compressing and decompressing the data packet, improves the transmission efficiency of the data packet, and realizes the double optimization of the transmission rate and the transmission cost.
Description
Technical Field
The embodiments of the present disclosure relate to the field of data compression technologies, and in particular, to a data compression method, apparatus, device, and storage medium.
Background
The many-core system is composed of one or more multi-core processors to solve the problems of low speed and high heat generation when a single-core chip performs large-data-volume calculation, wherein the multi-core processor is formed by integrating a plurality of complete calculation engines (cores) on one processor, and the cores in one processor chip or among a plurality of processor chips can work in a mutual cooperation mode. For many-core systems, data needs to be communicated and interacted between multiple cores and multiple chips.
In the conventional method, a uniform compression coding scheme is usually adopted for data communication between chips, that is, a data packet for data communication is compressed and transmitted in a uniform compression mode. However, the data communication method that performs compression and transmission by using the unified compression mode does not consider the difference between transmission media and the difference between transmission distances, and the transmission cost of data in the transmission process also changes, so that the double optimization of the transmission efficiency and the transmission cost cannot be achieved.
Disclosure of Invention
The present disclosure provides a data compression method, apparatus, device and storage medium, so as to implement selection of a data compression mode and implement dual optimization of transmission rate and transmission cost.
In a first aspect, an embodiment of the present disclosure provides a data compression method, including:
acquiring a starting place and a destination of data packet transmission;
determining the transmission cost of data packet transmission according to the starting place and the destination;
and determining a compression mode of the data packet according to the transmission cost, compressing the data packet according to the compression mode, and storing the compression mode in the compressed packet after the data packet is compressed.
Further, determining the transmission cost of the data packet transmission according to the starting place and the destination comprises:
acquiring transmission path information between an initial place and a destination;
determining each communication medium between the starting place and the destination, the transmission times on each communication medium and the single cost of transmission on each communication medium according to the transmission path information;
and determining the transmission cost of the data packet transmission according to each communication medium, each transmission frequency and each single cost.
Further, determining the transmission cost of the data packet transmission according to each communication medium, each transmission number and each single cost, includes:
determining the product of the transmission times and the single cost in each communication medium in the transmission path information;
and taking the sum of the products of the transmission times and the single cost in each communication medium as the transmission cost of the data packet transmission.
Further, determining the compressed mode of the data packet according to the transmission cost includes:
determining the ratio of the transmission cost to the reference single cost as a normalized cost;
and searching a corresponding compression scheme according to the normalized cost, and determining the compression mode of the data packet according to the compression scheme.
Further, the communication medium includes: and at least one of a co-board inter-chip communication medium, a co-board inter-board communication medium, a co-cluster inter-machine communication medium, and a remote communication medium.
Further, when the number of the datagrams is multiple, compressing the data packet according to the compression mode, and storing the compression mode in the compressed packet after the data packet is compressed, the method includes:
acquiring a group of data packets with the same starting place and the same destination;
compressing a group of data packets into the same compressed packet according to a compression mode;
and saving the compression mode into the compression packet.
In a second aspect, an embodiment of the present disclosure further provides a data compression apparatus, where the data compression apparatus includes:
the acquisition module is used for acquiring the starting place and the destination of the data packet transmission;
a transmission cost determining module, configured to determine a transmission cost for sending the data packet according to the starting location and the destination;
And the compression module is used for determining a compression mode of the data packet according to the transmission cost, compressing the data packet according to the compression mode and storing the compression mode in the compressed packet after the data packet is compressed.
Further, the transmission cost determination module includes:
a path information acquisition unit for acquiring transmission path information between a start place and a destination;
a path information determination unit for determining each communication medium between the start and destination, the number of transmissions on each communication medium, and a single cost of transmission on each communication medium, based on the transmission path information;
and the transmission cost determining unit is used for determining the transmission cost of the data packet transmission according to each communication medium, each transmission frequency and each single cost.
In a third aspect, an embodiment of the present disclosure further provides an apparatus, where the apparatus includes:
one or more processors;
storage means for storing one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement a data compression method as provided in any embodiment of the present disclosure.
In a fourth aspect, embodiments of the present disclosure also provide a storage medium containing computer-executable instructions for performing a data compression method as provided in any of the embodiments of the present disclosure when executed by a computer processor.
The method comprises the steps of acquiring a starting place and a destination of data packet transmission; determining the transmission cost of data packet transmission according to the starting place and the destination; and determining a compression mode of the data packet according to the transmission cost, compressing the data packet according to the compression mode, and storing the compression mode in the compressed packet after the data packet is compressed. The method comprises the steps of obtaining an initial place and a destination of data packet transmission, determining a transmission distance and a transmission medium which need to pass in the data packet transmission process according to the initial place and the destination, further determining transmission cost needed by the data packet to be transmitted from the initial place to the destination, and selecting a proper data packet compression mode to compress the data packet according to the transmission cost.
Drawings
FIG. 1 is a flow chart of a method of data compression in an exemplary embodiment of the present disclosure;
FIG. 2 is a flow chart of a method of data compression in an exemplary embodiment of the present disclosure;
fig. 3 is a flow chart of determining transmission costs for packet transmission according to each transmission number and each single cost according to each communication medium in an exemplary embodiment of the present disclosure;
FIG. 4 is an exemplary diagram of a data packet transmission process in an exemplary embodiment of the present disclosure;
FIG. 5 is a schematic diagram of a data compression apparatus in an exemplary embodiment of the present disclosure;
fig. 6 is a schematic structural diagram of an apparatus in an exemplary embodiment of the present disclosure.
Detailed Description
The present disclosure is described in further detail below with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the disclosure and are not limiting of the disclosure. It should be further noted that, for the convenience of description, only some of the structures relevant to the present disclosure are shown in the drawings, not all of them. Furthermore, the embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict.
Fig. 1 is a flowchart of a data compression method according to an exemplary embodiment of the present disclosure, where the present embodiment is applicable to a case where data is compressed before being transmitted between a plurality of different computing engines of the same chip or between different computing engines of a plurality of chips, and the method may be executed by a data compression apparatus, where the method includes the following steps:
S101, acquiring a starting place and a destination of data packet transmission.
A data packet is understood as a data unit in communication transmission, and when a computing engine needs to transmit a message or a group of data to another computing engine, the message or the group of data may be divided into a plurality of data blocks, which are called data packets and contain address information of a receiver, and each data packet is transmitted to the address of the receiver along different paths according to the difference of the address information.
In one possible implementation, at the start of the routing packet, the computing engine corresponding to the address information of the receiver in the data packet is determined as the destination by reading the address information of the receiver in the data packet. In order to clarify the transmission route of the data packet, the determination of the starting place and the destination of the data packet can be realized by reading the address information of the receiver in the data packet, so as to determine the transmission route of the data packet according to the starting place and the destination.
And S102, determining the transmission cost of the data packet according to the starting place and the destination.
The transmission cost is understood to be transmission delay and transmission energy consumption of data in the process of transmitting data from one computing engine to another computing engine or transmitting data from a chip where one computing engine is located to a chip where another computing engine is located, namely time taken for transmission and power consumption caused by transmission. The transmission cost is related to the amount of transmitted data and the transmission distance, the larger the amount of transmitted data is, the higher the transmission cost is, meanwhile, the transmission cost is related to the medium experienced in the transmission process, and the farther the transmission distance is, the more the high energy consumption or high delay medium is experienced in the transmission process, and the higher the transmission cost is.
According to the obtained starting place and destination, determining transmission nodes experienced in the process that a data packet is transmitted from a starting calculation engine to a destination calculation engine, obtaining single transmission cost for single data transmission between adjacent transmission nodes, and taking the sum of the single transmission cost between the transmission nodes in the transmission process as the transmission cost of the data packet, wherein the transmission nodes can be understood as routing nodes experienced in the transmission process, and the transmission nodes used for calculating the transmission cost comprise the starting calculation engine corresponding to the starting place and the destination calculation engine corresponding to the destination.
And determining a transmission route of the data packet according to the starting place and the destination, and determining the transmission cost of the data packet according to the transmission nodes in the transmission route so as to select a proper compression mode according to the determined transmission cost.
S103, determining a compression mode of the data packet according to the transmission cost, compressing the data packet according to the compression mode, and storing the compression mode in the compressed packet after the data packet is compressed.
The compression mode can be understood as a compression scheme name and parameter list, which can be used to measure the compression efficiency of data compression, and can be determined according to the required compression ratio, wherein one compression mode corresponds to one compression ratio.
According to a possible implementation mode, the transmission type of the data packet from the starting place to the destination can be determined according to the transmission cost, when the transmission cost is low, the time delay and the energy consumption in the data packet transmission process are low, if the compression mode with larger compression ratio is adopted to compress the data packet, the data compression and decompression time is increased, and when the transmission cost is low, the data packet can be transmitted by adopting a lower compression ratio or without compression. Meanwhile, as the transmission cost increases, both the time delay and the energy consumption increase in the data packet transmission process, and at this time, a compression mode with a higher compression ratio needs to be selected to reduce the transmitted data volume, thereby reducing the transmission cost. Therefore, the compression ratio with the shortest sum of the transmission delay and the compression time can be determined according to the transmission cost and serves as the determined compression ratio information, the corresponding compression mode is selected according to the determined compression ratio information to compress the data packet, and the compression mode is stored in the compressed packet after the data packet is compressed so as to decompress the compressed packet at the destination.
The method comprises the steps of acquiring a starting place and a destination of data packet transmission; determining the transmission cost of data packet transmission according to the starting place and the destination; and determining a compression mode of the data packet according to the transmission cost, compressing the data packet according to the compression mode, and storing the compression mode in the compressed packet after the data packet is compressed. The method comprises the steps of obtaining an initial place and a destination of data packet transmission, determining a transmission distance and a transmission medium which need to pass in the data packet transmission process according to the initial place and the destination, further determining transmission cost needed by the data packet to be transmitted from the initial place to the destination, and selecting a proper data packet compression mode to compress the data packet according to the transmission cost.
Fig. 2 is a flowchart of a data compression method according to an exemplary embodiment of the present disclosure. The technical scheme of the embodiment is further refined on the basis of the technical scheme, and can comprise the following steps:
s201, acquiring the starting place and the destination of the data packet transmission.
S202, acquiring transmission path information between the starting place and the destination.
The transmission path information is understood to be transmission node information experienced in the transmission process of a data packet from an initial source to a destination. The transmission path information may be used to determine communication medium information experienced during transmission, transmission frequency information on each communication medium, and the like, where the transmission node may include a chip corresponding to a destination at a start point during transmission, a chip passing through during transmission, and a data transmission interface on the motherboard when data needs to be transmitted across the motherboard.
When the position relation and the communication connection relation between the chips are determined, the logic position relation between the chips is also determined, and a path required by one chip to send data to the other chip through the chip and/or the data transmission interface is also determined, namely the logic position relation between the corresponding chip at the beginning and the corresponding chip at the destination and the path required by the chip and/or the data transmission interface when the data transmission is carried out are determined. Determining the type of the communication medium which the communication medium information between the adjacent chips on the path can pass through for one-time data transmission, and the times of data transmission on each type of communication medium, and taking the type of the communication medium and the time information as the transmission path information between the starting place and the destination.
Optionally, the determination of the transmission path information may be determined by looking up a preset address information table, or may be determined by configuring a preset route, which is not limited in this disclosure.
S203, determining each communication medium between the starting place and the destination, the transmission times on each communication medium and the single cost of transmission on each communication medium according to the transmission path information.
Communication media may be understood to be transmission media through which data packets are transmitted, and may include wired media and wireless media. In an embodiment of the present disclosure, the communication medium may include: and at least one of a co-board inter-chip communication medium, a co-board inter-board communication medium, a co-cluster inter-machine communication medium, and a remote communication medium.
The single cost is understood to mean the transmission time and transmission consumption required for a data transmission over a medium, and is a known constant value that is collected from the device or estimated by design.
According to a possible implementation manner, a chip where a computing engine is located can be installed on different mainboards, a plurality of mainboards can be combined to form a computer device, and a computer cluster can be obtained after the plurality of computer devices are integrated. Determining a communication medium between the origin and the destination according to the transmission path information may include: the communication medium for communicating the chips and the data transmission interface on the same mainboard is a communication medium between chips on the same mainboard; the communication medium for the data transmission interfaces between different main boards on the same computer equipment to communicate is the communication medium between the same main boards; the communication medium of data communication between different computer devices in the same computer cluster is the communication medium between computers in the same cluster; and communication media other than those used for data communication to remote devices are long-range communication media. From the transmission path information, it is possible to determine the communication medium through which the data has been transmitted from the origin to the destination, the number of transmissions over the same communication medium, and the transmission cost required for one propagation over each communication medium.
Determining each communication medium between the starting place and the destination and the single cost transmitted on each communication medium according to the transmission path information, so that the transmission cost required by the transmission of the data packet on different media can be determined, and further the determination of the transmission cost when the data packet is transmitted on different transmission paths is realized.
And S204, determining the transmission cost of the data packet according to each communication medium, each transmission frequency and each single cost.
According to a possible implementation manner, transmission costs on the same communication medium can be determined according to each transmission frequency and each single cost, and transmission costs of a data packet in the whole transmission process can be determined according to corresponding transmission costs on each communication medium.
Further, fig. 3 is a flowchart for determining a transmission cost for sending a data packet according to each communication medium, each transmission time and each single cost according to an exemplary embodiment of the present disclosure, where the flowchart may include the following steps:
s2041, determining the product of the transmission times and the single cost in each communication medium in the transmission path information.
In a possible implementation manner, in the process of performing one data transmission, the transmissions on the same communication medium may not be continuous, but since the single cost required for performing a single transmission on the same communication medium is the same, the product of the transmission times and the single cost in the same communication medium in the transmission path information is used as the transmission cost required for the data transmission on the corresponding transmission path by the communication medium, and the transmission cost required for the data transmission on the transmission path is calculated for each communication medium in the transmission path information.
And S2042, taking the sum of the products of the transmission times and the single cost in each communication medium as the transmission cost of the data packet transmission.
In a possible implementation manner, if the number of data transmission times on a communication medium between chips on the same board is Nb and the single transmission cost is Cb in one data transmission process; the data transmission times on the same-machine inter-board communication medium are Nm, and the single transmission cost is Cm; the data transmission times on the communication medium between the same cluster machines are Nc, and the single transmission cost is Cc; the number of data transmissions over a long distance communication medium is Nr, and the single cost of the transmission is Cr. The transmission cost of the data packet transmission in the data transmission process can be expressed as:
C＝Nb*Cb+Nm*Cm+Nc*Cc+Nr*Cr
fig. 4 is an exemplary diagram of a data packet transmission process according to an exemplary embodiment of the disclosure, and as can be seen from fig. 4, in the process of transmitting a data packet from the chip 1 to the chip 12, transmission paths of the data packet are: chip 1- > chip 2- > chip 3- > data transmission interface 1- > data transmission interface 2- > chip 10- > chip 11- > chip 12. Since the transmission cost of the information transmission between the chip and the chip on the same main board is substantially the same as the transmission cost of the information transmission between the chip and the data transmission interface, the transmission path includes 6 times of the transmission of the inter-chip communication medium on the same board and 1 time of the transmission of the inter-board communication medium on the same board, and the transmission cost from the starting chip 1 to the destination chip 12 can be expressed as
C＝6Cb+Cm
And S205, determining the ratio of the transmission cost to the reference single cost as a normalized cost.
The reference single cost may be any constant for normalizing the transmission cost, for example, the reference single cost may be a single cost for data transmission on any communication medium.
According to a possible implementation manner, since the single cost of data transmission on the communication medium between the chips on the same board is the minimum, the single cost of the communication medium between the chips on the same board can be used as a standard to perform normalization processing on the transmission cost of a data packet on a determined transmission path so as to determine the type of the communication medium of the whole data transmission on the transmission path, so that a compression scheme can be selected conveniently. The normalized cost may be expressed as a ratio of the transmission cost to a single cost of the inter-chip communication medium on the same board, and may be expressed as:
NC＝C/Cb。
in order to facilitate comparison of transmission costs among different transmission paths, normalization processing is performed on the transmission costs, so that the transmission costs of different transmission paths can be compared under the same standard, and selection of a suitable compression mode is facilitated.
S206, searching a corresponding compression scheme according to the normalized cost, and determining the compression mode of the data packet according to the compression scheme.
The compression scheme can be understood as a compression mode corresponding to the normalized cost range and including the compression mode.
One possible implementation manner is to determine the ratio of the transmission cost to the single cost of the communication medium between chips on the same board as the normalized cost, and when one normalized cost NC is less than or equal to 1, the transmission can be regarded as the communication within the chip, and since the transmission cost required by the communication within the chip is extremely low and compression is not required, the corresponding compression scheme is uncompressed when NC is less than or equal to 1; when the normalized cost 1 is less than or equal to NC and less than or equal to 10, the transmission is considered to be communication between chips on the same mainboard, the transmission cost required by the communication is low, the data compression in the high compression ratio compression mode can cause the increase of compression and decompression time instead, so that the total delay time is increased, at the moment, the obtained transmission delay can be minimum only by compressing the data packet in the compression mode with a lower compression ratio, and therefore, the corresponding compression scheme is compression with a low compression ratio when the NC is less than or equal to 1 and less than or equal to 10; when the normalized cost is more than 10 and less than or equal to NC and less than or equal to 100, the transmission can be considered as transmission across the mainboard, and because the transmission cost of the transmission across the mainboard is high, a compression mode with a relatively large compression ratio is needed to minimize the total delay time, so that a corresponding compression scheme when the normalized cost is more than or equal to 10 and less than or equal to NC and less than or equal to 100 adopts medium-pressure compression ratio compression; when the normalized cost NC is greater than 100, the transmission can be regarded as remote board crossing, cross-computer equipment or cross-cluster communication, the transmission cost required by the communication is extremely high, transmission delay caused by relatively large data volume transmission is reduced, time consumption for compressing and decompressing the data packet by the compression mode with higher compression ratio is reduced, therefore, the data packet needs to be compressed by higher compression ratio to ensure that the obtained transmission delay is minimum, and the corresponding compression scheme when the NC is greater than 100 is compression with high compression ratio.
Optionally, the low compression ratio, the medium compression ratio, and the high compression ratio in the compression scheme may be preset, and the same compression ratio may be a fixed value or a set of values corresponding to the normalization cost one by one, which is not limited by the embodiment of the present disclosure.
And selecting the compression mode under the corresponding compression scheme according to different normalization costs, so that the data packets of different transmission paths can be compressed and transmitted by adopting different compression ratios, the operation time of data packet compression and decompression is reduced, and the optimization of transmission efficiency and cost is realized. It should be understood that the ratio of the transmission cost to the single cost of other communication mediums can also be determined as the normalized cost, which is not limited by the present disclosure.
S207, compressing the data packet according to the compression mode, and storing the compression mode in the compressed packet after the data packet is compressed.
According to a possible implementation manner, a coefficient in a compression process is adjusted according to a determined compression mode to compress a data packet, and a corresponding compression mode is written in a compressed packet obtained after the data packet is compressed, so that decompression of the compressed packet after the compressed packet is sent to a destination is facilitated. The method for compressing the data packet may include a compression method based on puncturing error redundancy control coded data, an information source compression method, a huffman compression method, and the like, which is not limited in the embodiments of the present disclosure.
Further, when the number of the data packets is multiple, a group of data packets with the same starting place and the same destination are obtained; compressing a group of data packets into the same compressed packet according to a compression mode; and saving the compression mode into the compression packet.
In a possible implementation manner, when the number of the data packets is multiple, for the data packets with the same starting point and destination, in order to save the calculation resources, it is not necessary to perform the compression operation on each data packet. Since the transmission paths of the data packets with the same starting point and destination are the same in the transmission process, the corresponding transmission costs are the same, and correspondingly, the compression mode is also the same. At this time, grouping a plurality of data packets, grouping the data packets with the same starting place and destination into a group, obtaining a compression mode corresponding to one data packet in a group of data packets, uniformly compressing the group of data packets in the compression mode to the same compression packet, and storing the compression mode in the compression packet, thereby realizing the uniform compression and transmission of the data packets in the same transmission path.
According to the technical scheme of the embodiment of the disclosure, the path information between the sending start place and the destination of the data packet is determined, the different communication media through which the data packet is transmitted and the transmission times on the different communication media are determined, the transmission cost of the data packet in the whole sending process is determined according to different single transmission costs on the different communication media, and then the appropriate compression mode is determined according to the transmission cost, so that the total transmission delay time is reduced, the transmission efficiency of the data packet is improved, and the double optimization of the transmission rate and the transmission cost is realized.
Fig. 5 is a schematic structural diagram of a data compression apparatus according to an exemplary embodiment of the present disclosure, where the data compression apparatus includes: an address acquisition module 31, a transmission cost determination module 32 and a data compression module 33.
The address obtaining module 31 is configured to obtain a start location and a destination of data packet transmission; a transmission cost determining module 32, configured to determine a transmission cost for sending the data packet according to the starting location and the destination; the data compression module 33 is configured to determine a compression mode of the data packet according to the transmission cost, compress the data packet according to the compression mode, and store the compression mode in the compressed packet after the data packet is compressed.
The technical scheme of the embodiment solves the problems that the transmission cost of the compression mode with a low compression ratio is high in the long-distance transmission due to the fact that the compression mode with a low compression ratio is adopted in the traditional data communication, and the operation amount is large and the consumed time is long when the compression mode with a high compression ratio is adopted to compress and decompress the data packet, reduces the operation time of the compression and decompression of the data packet, improves the transmission efficiency of the data packet, and realizes double optimization of the transmission rate and the transmission cost.
Optionally, the transmission cost determining module 32 includes:
A path information acquiring unit for acquiring transmission path information between a start location and a destination.
And the path information determining unit is used for determining each communication medium between the starting place and the destination, the transmission times on each communication medium and the single cost of transmission on each communication medium according to the transmission path information.
And the transmission cost determining unit is used for determining the transmission cost of the data packet transmission according to each communication medium, each transmission frequency and each single cost.
Optionally, the data compression module 33 includes:
the compression mode determining unit is used for determining the ratio of the transmission cost to the single cost of the communication medium between chips on the same board as the normalization cost; and searching a corresponding compression scheme according to the normalized cost, and determining the compression mode of the data packet according to the compression scheme.
And the compression unit is used for compressing the data packet according to the compression mode and storing the compression mode in the compressed packet after the data packet is compressed.
Further, when the number of the data packets is multiple, the compressing unit is further configured to: acquiring a group of data packets with the same starting place and the same destination; compressing a group of data packets into the same compressed packet according to a compression mode; and saving the compression mode into the compression packet.
The data compression device provided by the embodiment of the disclosure can execute the data compression method provided by any embodiment of the disclosure, and has corresponding functional modules and beneficial effects of the execution method.
Fig. 6 is a schematic structural diagram of an apparatus according to an exemplary embodiment of the present disclosure, as shown in fig. 6, the apparatus includes a processor 41, a storage device 42, an input device 43, and an output device 44; the number of processors 41 in the device may be one or more, and one processor 41 is taken as an example in fig. 6; the processor 41, the storage means 42, the input means 43 and the output means 44 in the device may be connected by a bus or other means, as exemplified by the bus connection in fig. 6.
The storage device 42, which is a computer-readable storage medium, may be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules (e.g., the address acquisition module 31, the transmission cost determination module 32, and the data compression module 33) corresponding to the data compression method in the embodiments of the present disclosure. The processor 41 executes various functional applications of the apparatus and data processing by executing software programs, instructions, and modules stored in the storage device 42, that is, implements the above-described data compression method.
The storage device 42 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the storage 42 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, storage 42 may further include memory located remotely from processor 41, which may be connected to the device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input means 43 may be used to receive input numeric or character information and to generate key signal inputs relating to user settings and function control of the device, and may include a touch screen, a keyboard, a mouse, and the like. The output device 44 may include a display device such as a display screen.
An exemplary embodiment of the present disclosure also provides a storage medium containing computer-executable instructions which, when executed by a computer processor, perform a method of data compression, the method comprising:
Acquiring a starting place and a destination of data packet transmission;
determining the transmission cost of data packet transmission according to the starting place and the destination;
and determining a compression mode of the data packet according to the transmission cost, compressing the data packet according to the compression mode, and storing the compression mode in the compressed packet after the data packet is compressed.
Of course, the storage medium provided by the embodiments of the present disclosure contains computer-executable instructions, and the computer-executable instructions are not limited to the method operations described above, and may also perform related operations in the data compression method provided by any embodiment of the present disclosure.
From the above description of the embodiments, it is obvious for a person skilled in the art that the present disclosure can be implemented by software and necessary general hardware, and certainly can be implemented by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present disclosure may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present disclosure.
It should be noted that, in the embodiment of the above search apparatus, each included unit and module are merely divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only used for distinguishing one functional unit from another, and are not used for limiting the protection scope of the present disclosure.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present disclosure and the technical principles employed. Those skilled in the art will appreciate that the present disclosure is not limited to the particular embodiments described herein, and that various obvious changes, adaptations, and substitutions are possible, without departing from the scope of the present disclosure. Therefore, although the present disclosure has been described in greater detail with reference to the above embodiments, the present disclosure is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present disclosure, the scope of which is determined by the scope of the appended claims.
Claims (10)
1. A method of data compression, comprising:
acquiring a starting place and a destination of data packet transmission;
determining the transmission cost of the data packet transmission according to the starting place and the destination;
And determining a compression mode of the data packet according to the transmission cost, compressing the data packet according to the compression mode, and storing the compression mode in a compressed packet after the data packet is compressed.
2. The method of claim 1, wherein determining the transmission cost for the data packet transmission according to the origination and the destination comprises:
acquiring transmission path information between the starting place and the destination;
determining each communication medium between the starting place and the destination, the transmission times of each communication medium and the single cost of transmission on each communication medium according to the transmission path information;
and determining the transmission cost of the data packet transmission according to each communication medium, each transmission number and each single cost.
3. The method of claim 2, wherein determining the transmission cost for the data packet transmission according to the communication media, the transmission times and the single cost comprises:
determining a product of the transmission times and the single cost in each communication medium in the transmission path information;
And taking the sum of the products of the transmission times and the single cost in each communication medium as the transmission cost of the data packet transmission.
4. The method of claim 1, wherein determining the compressed mode for the data packet based on the transmission cost comprises:
determining the ratio of the transmission cost to a reference single cost as a normalized cost;
and searching a corresponding compression scheme according to the normalized cost, and determining the compression mode of the data packet according to the compression scheme.
5. The method according to any of claims 1 to 4, wherein the communication medium comprises: and at least one of a co-board inter-chip communication medium, a co-board inter-board communication medium, a co-cluster inter-machine communication medium, and a remote communication medium.
6. The method according to claim 1, wherein when the number of the data packets is plural, said compressing the data packets according to the compression mode and storing the compression mode in the compressed packets after the data packets are compressed comprises:
acquiring a group of data packets with the same starting place and the same destination;
compressing the group of data packets into the same compressed packet according to the compression mode;
Saving the compression mode to the compression packet.
7. A data compression apparatus, comprising:
the acquisition module is used for acquiring the starting place and the destination of the data packet transmission;
a transmission cost determining module, configured to determine a transmission cost for sending the data packet according to the starting location and the destination;
and the compression module is used for determining a compression mode of the data packet according to the transmission cost, compressing the data packet according to the compression mode, and storing the compression mode in a compressed packet after the data packet is compressed.
8. The apparatus of claim 7, wherein the transmission cost determination module comprises:
a path information acquiring unit for acquiring transmission path information between the origin and the destination;
a path information determining unit, configured to determine, according to the transmission path information, each communication medium between the origin and the destination, a number of transmissions on each communication medium, and a single cost of transmission on each communication medium;
and a transmission cost determining unit, configured to determine, according to each of the communication media, each of the transmission times and each of the single costs, a transmission cost for transmitting the data packet.
9. An apparatus, characterized in that the apparatus comprises:
one or more processors;
storage means for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a method of data compression as claimed in any one of claims 1-6.
10. A storage medium containing computer-executable instructions for performing the method of data compression of any one of claims 1-6 when executed by a computer processor.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010674720.1A CN111865326B (en) | 2020-07-14 | 2020-07-14 | Data compression method, device, equipment and storage medium |
| PCT/CN2021/106218 WO2022012581A1 (en) | 2020-07-14 | 2021-07-14 | Data compression and transmission methods and apparatuses, device, many-core system, and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010674720.1A CN111865326B (en) | 2020-07-14 | 2020-07-14 | Data compression method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111865326A true CN111865326A (en) | 2020-10-30 |
| CN111865326B CN111865326B (en) | 2023-06-23 |
Family
ID=72984227
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010674720.1A Active CN111865326B (en) | 2020-07-14 | 2020-07-14 | Data compression method, device, equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111865326B (en) |
| WO (1) | WO2022012581A1 (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113067674A (en) * | 2021-03-17 | 2021-07-02 | 北京智慧星光信息技术有限公司 | Transmission method, system, electronic device and storage medium of network text data |
| WO2022012581A1 (en) * | 2020-07-14 | 2022-01-20 | 北京灵汐科技有限公司 | Data compression and transmission methods and apparatuses, device, many-core system, and medium |
| WO2023246190A1 (en) * | 2022-06-24 | 2023-12-28 | 中兴通讯股份有限公司 | Data stream compression method, network device and storage medium |
Citations (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5805932A (en) * | 1994-04-22 | 1998-09-08 | Sony Corporation | System for transmitting compressed data if compression ratio is at least preset ratio and pre-compressed data if compression ratio is less than preset ratio |
| US20060077412A1 (en) * | 2004-10-11 | 2006-04-13 | Vikram Phogat | Method and apparatus for optimizing data transmission costs |
| CN101383616A (en) * | 2007-09-07 | 2009-03-11 | 三星电子株式会社 | Data compression apparatus and method |
| US20100020682A1 (en) * | 2006-09-12 | 2010-01-28 | Yamaha Corporation | Communication device, communication method, and recording medium |
| US20100103183A1 (en) * | 2008-10-23 | 2010-04-29 | Hung-Ming Lin | Remote multiple image processing apparatus |
| US20130088373A1 (en) * | 2011-10-07 | 2013-04-11 | Fujitsu Limited | Data compression apparatus and data compression method |
| CN103957582A (en) * | 2014-05-17 | 2014-07-30 | 浙江大学宁波理工学院 | Wireless sensor network self-adaptation compression method |
| CN104168603A (en) * | 2013-05-15 | 2014-11-26 | 上海交通大学 | Compressed sensing technology-based construction method of data gathering tree with high-energy efficiency |
| CN104866381A (en) * | 2014-02-20 | 2015-08-26 | 联想(北京)有限公司 | Information processing method and first electronic equipment |
| WO2015173889A1 (en) * | 2014-05-13 | 2015-11-19 | 株式会社日立製作所 | Storage device |
| US9521218B1 (en) * | 2016-01-21 | 2016-12-13 | International Business Machines Corporation | Adaptive compression and transmission for big data migration |
| CN109783033A (en) * | 2019-01-31 | 2019-05-21 | 上海赜睿信息科技有限公司 | A kind of date storage method and electronic equipment suitable for heterogeneous system |
| CN109921801A (en) * | 2019-03-01 | 2019-06-21 | 迅雷计算机（深圳）有限公司 | A kind of data compression and decompression method, device, system and data processing system |
| CN110769453A (en) * | 2019-10-31 | 2020-02-07 | 重庆大学 | Multi-modal monitoring data dynamic compression control method under unstable network environment |
| CN110891092A (en) * | 2019-12-09 | 2020-03-17 | 北京布袋森林科技有限责任公司 | Data compression transmission method and device |
| CN111178540A (en) * | 2019-12-29 | 2020-05-19 | 浪潮(北京)电子信息产业有限公司 | Training data transmission method, device, equipment and medium |
| CN111210481A (en) * | 2020-01-10 | 2020-05-29 | 大连理工大学 | Depth estimation acceleration method of multiband stereo camera |
| CN111262876A (en) * | 2020-01-22 | 2020-06-09 | 腾讯科技（深圳）有限公司 | Data processing method, device and equipment based on block chain and storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018086972A1 (en) * | 2016-11-08 | 2018-05-17 | Koninklijke Philips N.V. | Method for wireless data transmission range extension |
| CN106452920B (en) * | 2016-11-24 | 2019-07-19 | 广东电网有限责任公司电力调度控制中心 | Method and device for layout of grid data nodes based on cost optimization |
| CN110391873B (en) * | 2018-04-20 | 2022-02-11 | 伊姆西Ip控股有限责任公司 | Method, apparatus and computer program product for determining a data transfer mode |
| CN111865326B (en) * | 2020-07-14 | 2023-06-23 | 北京灵汐科技有限公司 | Data compression method, device, equipment and storage medium |
-
2020
- 2020-07-14 CN CN202010674720.1A patent/CN111865326B/en active Active
-
2021
- 2021-07-14 WO PCT/CN2021/106218 patent/WO2022012581A1/en active Application Filing
Patent Citations (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5805932A (en) * | 1994-04-22 | 1998-09-08 | Sony Corporation | System for transmitting compressed data if compression ratio is at least preset ratio and pre-compressed data if compression ratio is less than preset ratio |
| US20060077412A1 (en) * | 2004-10-11 | 2006-04-13 | Vikram Phogat | Method and apparatus for optimizing data transmission costs |
| US20100020682A1 (en) * | 2006-09-12 | 2010-01-28 | Yamaha Corporation | Communication device, communication method, and recording medium |
| CN101383616A (en) * | 2007-09-07 | 2009-03-11 | 三星电子株式会社 | Data compression apparatus and method |
| US20100103183A1 (en) * | 2008-10-23 | 2010-04-29 | Hung-Ming Lin | Remote multiple image processing apparatus |
| US20130088373A1 (en) * | 2011-10-07 | 2013-04-11 | Fujitsu Limited | Data compression apparatus and data compression method |
| CN104168603A (en) * | 2013-05-15 | 2014-11-26 | 上海交通大学 | Compressed sensing technology-based construction method of data gathering tree with high-energy efficiency |
| CN104866381A (en) * | 2014-02-20 | 2015-08-26 | 联想(北京)有限公司 | Information processing method and first electronic equipment |
| WO2015173889A1 (en) * | 2014-05-13 | 2015-11-19 | 株式会社日立製作所 | Storage device |
| CN103957582A (en) * | 2014-05-17 | 2014-07-30 | 浙江大学宁波理工学院 | Wireless sensor network self-adaptation compression method |
| US9521218B1 (en) * | 2016-01-21 | 2016-12-13 | International Business Machines Corporation | Adaptive compression and transmission for big data migration |
| CN109783033A (en) * | 2019-01-31 | 2019-05-21 | 上海赜睿信息科技有限公司 | A kind of date storage method and electronic equipment suitable for heterogeneous system |
| CN109921801A (en) * | 2019-03-01 | 2019-06-21 | 迅雷计算机（深圳）有限公司 | A kind of data compression and decompression method, device, system and data processing system |
| CN110769453A (en) * | 2019-10-31 | 2020-02-07 | 重庆大学 | Multi-modal monitoring data dynamic compression control method under unstable network environment |
| CN110891092A (en) * | 2019-12-09 | 2020-03-17 | 北京布袋森林科技有限责任公司 | Data compression transmission method and device |
| CN111178540A (en) * | 2019-12-29 | 2020-05-19 | 浪潮(北京)电子信息产业有限公司 | Training data transmission method, device, equipment and medium |
| CN111210481A (en) * | 2020-01-10 | 2020-05-29 | 大连理工大学 | Depth estimation acceleration method of multiband stereo camera |
| CN111262876A (en) * | 2020-01-22 | 2020-06-09 | 腾讯科技（深圳）有限公司 | Data processing method, device and equipment based on block chain and storage medium |
Non-Patent Citations (4)
| Title |
|---|
| YUHUA BAI: "An improved method for lossless data compression", 《DATA COMPRESSION CONFERENCE》 * |
| 刘慧: "面向大数据的列存储数据库关键技术研究", 《中国优秀博硕士学位论文全文数据库(博士)信息科技辑》 * |
| 樊宽刚: "《数据感知优化技术及其应用》", 31 August 2018 * |
| 黄金: "无线传感器网络中能量有效的压缩算法及性能评估研究", 《中国优秀硕士学位论文全文数据库 (信息科技辑)》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022012581A1 (en) * | 2020-07-14 | 2022-01-20 | 北京灵汐科技有限公司 | Data compression and transmission methods and apparatuses, device, many-core system, and medium |
| CN113067674A (en) * | 2021-03-17 | 2021-07-02 | 北京智慧星光信息技术有限公司 | Transmission method, system, electronic device and storage medium of network text data |
| WO2023246190A1 (en) * | 2022-06-24 | 2023-12-28 | 中兴通讯股份有限公司 | Data stream compression method, network device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111865326B (en) | 2023-06-23 |
| WO2022012581A1 (en) | 2022-01-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111865326B (en) | Data compression method, device, equipment and storage medium | |
| CN102801784B (en) | A kind of distributed data storage method and equipment | |
| CN113961510B (en) | File processing method, device, equipment and storage medium | |
| CN108595483B (en) | Data processing method and related device | |
| CN111327677B (en) | Industrial Internet of things resource scheduling system and method based on edge calculation | |
| CN103716056B (en) | Data compression method, uncompressing data and equipment | |
| CN108965333A (en) | A kind of data compression, decompression method, system and electronic equipment | |
| CN114707651A (en) | A method, device, device and medium for topology selection of reduction operation | |
| CN114661726A (en) | Data type mapping processing method and its device, equipment, medium and product | |
| CN110619008A (en) | Database query method and device, electronic equipment and storage medium | |
| CN116226139A (en) | A method and system for distributed storage and processing of large-scale ocean data | |
| CN114916033A (en) | Data transmission method, system, electronic equipment and storage medium | |
| CN115567589A (en) | Compression transmission method, device, equipment and storage medium of JSON data | |
| CN113220651B (en) | Method, device, terminal equipment and storage medium for compressing operation data | |
| CN104077272A (en) | Method and device for compressing dictionary | |
| WO2020024392A1 (en) | Node processing method and apparatus, storage medium and electronic device | |
| CN107643906A (en) | Data processing method and device | |
| CN104636432A (en) | Method and device for journal file compression and decompression | |
| CN114465718B (en) | Multi-protocol translation method and related equipment for quantum key distribution business | |
| CN110727391A (en) | Transaction data storage method and device, blockchain system, medium and electronic device | |
| CN112395245B (en) | Access device and method of processor and computer equipment | |
| KR20230049526A (en) | Apparatus for determining of tree, network switch apparatus, system of in-network data aggregation including the same and method thereof | |
| CN114329161B (en) | Data query method, device and electronic equipment | |
| CN111092943B (en) | Multi-cluster remote sensing method and system of tree structure and electronic equipment | |
| CN116881310B (en) | Method and device for calculating set of big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |