CN111865326B - Data compression method, device, equipment and storage medium - Google Patents
Data compression method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111865326B CN111865326B CN202010674720.1A CN202010674720A CN111865326B CN 111865326 B CN111865326 B CN 111865326B CN 202010674720 A CN202010674720 A CN 202010674720A CN 111865326 B CN111865326 B CN 111865326B
- Authority
- CN
- China
- Prior art keywords
- transmission
- data packet
- cost
- compression
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
- H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
- H03M7/60—General implementation details not specific to a particular type of compression
- H03M7/6047—Power optimization with respect to the encoder, decoder, storage or transmission
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The embodiment of the disclosure discloses a data compression method, a device, equipment and a storage medium. The method comprises the following steps: acquiring a starting place and a destination of data packet transmission; determining the transmission cost of data packet transmission according to the starting place and the destination; and determining a compression mode of the data packet according to the transmission cost, compressing the data packet according to the compression mode, and storing the compression mode in the compressed data packet after the data packet is compressed. According to the technical scheme, the problems that a compression mode with a low compression ratio is high in transmission cost in long-distance transmission caused by adopting a uniform compression mode for compression in traditional data communication, and the operation amount is large and the time consumption is long when a data packet is compressed and decompressed by adopting the compression mode with a high compression ratio are solved, the operation time of data packet compression and decompression is shortened, the transmission efficiency of the data packet is improved, and the double optimization of the transmission rate and the transmission cost is realized.
Description
Technical Field
The embodiment of the disclosure relates to the technical field of data compression, in particular to a data compression method, a device, equipment and a storage medium.
Background
The many-core system consists of one or more multi-core processors to solve the problem that a single-core chip has low speed when performing large data amount calculation and generates high heat, wherein the multi-core processor is characterized in that a plurality of complete calculation engines (cores) are integrated on one processor, and the cores in one processor chip or among the processor chips can work cooperatively. For many-core systems, data needs to be communicated and interacted between multiple cores and multiple chips.
In the conventional method, a unified compression coding scheme is often adopted for data communication between chips, that is, data packets for data communication are compressed and transmitted in a unified compression mode. However, the data communication method adopting the unified compression mode for compression and transmission does not consider that the transmission cost of the data in the transmission process is changed according to the difference of transmission media and the difference of transmission distances, and the double optimization of the transmission efficiency and the transmission cost cannot be achieved.
Disclosure of Invention
The disclosure provides a data compression method, a device, equipment and a storage medium, so as to realize the selection of a data compression mode and realize the double optimization of transmission rate and transmission cost.
In a first aspect, an embodiment of the present disclosure provides a data compression method, including:
acquiring a starting place and a destination of data packet transmission;
determining the transmission cost of data packet transmission according to the starting place and the destination;
and determining a compression mode of the data packet according to the transmission cost, compressing the data packet according to the compression mode, and storing the compression mode in the compressed data packet after the data packet is compressed.
Further, determining a transmission cost of the data packet according to the origin and the destination includes:
acquiring transmission path information between an origin and a destination;
determining each communication medium between an origin and a destination according to the transmission path information, the number of transmission times on each communication medium and the single cost of transmission on each communication medium;
and determining the transmission cost of data packet transmission according to each transmission frequency and each single cost according to each communication medium.
Further, determining, according to each communication medium, each transmission time and each single cost, a transmission cost of sending the data packet, including:
determining the product of the transmission times and the single cost in each communication medium in the transmission path information;
and taking the sum of products of the transmission times and the single cost in each communication medium as the transmission cost of data packet transmission.
Further, determining a compression mode of the data packet according to the transmission cost includes:
determining the ratio of the transmission cost to the reference single cost as a normalized cost;
and searching a corresponding compression scheme according to the normalized cost, and determining the compression mode of the data packet according to the compression scheme.
Further, the communication medium includes: at least one of inter-chip communication medium, inter-board communication medium, inter-cluster communication medium and remote communication medium.
Further, when the number of datagrams is plural, the data packet is compressed according to the compression mode, and the compression mode is stored in the compressed packet after the data packet is compressed, including:
acquiring a group of data packets with the same starting place and the same destination;
compressing a group of data packets into the same compression packet according to a compression mode;
the compressed mode is saved to the compressed package.
In a second aspect, embodiments of the present disclosure further provide a data compression apparatus, including:
the acquisition module is used for acquiring the starting place and the destination of the data packet transmission;
the transmission cost determining module is used for determining the transmission cost of data packet transmission according to the starting place and the destination;
the compression module is used for determining the compression mode of the data packet according to the transmission cost, compressing the data packet according to the compression mode, and storing the compression mode in the compressed data packet after the data packet is compressed.
Further, the transmission cost determining module includes:
a path information acquisition unit configured to acquire transmission path information between a start point and a destination;
a path information determining unit configured to determine each communication medium between the origin and the destination, the number of transmissions on each communication medium, and a single cost of transmission on each communication medium, based on the transmission path information;
and the transmission cost determining unit is used for determining the transmission cost of the data packet transmission according to each communication medium, each transmission time and each single cost.
In a third aspect, embodiments of the present disclosure further provide an apparatus, the apparatus comprising:
one or more processors;
a storage means for storing one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the data compression method as provided in any embodiment of the present disclosure.
In a fourth aspect, the presently disclosed embodiments also provide a storage medium containing computer-executable instructions for performing a data compression method as provided by any of the embodiments of the present disclosure when executed by a computer processor.
The embodiment of the disclosure obtains the starting place and the destination of data packet transmission; determining the transmission cost of data packet transmission according to the starting place and the destination; and determining a compression mode of the data packet according to the transmission cost, compressing the data packet according to the compression mode, and storing the compression mode in the compressed data packet after the data packet is compressed. The method comprises the steps of obtaining the starting place and the destination of data packet transmission, determining the transmission distance and the transmission medium which need to pass in the data packet transmission process according to the starting place and the destination, further determining the transmission cost of the data packet which needs to be transmitted to the destination from the starting place, selecting a proper data packet compression mode according to the transmission cost to compress the data packet, solving the problems that the transmission cost is high in long-distance transmission due to the adoption of a low compression mode with a uniform compression ratio in the traditional data communication, and the operation amount is large and the time consumption is long when the data packet is compressed and decompressed due to the adoption of the compression mode with a high compression ratio, reducing the operation time of the data packet compression and decompression, improving the transmission efficiency of the data packet, and realizing the double optimization of the transmission rate and the transmission cost.
Drawings
FIG. 1 is a flow chart of a data compression method in an exemplary embodiment of the present disclosure;
FIG. 2 is a flow chart of a method of data compression in an exemplary embodiment of the present disclosure;
FIG. 3 is a flow chart of determining a transmission cost for a data packet transmission based on a number of transmissions and a single cost for each communication medium in an exemplary embodiment of the present disclosure;
FIG. 4 is an exemplary diagram of a packet transmission process in an exemplary embodiment of the present disclosure;
FIG. 5 is a schematic diagram of a data compression device in an exemplary embodiment of the present disclosure;
fig. 6 is a schematic structural view of an apparatus in an exemplary embodiment of the present disclosure.
Detailed Description
The present disclosure is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the present disclosure and not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present disclosure are shown in the drawings. Furthermore, embodiments of the present disclosure and features of embodiments may be combined with each other without conflict.
Fig. 1 is a flowchart of a data compression method according to an exemplary embodiment of the present disclosure, where the method may be applied to a case where data is compressed between a plurality of different computing engines of the same chip or before transmission between the different computing engines of the plurality of chips, and the method may be performed by a data compression device, where the method includes the following steps:
s101, acquiring the starting place and the destination of data packet transmission.
A data packet is understood to be a unit of data in a communication transmission, and when a computing engine needs to transmit a message or a set of data to another computing engine, the message or the set of data may be divided into a plurality of data blocks, which are called data packets, and include address information of a receiver, where each data packet is transmitted to an address of the receiver along different paths according to the address information.
One possible implementation manner, at the start of the routing packet, determines, as a destination, a computing engine corresponding to the address information of the receiver in the data packet by reading the address information of the receiver in the data packet. To determine the transmission route of the data packet, the determination of the start point and the destination of the data packet can be achieved by reading the address information of the receiver in the data packet, so as to determine the transmission route of the data packet according to the start point and the destination.
S102, determining the transmission cost of the data packet according to the starting place and the destination.
The transmission cost is understood to be the transmission delay and the transmission energy consumption of data in the process of being transmitted from one computing engine to another computing engine or from a chip where one computing engine is located to another computing engine, namely the time spent for transmission and the electric energy consumption caused by transmission. The transmission cost is related to the data quantity and the transmission distance, the transmission cost is higher as the data quantity is larger, meanwhile, the transmission cost is related to the medium experienced in the transmission process, the transmission distance is longer, the high energy consumption or the high delay medium experienced in the transmission process is more, and the transmission cost is higher.
One possible implementation manner is to determine a transmission node experienced in the process of transmitting a data packet from an initial computing engine to a destination computing engine according to the acquired initial destination and destination, acquire a single transmission cost of performing single data transmission between each adjacent transmission node, and take the sum of the single transmission costs between each transmission node in the transmission process as the transmission cost of the data packet, wherein the transmission node can be understood as a routing node experienced in the transmission process, and the transmission node for calculating the transmission cost comprises the initial computing engine corresponding to the initial destination and the destination computing engine corresponding to the destination.
Determining a transmission route of the data packet according to the starting place and the destination, and determining the transmission cost of the data packet according to the transmission nodes in the transmission route so as to select a proper compression mode according to the determined transmission cost.
S103, determining a compression mode of the data packet according to the transmission cost, compressing the data packet according to the compression mode, and storing the compression mode in the compressed data packet after the data packet is compressed.
The compression mode can be understood as a compressed scheme name and parameter list, which can be used for measuring the compression efficiency of data compression, and can be determined according to the required compression ratio, wherein one compression mode corresponds to one compression ratio.
In one possible implementation, the type of transmission experienced by the data packet from the start to the destination may be determined according to the transmission cost, and when the transmission cost is low, the delay and the energy consumption in the transmission process of the data packet are low, and if the data packet is compressed by adopting the compression mode with a larger compression ratio, the data compression and decompression time is increased, so when the transmission cost is low, the data packet can be transmitted with a lower compression ratio or without compression. Meanwhile, as the transmission cost increases, the time delay and the energy consumption in the data packet transmission process increase, and at the moment, a compression mode with a higher compression ratio is required to be selected to reduce the transmitted data volume, so that the transmission cost is reduced. Therefore, the compression ratio with the shortest sum of the transmission delay and the compression time can be determined according to the transmission cost and used as the determined compression ratio information, the corresponding compression mode is selected according to the determined compression ratio information to compress the data packet, and the compression mode is stored in the compressed data packet after the data packet is compressed so as to perform decompression operation on the compressed data packet at the destination.
The embodiment of the disclosure obtains the starting place and the destination of data packet transmission; determining the transmission cost of data packet transmission according to the starting place and the destination; and determining a compression mode of the data packet according to the transmission cost, compressing the data packet according to the compression mode, and storing the compression mode in the compressed data packet after the data packet is compressed. The method comprises the steps of obtaining the starting place and the destination of data packet transmission, determining the transmission distance and the transmission medium which need to pass in the data packet transmission process according to the starting place and the destination, further determining the transmission cost of the data packet which needs to be transmitted to the destination from the starting place, selecting a proper data packet compression mode according to the transmission cost to compress the data packet, solving the problems that the transmission cost is high in long-distance transmission due to the adoption of a low compression mode with a uniform compression ratio in the traditional data communication, and the operation amount is large and the time consumption is long when the data packet is compressed and decompressed due to the adoption of the compression mode with a high compression ratio, reducing the operation time of the data packet compression and decompression, improving the transmission efficiency of the data packet, and realizing the double optimization of the transmission rate and the transmission cost.
Fig. 2 is a flowchart of a data compression method according to an exemplary embodiment of the present disclosure. The technical scheme of the embodiment is further refined on the basis of the technical scheme, and the method can comprise the following steps:
s201, acquiring the starting place and the destination of data packet transmission.
S202, acquiring transmission path information between a starting place and a destination.
The transmission path information is understood to be the transmission node information experienced in the transmission of a data packet from the origin to the destination. The transmission path information may be used to determine communication medium information experienced in a transmission process, transmission frequency information on each communication medium, and the like, where the transmission node may include a chip corresponding to a destination at an origin in the transmission process, a chip passed in the transmission process, and a data transmission interface on a motherboard when data needs to be propagated across the motherboard.
When the inter-chip position relationship and the communication connection relationship are determined, the logic position relationship among the chips is also determined, and the path required to pass through the chip and/or the data transmission interface when one chip sends data to the other chip is also determined, that is, the logic position relationship between the corresponding chip at the start point and the corresponding chip at the destination and the path required to pass through the chip and/or the data transmission interface when data transmission is performed are determined. And determining the communication medium information between adjacent chips on the path to obtain the type of the communication medium which needs to be passed by one time of data transmission and the times of data transmission on various types of communication media, and taking the communication medium type and the times information as transmission path information between an origin and a destination.
Alternatively, the determination of the transmission path information may be determined by searching a preset address information table, or may be determined by configuring a preset route, which is not limited in the embodiment of the present disclosure.
S203, determining each communication medium between the starting place and the destination according to the transmission path information, the transmission times on each communication medium and the single cost of the transmission on each communication medium.
Communication media may be understood as transmission media through which data packets travel during transmission, and may include wired media and wireless media. In an embodiment of the present disclosure, the communication medium may include: at least one of inter-chip communication medium, inter-board communication medium, inter-cluster communication medium and remote communication medium.
The single cost is understood to mean the transmission time and transmission effort required for a data transmission on a medium, a known constant obtained from the device or from a design estimate.
In one possible implementation manner, the chip where the computing engine is located may be mounted on different mainboards, and a plurality of mainboards may be combined to form a computer device, and a plurality of computer devices may be integrated to obtain a computer cluster. Determining a communication medium between an origin and a destination based on the transmission path information may include: the communication medium for the chips and the data transmission interfaces on the same main board is the communication medium between the chips of the same board; the communication medium for communicating the data transmission interfaces between different mainboards on the same computer equipment is the communication medium between the same mainboards; the communication medium for data communication between different computer devices in the same computer cluster is the communication medium between the same cluster machines; and the communication medium in which data communication is performed to a device further away is a long-range communication medium. The number of transmissions on the same type of communication medium and the transmission cost required for one transmission on each communication medium can be determined from the transmission path information.
According to the transmission path information, each communication medium between the origin and the destination and the single cost of transmission on each communication medium are determined, so that the transmission cost required by the transmission of the data packet on different media can be definitely determined, and further the determination of the transmission cost when the data packet propagates on different transmission paths is realized.
S204, determining the transmission cost of the data packet according to each transmission time and each single cost according to each communication medium.
One possible implementation manner may determine a transmission cost on the same type of communication medium according to each transmission number and each single cost, and may determine a transmission cost of the data packet in the whole transmission process according to a corresponding transmission cost on each communication medium.
Further, fig. 3 is a flowchart of determining a transmission cost of a data packet according to each transmission time and each single cost according to each communication medium according to an exemplary embodiment of the present disclosure, which may include the following steps:
s2041, determining the product of the number of transmissions in each communication medium in the transmission path information and the single cost.
In one possible implementation manner, during a data transmission process, the transmission on the same communication medium may not be continuous, but because the single cost required for performing a single transmission on the same communication medium is the same, the product of the number of times of transmission in the same communication medium in the transmission path information and the single cost is used as the transmission cost required for performing data transmission on the corresponding transmission path by the communication medium, and the transmission cost required for performing data transmission on the transmission path is calculated for each communication medium in the transmission path information.
And S2042, taking the sum of products of the transmission times and the single cost in each communication medium as the transmission cost of data packet transmission.
One possible implementation way, if in one data transmission process, the number of times of data transmission is Nb on the communication medium between the chips of the same board, and the single cost of transmission is Cb; the number of times of data transmission on the communication medium between the same boards is Nm, and the single cost of transmission is Cm; the number of times of data transmission on the communication medium between the same cluster machines is Nc, and the single cost of transmission is Cc; the number of data transmissions over the remote communication medium is Nr and the single cost of the transmission is Cr. The transmission cost of the data packet transmission in the data transmission process can be expressed as:
C＝Nb*Cb+Nm*Cm+Nc*Cc+Nr*Cr
fig. 4 is an exemplary diagram of a packet transmission process according to an exemplary embodiment of the present disclosure, and as can be seen from fig. 4, a transmission path of a packet transmitted from a chip 1 to a chip 12 is: chip 1- > chip 2- > chip 3- > data transfer interface 1- > data transfer interface 2- > chip 10- > chip 11- > chip 12. Since the transmission cost of the information transmission between the chips on the same motherboard is substantially the same as the transmission cost of the information transmission between the chips and the data transmission interface, the transmission path includes 6 transmissions of the communication medium between the chips on the same motherboard and 1 transmission of the communication medium between the chips on the same motherboard, and the transmission cost from the origin chip 1 to the destination chip 12 can be expressed as
C＝6Cb+Cm
S205, determining the ratio of the transmission cost to the reference single cost as a normalized cost.
The reference single cost may be any constant used to normalize the transmission cost, for example, a single cost of data transmission over any communication medium.
In one possible implementation manner, since the single cost of data transmission on the communication medium between the same-board chips is minimum, the transmission cost of the data packet on the determined transmission path can be normalized by taking the single cost of the communication medium between the same-board chips as a standard, so as to determine the type of the communication medium of the whole data transmission on the transmission path, and thus, the compression scheme can be selected conveniently. Where the normalized cost may be expressed as a ratio of the transmission cost to the single cost of the on-chip communication medium, e.g., as:
NC＝C/Cb。
in order to facilitate comparison of transmission costs between different transmission paths, normalization processing is performed on the transmission costs, so that the transmission costs of different transmission paths can be compared under the same standard, and selection of a suitable compression mode is facilitated.
S206, searching a corresponding compression scheme according to the normalized cost, and determining the compression mode of the data packet according to the compression scheme.
The compression scheme can be understood as a compression mode corresponding to the normalized cost range and comprising a compression mode.
One possible implementation manner is to take the ratio of the transmission cost to the single cost of the communication medium between the same board chips as an example as the normalization cost, when the normalization cost NC is less than or equal to 1, the transmission can be considered as communication in the chip, and the transmission cost required by the communication in the chip is extremely low and compression is not needed, so that the compression scheme corresponding to NC is not compressed when NC is less than or equal to 1; when the normalization cost is 1 < NC and less than or equal to 10, the transmission can be considered to be communication between chips on the same main board, at the moment, the transmission cost required by communication is lower, the data compression in the compression mode with high compression ratio can cause the increase of compression and decompression time instead, so that the total delay time is increased, at the moment, the obtained transmission delay is minimum only by compressing the data packet in the compression mode with lower compression ratio, and therefore, the compression scheme corresponding to the NC of 1 < NC and less than or equal to 10 adopts low compression ratio compression; when the normalization cost is more than 10 and less than or equal to 100, the transmission can be regarded as transmission across the main board, and because the transmission cost of the transmission across the main board is higher, a compression mode with relatively larger compression ratio is needed to minimize the total delay time, so that the corresponding compression scheme is compression with a medium compression ratio when more than 10 and less than or equal to 100; when the normalized cost NC is greater than 100, the transmission can be regarded as remote cross-board, cross-computer equipment or cross-cluster communication, at this time, the transmission cost required by communication is extremely high, and the time for compressing and decompressing the data packet in a compression mode with a higher compression ratio is less, so that the data packet is compressed with a higher compression ratio to minimize the obtained transmission delay, and the corresponding compression scheme is compression with a high compression ratio when NC is greater than 100.
Optionally, the low compression ratio, the medium compression ratio, and the high compression ratio in the compression scheme may be preset, and the same compression ratio may be a fixed value or a set of values corresponding to normalized costs one to one, which is not limited by the embodiment of the present disclosure.
The compression modes under the corresponding compression schemes are selected according to different normalization costs, so that data packets of different transmission paths can be compressed by adopting different compression ratios and then sent, the operation time of data packet compression and decompression is reduced, and the optimization of transmission efficiency and cost is realized. It should be appreciated that it is also possible to determine the ratio of the transmission cost to the single cost of the other communication medium as a normalized cost, which is not limited by the present disclosure.
S207, compressing the data packet according to the compression mode, and storing the compression mode in the compressed data packet.
One possible implementation manner is to adjust the coefficient in the compression process according to the determined compression mode so as to compress the data packet, and write the corresponding compression mode into the compressed packet obtained after the data packet is compressed so as to facilitate the decompression work of the compressed packet after the compressed packet is sent to the destination. The method for compressing the data packet may include a compression method based on puncturing the data after the error redundancy control encoding, a source compression method, a huffman compression method, and the like, which is not limited in the embodiments of the present disclosure.
Further, when the number of the data packets is a plurality of, a group of data packets with the same starting place and the same destination is obtained; compressing a group of data packets into the same compression packet according to a compression mode; the compressed mode is saved to the compressed package.
In one possible implementation manner, when the number of data packets is multiple, for the data packets with the same starting point and destination, in order to save operation resources, it is not necessary to perform compression operation on each data packet. Because the transmission paths of the data packets with the same starting point and destination are the same in the transmission process, the corresponding transmission cost is the same, and the corresponding compression modes are the same. At this time, grouping operation is performed on the plurality of data packets, the data packets with the same starting point and destination are grouped into a group, a compression mode corresponding to one data packet in the group of data packets is obtained, the group of data packets are uniformly compressed in the compression mode, the data packets are compressed into the same compression packet, and the compression mode is stored in the compression packet, so that uniform compression and transmission of the data packets with the same transmission path are realized.
According to the technical scheme, the path information between the data packet transmission starting place and the destination is determined, different communication media through which the data packet is transmitted and the transmission times on the different communication media are determined, the transmission cost of the data packet in the whole transmission process is determined according to different single transmission costs on the different communication media, and then the proper compression mode is determined according to the transmission cost, so that the total delay time of transmission is reduced, the transmission efficiency of the data packet is improved, and double optimization of the transmission rate and the transmission cost is realized.
Fig. 5 is a schematic structural diagram of a data compression device according to an exemplary embodiment of the present disclosure, where the data compression device includes: the address acquisition module 31, the transmission cost determination module 32 and the data compression module 33.
The address obtaining module 31 is configured to obtain a start location and a destination of the data packet; a transmission cost determining module 32, configured to determine a transmission cost of sending the data packet according to the origin and the destination; the data compression module 33 is configured to determine a compression mode of the data packet according to the transmission cost, compress the data packet according to the compression mode, and store the compression mode in the compressed data packet after the data packet is compressed.
According to the technical scheme, the problems that a compression mode with a low compression ratio is high in transmission cost in long-distance transmission caused by adopting a uniform compression mode for compression in traditional data communication, and the operation amount is large and the time consumption is long when a data packet is compressed and decompressed by adopting the compression mode with a high compression ratio are solved, the operation time of data packet compression and decompression is shortened, the transmission efficiency of the data packet is improved, and the double optimization of the transmission rate and the transmission cost is realized.
Optionally, the transmission cost determining module 32 includes:
and a path information acquisition unit configured to acquire transmission path information between the origin and the destination.
And the path information determining unit is used for determining each communication medium between the starting place and the destination according to the transmission path information, the transmission times on each communication medium and the single cost of the transmission on each communication medium.
And the transmission cost determining unit is used for determining the transmission cost of the data packet transmission according to each communication medium, each transmission time and each single cost.
Optionally, the data compression module 33 includes:
a compression mode determining unit, configured to determine a ratio of a transmission cost to a single cost of a communication medium between the same-board chips as a normalized cost; and searching a corresponding compression scheme according to the normalized cost, and determining the compression mode of the data packet according to the compression scheme.
The compression unit is used for compressing the data packet according to the compression mode and storing the compression mode in the compressed data packet after the data packet is compressed.
Further, when the number of data packets is plural, the compression unit is further configured to: acquiring a group of data packets with the same starting place and the same destination; compressing a group of data packets into the same compression packet according to a compression mode; the compressed mode is saved to the compressed package.
The data compression device provided by the embodiment of the disclosure can execute the data compression method provided by any embodiment of the disclosure, and has the corresponding functional modules and beneficial effects of the execution method.
Fig. 6 is a schematic structural view of an apparatus according to an exemplary embodiment of the present disclosure, and as shown in fig. 6, the apparatus includes a processor 41, a storage device 42, an input device 43, and an output device 44; the number of processors 41 in the device may be one or more, one processor 41 being taken as an example in fig. 6; the processor 41, the storage means 42, the input means 43 and the output means 44 in the device may be connected by a bus or by other means, in fig. 6 by way of example.
The storage device 42 is a computer readable storage medium, and may be used to store a software program, a computer executable program, and a module, such as program instructions/modules (e.g., the address acquisition module 31, the transmission cost determination module 32, and the data compression module 33) corresponding to the data compression method in the embodiment of the disclosure. The processor 41 executes various functional applications of the apparatus and data processing, namely, implements the data compression method described above by running software programs, instructions and modules stored in the storage device 42.
The storage device 42 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, at least one application program required for functions; the storage data area may store data created according to the use of the terminal, etc. In addition, the storage 42 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some examples, storage 42 may further include memory located remotely from processor 41, which may be connected to the device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input means 43 may be used to receive entered numeric or character information and to generate key signal inputs related to user settings of the device and function control, which may include a touch screen, keyboard, mouse, etc. The output device 44 may include a display device such as a display screen.
An exemplary embodiment of the present disclosure also provides a storage medium containing computer-executable instructions, which when executed by a computer processor, are for performing a data compression method comprising:
acquiring a starting place and a destination of data packet transmission;
determining the transmission cost of data packet transmission according to the starting place and the destination;
and determining a compression mode of the data packet according to the transmission cost, compressing the data packet according to the compression mode, and storing the compression mode in the compressed data packet after the data packet is compressed.
Of course, a storage medium containing computer-executable instructions provided by the embodiments of the present disclosure is not limited to the method operations described above, but may also perform related operations in the data compression method provided by any of the embodiments of the present disclosure.
From the above description of embodiments, it will be apparent to those skilled in the art that the present disclosure may be implemented by means of software and necessary general purpose hardware, but may of course also be implemented by means of hardware, although in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present disclosure may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a FLASH Memory (FLASH), a hard disk, or an optical disk of a computer, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the embodiments of the present disclosure.
It should be noted that, in the above-mentioned embodiments of the search apparatus, each unit and module included are only divided according to the functional logic, but not limited to the above-mentioned division, as long as the corresponding functions can be implemented; in addition, the specific names of the functional units are also only for distinguishing from each other, and are not used to limit the protection scope of the present disclosure.
Note that the above is only a preferred embodiment of the present disclosure and the technical principle applied. Those skilled in the art will appreciate that the present disclosure is not limited to the specific embodiments described herein, and that various obvious changes, rearrangements and substitutions can be made by those skilled in the art without departing from the scope of the disclosure. Therefore, while the present disclosure has been described in connection with the above embodiments, the present disclosure is not limited to the above embodiments, but may include many other equivalent embodiments without departing from the spirit of the present disclosure, the scope of which is determined by the scope of the appended claims.
Claims (7)
1. A method of data compression, comprising:
acquiring a starting place and a destination of data packet transmission;
determining the transmission cost of the data packet transmission according to the starting place and the destination; the transmission cost comprises data packet transmission delay and transmission energy consumption;
determining a compression mode of the data packet according to the transmission cost, compressing the data packet according to the compression mode, and storing the compression mode in a compressed packet after the data packet is compressed;
the determining the transmission cost of the data packet according to the starting place and the destination includes:
acquiring transmission path information between the starting place and the destination;
determining each communication medium between the starting place and the destination according to the transmission path information, the transmission times on each communication medium and the single cost of the transmission on each communication medium;
determining transmission cost of the data packet according to the transmission times and the single cost of each communication medium;
the determining the compression mode of the data packet according to the transmission cost comprises the following steps:
determining the ratio of the transmission cost to the reference single cost as a normalized cost;
and searching a corresponding compression scheme according to the normalized cost, and determining the compression mode of the data packet according to the compression scheme.
2. The method of claim 1, wherein said determining a transmission cost for the transmission of the data packet based on each of the transmission times and each of the single costs for each of the communication media comprises:
determining the product of the transmission times and the single cost in each communication medium in the transmission path information;
and taking the sum of products of the transmission times and the single cost in each communication medium as the transmission cost of the data packet.
3. The method according to any one of claims 1 to 2, wherein the communication medium comprises: at least one of inter-chip communication medium, inter-board communication medium, inter-cluster communication medium and remote communication medium.
4. The method of claim 1, wherein when the number of data packets is plural, the compressing the data packets according to the compression mode and storing the compression mode in the compressed data packets includes:
acquiring a group of data packets with the same starting place and the same destination;
compressing the group of data packets into the same compression packet according to the compression mode;
and storing the compression mode into the compression packet.
5. A data compression apparatus, comprising:
the acquisition module is used for acquiring the starting place and the destination of the data packet transmission;
a transmission cost determining module, configured to determine a transmission cost of the data packet according to the origin and the destination; the transmission cost comprises data packet transmission delay and transmission energy consumption;
the compression module is used for determining a compression mode of the data packet according to the transmission cost, compressing the data packet according to the compression mode, and storing the compression mode in a compressed packet after the data packet is compressed;
the transmission cost determining module includes:
a path information acquisition unit configured to acquire transmission path information between the origin and the destination;
a path information determining unit configured to determine each communication medium between the origin and the destination, the number of transmissions on each communication medium, and a single cost of transmission on each communication medium, based on the transmission path information;
a transmission cost determining unit, configured to determine, according to each communication medium, each transmission time and each single cost, a transmission cost for transmitting the data packet;
the compression module is further used for determining the ratio of the transmission cost to the reference single cost as a normalized cost; and searching a corresponding compression scheme according to the normalized cost, and determining the compression mode of the data packet according to the compression scheme.
6. An electronic device, the electronic device comprising:
one or more processors;
a storage means for storing one or more programs;
when executed by the one or more processors, causes the one or more processors to implement the data compression method of any of claims 1-4.
7. A storage medium containing computer executable instructions which when executed by a computer processor are for performing the data compression method of any one of claims 1-4.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010674720.1A CN111865326B (en) | 2020-07-14 | 2020-07-14 | Data compression method, device, equipment and storage medium |
| PCT/CN2021/106218 WO2022012581A1 (en) | 2020-07-14 | 2021-07-14 | Data compression and transmission methods and apparatuses, device, many-core system, and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010674720.1A CN111865326B (en) | 2020-07-14 | 2020-07-14 | Data compression method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111865326A CN111865326A (en) | 2020-10-30 |
| CN111865326B true CN111865326B (en) | 2023-06-23 |
Family
ID=72984227
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010674720.1A Active CN111865326B (en) | 2020-07-14 | 2020-07-14 | Data compression method, device, equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111865326B (en) |
| WO (1) | WO2022012581A1 (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111865326B (en) * | 2020-07-14 | 2023-06-23 | 北京灵汐科技有限公司 | Data compression method, device, equipment and storage medium |
| CN113067674B (en) * | 2021-03-17 | 2022-09-23 | 北京智慧星光信息技术有限公司 | Transmission method, system, electronic device and storage medium of network text data |
| CN117319318A (en) * | 2022-06-24 | 2023-12-29 | 中兴通讯股份有限公司 | Data stream compression method, network device and storage medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111210481A (en) * | 2020-01-10 | 2020-05-29 | 大连理工大学 | Depth estimation acceleration method of multiband stereo camera |
Family Cites Families (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5805932A (en) * | 1994-04-22 | 1998-09-08 | Sony Corporation | System for transmitting compressed data if compression ratio is at least preset ratio and pre-compressed data if compression ratio is less than preset ratio |
| US20060077412A1 (en) * | 2004-10-11 | 2006-04-13 | Vikram Phogat | Method and apparatus for optimizing data transmission costs |
| JP2008099261A (en) * | 2006-09-12 | 2008-04-24 | Yamaha Corp | Communication device and program |
| KR101503829B1 (en) * | 2007-09-07 | 2015-03-18 | 삼성전자주식회사 | Device and method for compressing data |
| US20100103183A1 (en) * | 2008-10-23 | 2010-04-29 | Hung-Ming Lin | Remote multiple image processing apparatus |
| JP2013085071A (en) * | 2011-10-07 | 2013-05-09 | Fujitsu Ltd | Data compression apparatus and method |
| CN104168603B (en) * | 2013-05-15 | 2017-11-14 | 上海交通大学 | The building method of the data acquisition tree with high energy efficiency based on compressed sensing technology |
| CN104866381B (en) * | 2014-02-20 | 2018-10-12 | 联想(北京)有限公司 | A kind of information processing method and the first electronic equipment |
| WO2015173889A1 (en) * | 2014-05-13 | 2015-11-19 | 株式会社日立製作所 | Storage device |
| CN103957582B (en) * | 2014-05-17 | 2017-05-24 | 浙江大学宁波理工学院 | Wireless sensor network self-adaptation compression method |
| US9521218B1 (en) * | 2016-01-21 | 2016-12-13 | International Business Machines Corporation | Adaptive compression and transmission for big data migration |
| CN109923786B (en) * | 2016-11-08 | 2023-10-24 | 皇家飞利浦有限公司 | Method for wireless data transmission range extension |
| CN106452920B (en) * | 2016-11-24 | 2019-07-19 | 广东电网有限责任公司电力调度控制中心 | Method and device for layout of grid data nodes based on cost optimization |
| CN110391873B (en) * | 2018-04-20 | 2022-02-11 | 伊姆西Ip控股有限责任公司 | Method, apparatus and computer program product for determining a data transfer mode |
| CN109783033B (en) * | 2019-01-31 | 2022-04-15 | 上海赜睿信息科技有限公司 | Data storage method and electronic equipment suitable for heterogeneous system |
| CN109921801A (en) * | 2019-03-01 | 2019-06-21 | 迅雷计算机（深圳）有限公司 | A kind of data compression and decompression method, device, system and data processing system |
| CN110769453B (en) * | 2019-10-31 | 2020-11-06 | 重庆大学 | Dynamic Compression Control Method for Multimodal Monitoring Data in Unstable Network Environment |
| CN110891092B (en) * | 2019-12-09 | 2022-09-23 | 北京布袋森林科技有限责任公司 | Data compression transmission method and device |
| CN111178540A (en) * | 2019-12-29 | 2020-05-19 | 浪潮(北京)电子信息产业有限公司 | Training data transmission method, device, equipment and medium |
| CN111262876B (en) * | 2020-01-22 | 2022-05-27 | 腾讯科技（深圳）有限公司 | Data processing method, device and equipment based on block chain and storage medium |
| CN111865326B (en) * | 2020-07-14 | 2023-06-23 | 北京灵汐科技有限公司 | Data compression method, device, equipment and storage medium |
-
2020
- 2020-07-14 CN CN202010674720.1A patent/CN111865326B/en active Active
-
2021
- 2021-07-14 WO PCT/CN2021/106218 patent/WO2022012581A1/en active Application Filing
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111210481A (en) * | 2020-01-10 | 2020-05-29 | 大连理工大学 | Depth estimation acceleration method of multiband stereo camera |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111865326A (en) | 2020-10-30 |
| WO2022012581A1 (en) | 2022-01-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111865326B (en) | Data compression method, device, equipment and storage medium | |
| US9904577B2 (en) | Hybrid heterogeneous host system, resource configuration method and task scheduling method | |
| JP5466210B2 (en) | Table search device, table search method, and table search system | |
| CN108595483B (en) | Data processing method and related device | |
| CN114564509B (en) | Serialization method, deserialization method, device, chip, and medium | |
| CN106161633A (en) | A kind of based on the transmission method of packaging file under cloud computing environment and system | |
| CN109936472A (en) | Node computing capability description method, interaction method and device for fog computing | |
| CN113961510B (en) | File processing method, device, equipment and storage medium | |
| CN115952385B (en) | Parallel supernode sorting method and system for solving large-scale sparse equations | |
| CN114916033B (en) | Data transmission method, system, electronic equipment and storage medium | |
| CN116226139A (en) | A method and system for distributed storage and processing of large-scale ocean data | |
| CN102427420B (en) | Virtual network mapping method and device based on graph pattern matching | |
| CN101625673A (en) | Method for mapping task of network on two-dimensional grid chip | |
| CN113179154B (en) | Resource joint allocation method and related equipment in quantum key distribution Internet of things | |
| CN104636432A (en) | Method and device for journal file compression and decompression | |
| CN108235158B (en) | Method and device for processing optical network multicast service | |
| WO2020024392A1 (en) | Node processing method and apparatus, storage medium and electronic device | |
| WO2023061180A1 (en) | Multi frequency-based data sending method and apparatus, multi frequency-based data receiving method and apparatus, and device | |
| CN116961696A (en) | Dual-mode module communication method and device, electronic equipment and storage medium | |
| US7930489B2 (en) | Techniques for optimizing configuration partitioning | |
| CN116383240A (en) | FPGA-based multi-database accelerated query method, device, equipment and medium | |
| CN111147374B (en) | Distributed communication algorithm and device | |
| JP5594460B2 (en) | Transmission information control apparatus, method and program | |
| CN118467182B (en) | Memory access method, computer program product, electronic device, and medium | |
| CN114465718B (en) | Multi-protocol translation method and related equipment for quantum key distribution business |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |