KR20250104845A - Encoding apparatus for parallel encoding and image encoding method, and decoding apparatus and image decoding method - Google Patents

Abstract

A method for encoding an image by an encoding device may include: a step of identifying a plurality of segments from an image; a step of encoding the plurality of segments in parallel; a step of storing substreams generated through the parallel encoding in a plurality of buffers; a step of obtaining substream packets having a predetermined size from the plurality of buffers when a current state corresponds to a preset output condition while the substreams are stored in the plurality of buffers; and a step of arranging the substream packets in a predetermined order and outputting a bitstream in which the substream packets are arranged.

Description

Encoding apparatus for parallel encoding and image encoding method thereby, and decoding apparatus and image decoding method thereby {ENCODING APPARATUS FOR PARALLEL ENCODING AND IMAGE ENCODING METHOD, AND DECODING APPARATUS AND IMAGE DECODING METHOD}

The present disclosure relates to the field of image encoding and decoding, and more particularly, the present disclosure relates to a device and method for encoding and decoding an image using parallel processing techniques.

In encoding and decoding of images, the image is divided into blocks, and each block can be predicted and encoded or predicted and decoded through inter prediction or intra prediction.

Inter prediction is a technique for compressing images by removing temporal redundancy between images. In inter prediction, blocks of the current image can be predicted using a reference image. The reference block most similar to the current block can be searched for in the reference image. The current block can be predicted based on the reference block, and the prediction block generated as a result of the prediction can be subtracted from the current block to generate a residual block.

Intra prediction is a technique for compressing images by removing spatial redundancy within the image. In intra prediction, a prediction block can be generated based on the surrounding pixels of the current block according to the intra prediction mode. Then, a residual block can be generated by subtracting the prediction block from the current block.

Information about the residual block can be included in the bitstream. The decoder can obtain the residual block from the bitstream and reconstruct the current block using the prediction block generated through inter prediction or intra prediction and the residual block obtained from the bitstream.

As multi-core CPUs are widely used in mobile devices, parallelization technology has been adopted in standards such as HEVC. Parallelization technology is a method of dividing data to be processed by a program into several units and then assigning the divided data to different cores or threads to perform the same task in parallel. Through parallel encoding and parallel decoding of images, the time required for encoding and decoding processes can be reduced.

A method of encoding an image according to one embodiment may include a step of identifying a plurality of segments from the image.

A method of encoding an image according to one embodiment may include a step of encoding a plurality of segments in parallel.

A method of encoding an image according to one embodiment may include a step of storing substreams generated through parallel encoding in a plurality of buffers.

A method of encoding a video according to one embodiment may include a step of obtaining substream packets having a predetermined size from a plurality of buffers when a current state corresponds to a preset output condition while the substreams are stored in a plurality of buffers.

A method of encoding an image according to one embodiment may include the steps of arranging substream packets in a predetermined order and outputting a bitstream in which the substream packets are arranged.

A method of decoding an image according to one embodiment may include a step of identifying a plurality of segments within the image.

A method of decoding an image according to one embodiment may include a step of obtaining substream packets of a predetermined size arranged in a predetermined order from a bitstream. In one embodiment, the substream packets may be generated through parallel encoding of a plurality of segments.

A method of decoding an image according to one embodiment may include a step of decoding substream packets to restore a plurality of segments.

A method of decrypting an image according to one embodiment may include a step of generating a restored image including a plurality of restored segments.

In one embodiment, the substream packets may correspond to each of the plurality of segments in a predetermined order.

An encoding device according to one embodiment may include a control unit that identifies a plurality of segments from an image.

An encoding device according to one embodiment may include at least one core for encoding a plurality of segments in parallel.

An encoding device according to one embodiment may include a plurality of buffers.

In one embodiment, the control unit can store substreams generated through parallel encoding in multiple buffers.

In one embodiment, the control unit can obtain substream packets having a predetermined size from the plurality of buffers if a current state corresponds to a preset output condition while the substreams are stored in the plurality of buffers.

In one embodiment, the control unit can arrange the substream packets in a predetermined order and output a bitstream in which the substream packets are arranged.

A decoding device according to one embodiment may include a control unit that identifies a plurality of segments in an image and obtains substream packets of a predetermined size arranged in a predetermined order from a bitstream.

In one embodiment, substream packets may be generated via parallel encoding of multiple segments.

A decryption device according to one embodiment may include at least one core for decrypting substream packets to restore a plurality of segments.

In one embodiment, the substream packets may correspond to each of the plurality of segments in a predetermined order.

In one embodiment, the control unit can generate a restored image comprising a plurality of restored segments.

FIG. 1 is a diagram for explaining an image encoding and decoding process according to one embodiment.
FIG. 2 is a block diagram illustrating the configuration of an encoding device according to one embodiment.
FIG. 3a is a diagram illustrating images to which parallel encoding can be applied according to one embodiment.
FIG. 3b is a diagram illustrating slices to which parallel encoding can be applied according to one embodiment.
FIG. 3c is a diagram illustrating tiles to which parallel encoding can be applied according to one embodiment.
FIG. 3d is a diagram illustrating a maximum encoding unit row to which parallel encoding can be applied according to one embodiment.
FIG. 4 is a table for explaining a method of allocating multiple segments according to one embodiment.
FIG. 5 is a diagram for explaining a process in which multiple segments are encoded in parallel according to one embodiment.
FIG. 6 is a diagram for explaining a process in which multiple segments are encoded in parallel according to one embodiment.
FIG. 7a is a diagram for explaining a process of outputting a bitstream from multiple buffers according to one embodiment.
FIG. 7b is a diagram for explaining a process of outputting a bitstream from multiple buffers according to one embodiment.
FIG. 8 is a diagram for explaining a process of outputting a bitstream from multiple buffers according to one embodiment.
FIG. 9 is a diagram for explaining a process of outputting a bitstream from multiple buffers according to one embodiment.
Fig. 10 is a flowchart for explaining a method of encoding an image according to one embodiment.
Fig. 11 is a block diagram illustrating the configuration of a decryption device according to one embodiment.
FIG. 12 is a diagram illustrating a method for restoring multiple segments from a bitstream according to one embodiment.
FIG. 13 is a diagram illustrating a method for restoring multiple segments from a bitstream according to one embodiment.
FIG. 14 is a diagram illustrating a method for restoring multiple segments from a bitstream according to one embodiment.
Fig. 15 is a flowchart for explaining a method of decoding an image according to one embodiment.
FIG. 16 is a block diagram illustrating a configuration of a terminal device including an encoding device or a decoding device according to one embodiment.

Since the present disclosure can have various modifications and various embodiments, embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the embodiments of the present disclosure, and the present disclosure can include all modifications, equivalents, or substitutes included in the spirit and technical scope of the various embodiments.

In describing the embodiments, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the present disclosure, the detailed description may be omitted. In addition, numbers (e.g., first, second, etc.) used in the description of the embodiments may correspond to identification symbols for distinguishing one component from another.

In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

In this disclosure, when a component is referred to as being "connected" or "connected" to another component, the component may be directly connected or connected to the other component, but unless otherwise specifically stated, the component may also be connected or connected via another component in between.

In the present disclosure, components expressed as 'unit', 'module', etc. may be two or more components combined into one component, or one component may be divided into two or more more detailed components. In addition, each of the components described below may additionally perform some or all of the functions performed by other components in addition to its own main function, and some of the main functions performed by each component may be performed exclusively by other components.

In the present disclosure, 'image' may represent a picture, a still image, a frame, a moving image composed of a plurality of consecutive still images, or a video.

In the present disclosure, a 'sample' may mean data that is assigned to a sampling location of an image and is a target of processing. For example, a pixel in an image in a spatial domain may correspond to a sample.

In the present disclosure, a 'block' may include a slice, tile, maximum coding unit, encoding unit, transform unit or prediction unit segmented from an image. A block may be composed of one or more samples.

FIG. 1 is a diagram for explaining an image encoding and decoding process according to one embodiment.

The encoding unit (110) of the image encoding and decoding system (100) transmits a bitstream generated through encoding of an image, and the decoding unit (150) can output a restored image by decoding the bitstream.

In one embodiment, the encoding unit (110) may include an encoding device (200) described below, and the decoding unit (150) may include a decoding device (1100) described below.

In the encoding unit (110), the prediction encoding unit (115) can output prediction data through inter prediction or intra prediction, and the transformation and quantization unit (120) can output quantized transformation coefficients of residual data between the prediction data and the current input image.

The entropy encoding unit (125) can encode quantized transform coefficients and output them as a bitstream.

The quantized transform coefficients are restored as residual data in the spatial domain through the inverse quantization and inverse transformation unit (130), and image data in the spatial domain can be restored from the residual data and the prediction data. The restored image data in the spatial domain can be output as a restored image through the deblocking filtering unit (135) and the loop filtering unit (140). The restored image can be used as a reference image for the next input image through the prediction encoding unit (115).

Encoded image data from the bitstream received by the decoding unit (150) can be restored to residual data in the spatial domain through the entropy decoding unit (155) and the inverse quantization and inverse transformation unit (160).

The prediction data and residual data output from the prediction decoding unit (175) are combined to form image data in the spatial domain, and the deblocking filtering unit (165) and the loop filtering unit (170) can perform filtering on the image data in the spatial domain to output a restored image corresponding to the original image. The restored image can be used as a reference image for the next image by the prediction decoding unit (175).

The loop filtering unit (140) of the encoding unit (110) can perform loop filtering using filter information input according to user input or system settings.

The filter information used by the loop filtering unit (140) can be transmitted to the decoding unit (150) together with the encoded image data through the entropy encoding unit (125). The loop filtering unit (170) of the decoding unit (150) can perform loop filtering based on the filter information transmitted to the decoding unit (150).

The time required for encoding and decoding an image can be reduced through parallel encoding and parallel decoding of the image. Hereinafter, an encoding device that encodes an image using parallel processing technology is described.

FIG. 2 is a block diagram illustrating the configuration of an encoding device according to one embodiment.

The encoding device (200) may include a control unit (210), at least one core (230), and a plurality of buffers (250).

Although FIG. 2 illustrates that the encoding device (200) includes one core (230), the number of cores (230) that may be included in the encoding device (200) may be multiple.

In addition, FIG. 2 illustrates that three buffers (250a, 250b, 250c) are included in the encoding device (200), and the number of buffers included in the encoding device (200) may be the same as the number of segments to be processed in parallel, which will be described later.

In one embodiment, the control unit (210) and at least one core (230) may be included in a processor.

In one embodiment, the control unit (210) may be included in the first processor, and at least one core (230) may be included in the second processor.

In one embodiment, when the encoding device (200) includes a plurality of cores (230), some of the plurality of cores (230) may be included in one processor, and other some of the cores (230) may be included in another processor.

In one embodiment, the processor for implementing the control unit (210) and the at least one core (230) may be a dedicated processor for encoding images. In one embodiment, the processor for implementing the control unit (210) and the at least one core (230) may be a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphic processing unit (GPU).

In one embodiment, at least one instruction for encoding an image may be stored in a memory, and the control unit (210) and at least one core (230) may operate according to the at least one instruction stored in the memory.

In one embodiment, the plurality of buffers (250) may be physically distinct from one another. In one embodiment, each of the plurality of buffers (250) may mean a storage area identified by hardware or software within one or more buffers.

In one embodiment, the plurality of buffers (250) may be FIFO (First In First Out) buffers. Accordingly, the data input first into the plurality of buffers (250) may be output first.

In one embodiment, the control unit (210) can control the operation of at least one core (230) and a plurality of buffers (250) for parallel encoding of an image.

In one embodiment, the control unit (210) can identify multiple segments from an image for parallel encoding of the image.

Parallel encoding of multiple segments may mean that multiple segments are encoded independently. In other words, since there is little dependency between the segments, encoding of one segment may proceed simultaneously with encoding of another segment.

In one embodiment, there may be some time delay between the encoding of one segment and the encoding of another segment, such as in wavefront parallel processing (WPP) described below.

A segment may mean a unit of an image that is the target of parallel encoding. For example, a segment may be a block divided from an image. In addition, for example, a segment may be a frame or image that constitutes an image sequence.

Each of the plurality of segments is assigned to at least one core (230) and can be encoded in parallel by at least one core (230).

Segments that are targets of parallel encoding are described with reference to FIGS. 3a to 3d.

FIG. 3a is a diagram illustrating images to which parallel encoding can be applied according to one embodiment.

In a video codec, a group of pictures (GOP) can be encoded and decoded according to a hierarchical structure. In this case, the images that make up the GOP are divided into layers, and the images of lower layers can be encoded and decoded by referring to the images of higher layers.

If the images in a specific layer do not reference each other, the images in the specific layer can be the target of parallel processing.

Referring to FIG. 3a, images belonging to layer i+1 can be encoded and decoded with reference to images belonging to layer i. Since images belonging to layer i+1 do not refer to each other, the control unit (210) can identify images belonging to the same layer among images included in the image sequence, for example, image 0, image 1, image 2, image 3, image 4, and image 5 illustrated in FIG. 3a, as multiple segments.

At least one core (230) can independently encode each of image 0, image 1, image 2, image 3, image 4, and image 5. In one embodiment, at least one core (230) can divide each of image 0, image 1, image 2, image 3, image 4, and image 5 into a plurality of maximum coding units, and encode the plurality of maximum coding units according to a predetermined scan order (e.g., a raster scan order).

FIG. 3b is a diagram illustrating slices to which parallel encoding can be applied according to one embodiment.

The control unit (210) can divide the image (300) into a plurality of slices. Each of the plurality of slices can include one or more maximum coding units.

In one embodiment, since the slices do not have dependencies on each other, the control unit (210) can identify slice 0, slice 1, slice 2, and slice 3 as shown in FIG. 3b as multiple segments.

At least one core (230) can independently encode each of slice 0, slice 1, slice 2, and slice 3. In one embodiment, at least one core (230) can encode the maximum encoding units included in slice 0, slice 1, slice 2, and slice 3 according to a predetermined scan order.

FIG. 3c is a diagram illustrating tiles to which parallel encoding can be applied according to one embodiment.

The control unit (210) can divide the image (300) into a plurality of tiles. Each of the plurality of tiles can include one or more maximum coding units.

In one embodiment, since the tiles do not have dependencies on each other, the control unit (210) can identify tile 0, tile 1, tile 2, and tile 3 illustrated in FIG. 3c as multiple segments.

At least one core (230) can independently encode each of tile 0, tile 1, tile 2, and tile 3. In one embodiment, at least one core (230) can encode the maximum encoding units included in tile 0, tile 1, tile 2, and tile 3 according to a predetermined scan order.

FIG. 3d is a diagram illustrating a maximum encoding unit row to which parallel encoding can be applied according to one embodiment.

WPP (wavefront parallel processing) technology has been adopted in codec standards such as HEVC. In WPP, maximum coding unit rows divided from an image (300) can be processed in parallel.

The image (300) can be divided into a plurality of maximum coding units, and the maximum coding unit rows in the image (300) can be identified as a plurality of segments. Accordingly, the maximum coding unit row 0, the maximum coding unit row 1, the maximum coding unit row 2, the maximum coding unit row 3, and the maximum coding unit row 4 illustrated in FIG. 3d can be encoded in parallel by at least one core (230).

In WPP, since a maximum coding unit included in one maximum coding unit row can refer to two maximum coding units included in the upper row, there may be some delay in the process of parallel encoding of maximum coding unit rows.

For example, since the maximum coding unit (321) located at the leftmost position in the maximum coding unit row 1 can refer to data generated during the encoding process of the maximum coding unit (311) located at the leftmost position in the maximum coding unit row 0 and the maximum coding unit (312) located to the right of the maximum coding unit (311) located at the leftmost position, after the encoding of the two maximum coding units (311, 312) in the maximum coding unit row 0 is completed, the encoding of the maximum coding unit row 1 can be started. Similarly, after the encoding of the two maximum coding units in the maximum coding unit row 1 is completed, the encoding of the maximum coding unit row 2 can be started.

Referring again to FIG. 2, in one embodiment, when a plurality of segments are identified from an image, the control unit (210) may assign the plurality of segments to at least one core (230) for parallel encoding of the plurality of segments.

In one embodiment, the control unit (210) may allocate a plurality of segments to at least one core (230) by taking into account performance information of parallel encoding.

The performance information for parallel encoding may include at least one of the number of cores (230) capable of parallel processing of segments, or the number of threads capable of parallel processing by at least one core (230). In one embodiment, the performance information for parallel encoding may include the number of segments capable of parallel encoding.

For example, if the number of segments that can be parallel encoded is n (n is a natural number greater than 1) and the number of segments identified from an image is m (m is a natural number greater than n), then among the m segments, n segments can be first allocated to at least one core (230) and parallel encoded, and m-n segments can be parallel encoded by at least one core (230) after encoding of the n segments is completed.

A method of allocating multiple segments to at least one core (230) by considering performance information of parallel encoding is described with reference to FIG. 4.

FIG. 4 is a table for explaining a method of allocating multiple segments according to one embodiment.

Five segments are identified from the image, and it is assumed that the number of cores (230) (or threads) capable of parallel processing is three.

Since three segments can be parallel encoded at one time, the control unit (210) can first allocate the first segment, the second segment, and the third segment to the first core (or the first thread), the second core (or the second thread), and the third core (or the third thread), respectively. Then, the control unit (210) can additionally allocate the remaining fourth segment and fifth segment to the first core (or the first thread) and the second core (or the second thread).

The first core, the second core, and the third core can encode the first segment, the second segment, and the third segment in parallel, respectively, and when the encoding of the first segment and the second segment is completed, the first core and the second core can encode the fourth segment and the fifth segment in parallel, respectively.

In one embodiment, when a hyper threading technology capable of generating and processing multiple threads for at least one core (230) is employed, the first segment, the second segment, and the third segment may be allocated to the first thread, the second thread, and the third thread. At least one core (230) may parallel encode the first segment, the second segment, and the third segment corresponding to the first thread, the second thread, and the third thread. When the parallel encoding of the first segment and the second segment is completed, at least one core (230) may parallel encode the fourth segment and the fifth segment additionally allocated to the first thread and the second thread. In one embodiment, each of the fourth segment and the fifth segment may be allocated to the fourth thread and the fifth thread and parallel encoded by at least one core (230).

FIG. 5 is a diagram for explaining a process in which multiple segments are encoded in parallel according to one embodiment.

The control unit (210) can assign each of the first to third segments to the first core (230a) to the third core (230c) when five segments are identified from the image and the number of cores (230a, 230b, 230c) that can be used for parallel processing is three. Since the number of segments that can be parallel encoded at one time is three, the control unit (210) can additionally assign the fourth segment and the fifth segment to the first core (230a) and the second core (230b).

Each of the first core (230a), the second core (230b), and the third core (230c) may encode the first segment, the second segment, and the third segment, and substreams (240a, 240b, 240c) may be generated as encoding results. The substreams (240a, 240b, 240c) may include results of entropy encoding data generated through encoding the first segment, the second segment, and the third segment. The substreams (240a, 240b, 240c) may be composed of bins having values of 0 or 1.

As the first segment, the second segment, and the third segment are encoded in parallel, a first substream (240a) corresponding to the first segment, a second substream (240b) corresponding to the second segment, and a third substream (240c) corresponding to the third segment can be output from each of the first core (230a), the second core (230b), and the third core (230c).

Each of the first substream (240a), the second substream (240b), and the third substream (240c) can be stored in multiple buffers (250a, 250b, 250c).

In one embodiment, the number of buffers (250a, 250b, 250c) included in the encoding device (200) may be equal to a number determined from performance information of parallel encoding, for example, the number of cores (230a, 230b, 230c), the number of threads, or the number of segments that can be encoded in parallel.

As illustrated in FIG. 5, when three segments can be parallel encoded at a time, the encoding device (200) may include three buffers, i.e., a first buffer (250a), a second buffer (250b), and a third buffer (250c). The first substream (240a), the second substream (240b), and the third substream (240c) may be stored in the first buffer (250a), the second buffer (250b), and the third buffer (250c), respectively.

Since the fourth segment and the fifth segment are assigned to the first core (230a) and the second core (230b), when the encoding of the first segment and the second segment is completed, each of the first core (230a) and the second core (230b) can additionally encode the fourth segment and the fifth segment. Then, the fourth substream corresponding to the fourth segment can be stored in the first buffer (250a), and the fifth substream corresponding to the fifth segment can be stored in the second buffer (250b).

FIG. 6 is a diagram for explaining a process in which multiple segments are encoded in parallel according to one embodiment.

The control unit (210) can assign each of the first to third segments to the first thread (235a) to the third thread (235c) when five segments are identified from the image and the number of threads (235a, 235b, 235c) that can be processed in parallel at one time is three. In addition, the control unit (210) can additionally assign the fourth segment and the fifth segment to the first thread (235a) and the second thread (235b).

In one embodiment, since the first segment is assigned to the first thread (235a) and the second segment is assigned to the second thread (235b), the fourth segment may be assigned to the fourth thread and the fifth segment may be assigned to the fifth thread.

The core (230) may encode the first segment, the second segment, and the third segment corresponding to the first thread (235a), the second thread (235b), and the third thread (235c), and substreams (240a, 240b, 240c) may be generated as encoding results. The substreams (240a, 240b, 240c) may include results of entropy encoding data generated through encoding the first segment, the second segment, and the third segment. The substreams (240a, 240b, 240c) may be composed of bins having values of 0 or 1.

As the first segment, the second segment, and the third segment are encoded in parallel, a first substream (240a) corresponding to the first segment, a second substream (240b) corresponding to the second segment, and a third substream (240c) corresponding to the third segment can be output from the core (230).

The first substream (240a), the second substream (240b), and the third substream (240c) can be stored in the first buffer (250a), the second buffer (250b), and the third buffer (250c), respectively.

Since the 4th segment and the 5th segment are assigned to the 1st thread (235a) (or the 4th thread) and the 2nd thread (235b) (or the 5th thread), when the core (230) completes encoding of the 1st segment and the 2nd segment, the 4th segment and the 5th segment assigned to the 1st thread (235a) (or the 4th thread) and the 2nd thread (235b) (or the 5th thread) can be encoded in parallel. Then, the 4th substream corresponding to the 4th segment can be stored in the 1st buffer (250a), and the 5th substream corresponding to the 5th segment can be stored in the 2nd buffer (250b).

Below, the process of outputting a bitstream from an encoding device (200) is described.

In one embodiment, when encoding of the first segment, the second segment, the third segment, the fourth segment, and the fifth segment illustrated in FIGS. 5 and 6 is completed, a first substream (240a), a second substream (240b), a third substream (240c), a fourth substream, and a fifth substream corresponding to the first segment, the second segment, the third segment, the fourth segment, and the fifth segment, respectively, can be acquired for output from the buffers (250a, 250b, 250c). Then, the first substream (240a), the second substream (240b), the third substream (240c), the fourth substream, and the fifth substream can be arranged in an encoding order and output as a bitstream.

However, according to the output process of the bitstream as above, a delay may occur in the encoding and decoding process of the image. Specifically, the time required until the encoding of the first to fifth segments is completed and the time required to arrange the first to fifth substreams (240a) are required in the encoding process, so it may not be suitable for a server-client based real-time image transmission system.

According to one embodiment, if substreams stored in multiple buffers (250a, 250b, 250c) are packetized and transmitted even before encoding of segments that are targets of parallel encoding is completed, time delay can be minimized. This will be described with reference to FIGS. 7a, 7b, 8, and 9.

FIGS. 7A and 7B are diagrams for explaining a process of outputting a bitstream from a plurality of buffers according to one embodiment.

The first buffer (250a) can store a first substream corresponding to an encoding result for the first segment, and the second buffer (250b) can store a second substream corresponding to an encoding result for the second segment. Additionally, the third buffer (250c) can store a third substream corresponding to an encoding result for the third segment.

While the first substream, the second substream, and the third substream are stored in the first buffer (250a), the second buffer (250b), and the third buffer (250c) as parallel encoding progresses, the control unit (210) can output the bitstream (700) through the first buffer (250a), the second buffer (250b), and the third buffer (250c) even before parallel encoding is completed, if the current state corresponds to a preset output condition.

In one embodiment, the control unit (210) can receive output conditions from a user or obtain pre-stored output conditions.

In one embodiment, the output condition may be set based on at least one of: a total size of the substreams stored in the plurality of buffers (250a, 250b, 250c), a ratio between a capacity of the plurality of buffers (250a, 250b, 250c) and the total size of the substreams stored in the plurality of buffers (250a, 250b, 250c), an elapsed time of parallel encoding, or a progress of encoding for each of the plurality of segments.

For example, when the total size of substreams stored in multiple buffers (250a, 250b, 250c) is 1 megabyte (MB), an output condition can be set such that a bitstream (700) is output from multiple buffers (250a, 250b, 250c).

In addition, for example, when the ratio of the total size of the substreams stored in the plurality of buffers (250a, 250b, 250c) to the capacity of the plurality of buffers (250a, 250b, 250c) becomes 50%, the output condition can be set so that the bitstream (700) is output from the plurality of buffers (250a, 250b, 250c).

Additionally, for example, an output condition may be set such that when 10 ms has elapsed since parallel encoding for multiple segments has started, a bitstream (700) is output from multiple buffers (250a, 250b, 250c).

In addition, for example, when encoding of a predetermined number (e.g., 10) of maximum coding units included in each of a plurality of segments is completed, an output condition may be set such that a bitstream (700) is output from a plurality of buffers (250a, 250b, 250c).

In one embodiment, two or more of the output conditions exemplified above may be combined. For example, the output condition may be set so that the bitstream (700) is output at an earlier time among the time when the total size of the substreams stored in the plurality of buffers (250a, 250b, 250c) becomes 1 megabyte (MB) and the time when the ratio of the total size of the substreams stored in the plurality of buffers (250a, 250b, 250c) to the capacity of the plurality of buffers becomes 50%.

In one embodiment, the output conditions may be set according to the degree of delay required by the user or the service to which the encoding process is applied. For example, when low delay is required, the output conditions may be set so that the bitstream (700) is output from the plurality of buffers (250a, 250b, 250c) in a short period. Also, when low delay is not required, the output conditions may be set so that the bitstream (700) is output from the plurality of buffers (250a, 250b, 250c) in a relatively long period.

For example, when low delay is required, the output condition may be set so that the bitstream (700) is output when the total size of the substreams stored in the plurality of buffers (250a, 250b, 250c) is 1 megabyte (MB), and when low delay is not required, the output condition may be set so that the bitstream (700) is output when the total size of the substreams stored in the plurality of buffers (250a, 250b, 250c) is 2 megabytes (MB).

In addition, for example, when low delay is required, the output condition may be set so that the bitstream (700) is output when the ratio of the total size of the substreams stored in the plurality of buffers (250a, 250b, 250c) to the capacity of the plurality of buffers (250a, 250b, 250c) is 50%, and when low delay is not required, the output condition may be set so that the bitstream (700) is output when the ratio of the total size of the substreams stored in the plurality of buffers (250a, 250b, 250c) to the capacity of the plurality of buffers (250a, 250b, 250c) is 60%.

In one embodiment, a user can set a delay level for an encoding process, and the control unit (210) can set an output condition according to the delay level set by the user.

If the current state corresponds to a preset output condition, the control unit (210) can obtain substream packets (715, 725, 735) having a predetermined size from each of the plurality of buffers (250a, 250b, 250c). Here, the substream packet can be understood as a basic transmission unit of a substream included in the bitstream (700).

For example, if the predetermined size is 8 kilobytes (KB), 8 kilobyte substreams (710, 720, 730) can be extracted from each of the multiple buffers (250a, 250b, 250c) as substream packets (715, 725, 735).

Referring to FIG. 7a, if the current state corresponds to a preset output condition, the control unit (210) can obtain substreams (710, 720, 730) including 14 bins corresponding to a predetermined size among the substreams stored in the first buffer (250a), the second buffer (250b), and the third buffer (250c) as substream packets (715, 725, 735). Accordingly, the control unit (210) can obtain a first substream packet (715) including 14 bins from the first buffer (250a), and a second substream packet (725) including 14 bins from the second buffer (250b). In addition, the control unit (210) can obtain a third substream packet (735) including 14 bins from the third buffer (250c).

When multiple substream packets (715, 725, 735) are acquired from multiple buffers (250), the control unit (210) can arrange the multiple substream packets (715, 725, 735) in a predetermined order and output a bitstream (700) in which the multiple substream packets (715, 725, 735) are arranged.

In one embodiment, the predetermined order may correspond to an encoding order of segments corresponding to each of the plurality of sub-stream packets (715, 725, 735). For example, if the encoding order of the first segment corresponding to the first sub-stream packet (715) is earlier than the encoding order of the second segment corresponding to the second sub-stream packet (725), the first sub-stream packet (715) may be arranged before the second sub-stream packet (725).

In one embodiment, the control unit (210) may arrange the substream packets (715, 725, 735) in a pre-arranged order between the encoding device (200) and the decoding device (1100). For example, it may be arranged between the encoding device (200) and the decoding device (1100) to arrange the substream packets (715, 725, 735) in the reverse order of the encoding order of the plurality of segments.

In one embodiment, the control unit (210) may arrange the substream packets (715, 725, 735) in a predetermined order and add information necessary to restore the substream packets (715, 725, 735) to generate a bitstream (700).

In one embodiment, information necessary to restore the substream packets (715, 725, 735) may be included in the header of the bitstream (700).

In one embodiment, a bitstream (700) including substream packets (715, 725, 735) may be transmitted to a decryption device (1100) over a network.

In one embodiment, a bitstream (700) including substream packets (715, 725, 735) may be stored on a data storage medium including a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and the like.

In one embodiment, the control unit (210) may repeatedly arrange sub-stream packets corresponding to the number of segments to be parallel-encoded in a predetermined order. For example, if the number of segments that can be parallel-encoded at one time is 3, sub-stream packets (715, 725, 735) corresponding to each of the three segments may be arranged in a predetermined order, and subsequent sub-stream packets corresponding to each of the three segments may be arranged in a predetermined order behind the previously arranged sub-stream packets (715, 725, 735).

Referring to FIG. 7b, when the first sub-stream packet (715), the second sub-stream packet (725), and the third sub-stream packet (735) illustrated in FIG. 7a are acquired from the first buffer (250a), the second buffer (250b), and the third buffer (250c) and arranged in the bitstream (700), the sub-streams corresponding to the first sub-stream packet (715), the second sub-stream packet (725), and the third sub-stream packet (735) in the first buffer (250a), the second buffer (250b), and the third buffer (250c) may be deleted.

If the current state corresponds to a preset output condition, the control unit (210) may acquire a first substream (750) of a predetermined size, a second substream (760) of a predetermined size, and a third substream (770) of a predetermined size, each of which is stored in a plurality of buffers (250a, 250b, 250c), as a first substream packet (755), a second substream packet (765), and a third substream packet (775), and may arrange the acquired substream packets (755, 765, 775) in a predetermined order to generate a bitstream (700).

According to one embodiment, the control unit (210) generates a bitstream by grouping substreams into packets if the current state corresponds to a preset output condition even when encoding of multiple segments is in progress, so that time delay due to parallel encoding and parallel decoding can be minimized.

FIG. 8 is a diagram for explaining a process of outputting a bitstream from multiple buffers according to one embodiment.

While encoding is in progress for multiple segments and multiple substreams are stored in multiple buffers (250a, 250b, 250c), there may be a segment whose encoding progress is slower than that of other segments.

As described above, while the first substream, the second substream, and the third substream are stored in the first buffer (250a), the second buffer (250b), and the third buffer (250c), if the current state corresponds to a preset output condition, the substreams of a predetermined size stored in the first buffer (250a), the second buffer (250b), and the third buffer (250c) can be obtained as substream packets.

If there is a segment whose encoding progress is slow, the size of the substream generated through encoding for the segment may be smaller than a predetermined size, and thus it may be difficult to obtain the substream packet from the buffer storing the substream. For example, as illustrated in FIG. 8, the size of the second substream (820) stored in the second buffer (250b) may be smaller than a predetermined size.

In this case, in one embodiment, instead of obtaining the second sub-stream packet from the second buffer (250b), the control unit (210) may generate a second sub-stream packet (825) having a predetermined size and a predetermined value (e.g., 0), and arrange the generated second sub-stream packet (825) in the bitstream (800). At this time, the second sub-stream packet (825) may be understood as a garbage packet.

Accordingly, as illustrated in FIG. 8, the first sub-stream packet (815) corresponding to the first sub-stream (810) stored in the first buffer (250a), the second sub-stream packet (or garbage packet) (825) generated for the second buffer (250b), and the third sub-stream packet (835) corresponding to the third sub-stream (830) stored in the third buffer (250c) can be arranged in a predetermined order.

Since the second substream (820) stored in the second buffer (250b) is not used to generate the second substream packet (825), it is not deleted from the second buffer (250b) and can be included in the second substream packet at the next output time.

In one embodiment, the control unit (210) may prevent the decoding device (1100) from performing decoding of the second sub-stream packet (825) by including information in the bitstream (800) indicating that decoding of the second sub-stream packet (825) included in the bitstream (800) is not required. For example, the control unit (210) may include information indicating that decoding of the second sub-stream packet (825) is not required in the header of the bitstream (800).

In one embodiment, a constant bitrate can be maintained by generating a substream packet and including the generated substream packet in the bitstream (800) even if the substream packet cannot be obtained from the buffer.

FIG. 9 is a diagram for explaining a process of outputting a bitstream from multiple buffers according to one embodiment.

As described above with reference to FIGS. 5 and 6, each of the fourth segment and the fifth segment can be additionally allocated to the first core (230a) (or the first thread (235a)) and the second core (230b) (or the second thread (235b)), so that encoding of the fourth segment and the fifth segment can continue even after encoding of the first segment, the second segment, and the third segment is completed.

Referring to FIG. 9, when the encoding for the third segment is completed and all of the third substreams stored in the third buffer (250c) are output, no more data may be stored in the third buffer (250c). However, since encoding for the fourth and fifth segments is in progress, the fourth substream generated through encoding for the fourth segment may be stored in the first buffer (250a), and the fifth substream generated through encoding for the fifth segment may be stored in the second buffer (250b).

If the current state corresponds to a preset output condition, the control unit (210) can obtain a fourth substream (910) of a predetermined size stored in the first buffer (250a) as a fourth substream packet (915), and can obtain a fifth substream (920) of a predetermined size stored in the second buffer (250b) as a fifth substream packet (925). In addition, since no substream is stored in the third buffer (250c), the control unit (210) can generate a sixth substream packet (935) of a predetermined size and having a predetermined value.

The control unit (210) can arrange the fourth sub-stream packet (915), the fifth sub-stream packet (925), and the sixth sub-stream packet (or garbage packet) (935) in a predetermined order and generate a bitstream (900) including the fourth sub-stream packet (915), the fifth sub-stream packet (925), and the sixth sub-stream packet (935).

In one embodiment, the control unit (210) may add information to the bitstream (900) that decryption of the sixth substream packet (935) is not required.

Fig. 10 is a flowchart for explaining a method of encoding an image according to one embodiment.

At step S1010, the encoding device (200) can identify multiple segments from the image.

In one embodiment, each of the plurality of segments may correspond to an image constituting an image sequence, a slice segmented from an image, a tile segmented from an image, or a maximum coding unit row segmented from an image.

In one embodiment, each of the plurality of segments may include one or more maximum coding units.

At step S1020, the encoding device (200) can encode multiple segments in parallel.

In one embodiment, the encoding device (200) may assign multiple segments to multiple cores (230) and/or multiple threads for parallel encoding of the multiple segments.

For allocation of multiple segments, the encoding device (200) may consider performance information of parallel encoding.

In one embodiment, performance information of parallel encoding may include the number of cores (230) capable of parallel processing of segments, the number of threads capable of parallel processing by at least one core (230), or the number of segments capable of parallel encoding.

In one embodiment, the encoding device (200) can determine the number of segments that can be encoded in parallel at one time, and can assign the identified number of segments among the plurality of segments identified from the image to the plurality of cores (230) and/or the plurality of threads. In addition, if there are remaining segments, the encoding device (200) can additionally assign the remaining segments to the plurality of cores (230) and/or the plurality of threads.

The encoding device (200) can encode a plurality of segments in parallel using at least one core (230). In one embodiment, the encoding device (200) can encode each of the plurality of segments based on a maximum encoding unit.

In step S1030, the encoding device (200) can store substreams generated during parallel encoding of multiple segments in multiple buffers (250).

In one embodiment, the plurality of segments to be encoded in parallel and the plurality of buffers may correspond 1:1. Accordingly, the plurality of substreams generated through encoding each of the plurality of segments may be stored 1:1 in the plurality of buffers.

In one embodiment, the number of the plurality of buffers (250) may be equal to the number of cores (230) capable of parallel processing the segments, the number of threads capable of parallel processing by at least one core (230), or the number of segments capable of parallel encoding.

At step S1040, the encoding device (200) can obtain substream packets having a predetermined size from the plurality of buffers (250) while the substreams are stored in the plurality of buffers (250), if the current state corresponds to a preset output condition.

In one embodiment, the output condition may be set based on at least one of the total size of the substreams stored in the plurality of buffers (250), the ratio between the capacity of the plurality of buffers (250) and the total size of the substreams stored in the plurality of buffers (250), the elapsed time of parallel encoding, or the progress of encoding for each of the plurality of segments.

In step S1050, the encoding device (200) can arrange the substream packets in a predetermined order and output a bitstream including the substream packets arranged in the predetermined order.

In one embodiment, the bitstream may further include additional information necessary to reconstruct the plurality of segments from the substream packets. In one embodiment, the additional information may be included in a header of the bitstream.

In one embodiment, the predetermined order may correspond to an encoding order of the plurality of segments.

In one embodiment, the predetermined order may correspond to a pre-arranged order between the encoding device (200) and the decoding device (1100).

In one embodiment, the encoding device (200) may repeatedly arrange substream packets obtained from a plurality of buffers (250) in a predetermined number of sequences in a predetermined order. Here, the predetermined number may correspond to a number confirmed from performance information of parallel encoding.

Hereinafter, a decoding device (1100) corresponding to the above-described encoding device (200) will be described with reference to FIGS. 11 to 14.

FIG. 11 is a block diagram illustrating the configuration of a decryption device (1100) according to one embodiment.

The decryption device (1100) may include a control unit (1110) and at least one core (1130).

Although FIG. 11 illustrates that the decryption device (1100) includes one core (1130), the number of cores (1130) that may be included in the decryption device (1100) may be multiple.

In one embodiment, the control unit (1110) and at least one core (1130) may be included in a processor.

In one embodiment, the control unit (1110) may be included in the first processor, and at least one core (1130) may be included in the second processor.

In one embodiment, when the decryption device (1100) includes multiple cores (1130), some of the multiple cores (1130) may be included in one processor, and others may be included in another processor.

In one embodiment, the processor for implementing the control unit (1110) and the at least one core (1130) may be a dedicated processor for decoding images. In one embodiment, the processor for implementing the control unit (1110) and the at least one core (1130) may be a general-purpose processor such as an application processor (AP), a central processing unit (CPU), or a graphic processing unit (GPU).

In one embodiment, at least one instruction for decrypting an image may be stored in a memory, and the control unit (1110) and at least one core (1130) may operate according to the at least one instruction stored in the memory.

The control unit (1110) can obtain a bitstream corresponding to the encoding result of the image.

In one embodiment, the control unit (1110) can receive a bitstream from the encoding device (200) over a network.

In one embodiment, the bitstream may be obtained from a data storage medium including a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical storage medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and the like.

In one embodiment, a bitstream may include substream packets of a predetermined size, arranged in a predetermined order.

In one embodiment, the control unit (1110) can identify multiple segments within an image to be restored in order to restore the image from the substream packets.

Each of the plurality of segments may correspond to an image constituting a video sequence, a slice segmented from an image, a tile segmented from an image, or a maximum coding unit row segmented from an image.

In one embodiment, the control unit (1110) transmits substream packets obtained from the bitstream to at least one core (1130), and the at least one core (1130) can decode the substream packets to restore a plurality of segments.

Substream packets included in a bitstream can correspond to each of multiple segments in a predetermined order.

In one embodiment, the predetermined order may correspond to the decoding order of the plurality of segments. Accordingly, if the decoding order of the first segment is earlier than the decoding order of the second segment, the sub-stream packet arranged earlier among the first sub-stream packet and the second sub-stream packet may correspond to the first segment, and the sub-stream packet arranged later may correspond to the second segment.

In one embodiment, the control unit (1110) may skip decoding of a substream packet if the bitstream contains information indicating that any of the substream packets is not a target of decoding. Accordingly, the substream packet may be deleted, and the next substream packet may be decoded by at least one core (1130).

The above-described encoding device (200) can encode a plurality of segments in parallel to generate substream packets. In one embodiment, at least one core (1130) can decode the substream packets included in the bitstream in parallel, or can decode the substream packets sequentially in a predetermined order.

In one embodiment, if the decoding device (1100) has the same parallel encoding performance as the encoding device (200), the control unit (1110) can allocate substream packets that can be parallel decoded at one time to multiple cores (1130) or multiple threads.

In one embodiment, the control unit (1110) can predetermine which segment each of the substream packets included in the bitstream is for. As the substream packets are decoded by at least one core (1130), the segments corresponding to the substream packets can be restored.

Since the control unit (1110) can determine in advance which segment each of the substream packets is for, parallel decoding of the substream packets may not necessarily be necessary. This is because even if the substream packets are decoded sequentially, it is possible to know which segment the decoded data is for.

In one embodiment, the control unit (1110) can check in advance the performance information of the parallel encoding of the encoding device (200). The performance information of the parallel encoding can be included in the bitstream.

The control unit (1110) can know from the performance information of parallel encoding how many substream packets are arranged in a predetermined order, and accordingly can know which segment each substream packet is for. This will be described with reference to FIGS. 12 to 14.

FIG. 12 is a diagram illustrating a method for restoring multiple segments from a bitstream according to one embodiment.

Referring to FIG. 12, the bitstream may include a first substream packet to a seventh substream packet.

In one embodiment, the control unit (1110) can determine how often the substream packets are arranged and repeated from the performance information of the parallel encoding.

For example, if the number confirmed from the performance information of parallel encoding is 4, the control unit (1110) can know that the substream packets are arranged in a predetermined order every 4.

The control unit (1110) can identify the first segment, the second segment, the third segment, and the fourth segment from the image. The control unit (1110) can determine, based on the decoding order of the first segment, the second segment, the third segment, and the fourth segment or the order agreed upon in advance with the encoding device (200), that the first sub-stream packet corresponds to the first segment, the second sub-stream packet corresponds to the second segment, the third sub-stream packet corresponds to the third segment, and the fourth sub-stream packet corresponds to the fourth segment.

If the number confirmed from the performance information is 4, the sub-stream packets are arranged in a predetermined order every 4, so the control unit (1110) can determine that the 5th sub-stream packet corresponds to the 1st segment, the 6th sub-stream packet corresponds to the 2nd segment, and the 7th sub-stream packet corresponds to the 3rd segment.

At least one core (1130) can decrypt the first sub-stream packet and the fifth sub-stream packet to restore the first segment, decrypt the second sub-stream packet and the sixth sub-stream packet to restore the second segment, and decrypt the third sub-stream packet and the seventh sub-stream packet to restore the third segment. Additionally, at least one core (1130) can decrypt the fourth sub-stream packet to restore the fourth segment.

FIG. 13 is a diagram illustrating a method for restoring multiple segments from a bitstream according to one embodiment.

In the embodiment illustrated in FIG. 13, it is assumed that the first segment, the second segment, the third segment, and the third fourth segment are identified from the image, and the number confirmed from the performance information is 3.

Accordingly, the first sub-stream packet and the fourth sub-stream packet may correspond to the first segment, the second sub-stream packet and the fifth sub-stream packet may correspond to the second segment, and the third sub-stream packet and the sixth sub-stream packet may correspond to the third segment.

In one embodiment, if the restoration of the first segment from the fourth sub-stream packet is not completed, the seventh sub-stream packet may correspond to the first segment. This is because the sub-stream packets are arranged in a predetermined order every three, and the seventh sub-stream packet, which is located at the seventh position, corresponds to the first segment according to the predetermined order.

In one embodiment, if the restoration of the first segment is completed through decoding of the fourth sub-stream packet, the seventh sub-stream packet located at the seventh position may correspond to the fourth segment. That is, this may mean that the first segment and the fourth segment were sequentially encoded through one core (1130) or one thread.

FIG. 14 is a diagram illustrating a method for restoring multiple segments from a bitstream according to one embodiment.

In one embodiment, the control unit (1110) may obtain information from the bitstream that a second substream packet among the substream packets in the bitstream is not a target of decoding. In this case, decoding of the second substream packet may be skipped.

If the number confirmed from the performance information of parallel encoding is 4 and the first to fourth segments are identified from the image, the control unit (1110) can determine that the first sub-stream packet corresponds to the first segment. The control unit (1110) can determine that the second sub-stream packet does not correspond to any segment because it is not a target of decoding.

The control unit (1110) can determine that the third sub-stream packet corresponds to the third segment, and the fourth sub-stream packet corresponds to the fourth segment. Since the sub-stream packets are arranged in a repeating manner every four, the control unit (1110) can determine that the fifth sub-stream packet corresponds to the first segment, the sixth sub-stream packet corresponds to the second segment, and the seventh sub-stream packet corresponds to the third segment.

At least one core (1130) can decrypt the first sub-stream packet and the fifth sub-stream packet to restore the first segment, decrypt the sixth sub-stream packet to restore the second segment, and decrypt the third sub-stream packet and the seventh sub-stream packet to restore the third segment. Additionally, at least one core (1130) can decrypt the fourth sub-stream packet to restore the fourth segment.

In one embodiment, the control unit (1110) can generate a restored image using the restored plurality of segments when the plurality of segments are restored by at least one core (1130).

In one embodiment, the control unit (1110) can combine multiple restored segments to generate a restored image.

In one embodiment, filtering (e.g., deblocking filtering) may be applied to the boundaries between restored segments.

In one embodiment, when a restored image is generated, the control unit (1110) can transmit the restored image to a playback device such as a display.

Fig. 15 is a flowchart for explaining a method of decoding an image according to one embodiment.

At step S1510, the decoding device (1100) can identify multiple segments within the image.

In one embodiment, each of the plurality of segments may correspond to an image constituting an image sequence, a slice segmented from an image, a tile segmented from an image, or a maximum coding unit row segmented from an image.

In one embodiment, each of the plurality of segments may include one or more maximum coding units.

At step S1520, the decryption device (1100) can obtain substream packets of a predetermined size arranged in a predetermined order from the bitstream.

In one embodiment, the decoding device (1100) can obtain performance information of parallel encoding of the encoding device (200) from the bitstream, and, based on the obtained performance information, can determine how often the substream packets are arranged and repeated.

At step S1530, the decryption device (1100) can decrypt substream packets through at least one core (1130) to restore multiple segments.

In one embodiment, the decryption device (1100) can decrypt the substream packets in parallel by assigning each of the number of substream packets identified from the performance information to a plurality of cores (1130) or a plurality of threads.

In one embodiment, the substream packets may correspond to each of the plurality of segments in a predetermined order, so that at least one core (1130) may restore the plurality of segments using the substream packets corresponding to each of the plurality of segments.

In one embodiment, the decoding device (1100) can restore each of the plurality of segments based on a maximum coding unit.

At step S1540, the decryption device (1100) can generate a restored image using the restored plurality of segments.

In one embodiment, the decryption device (1100) can generate a restored image by combining multiple restored segments.

In one embodiment, filtering (e.g., deblocking filtering) may be applied to the boundaries between restored segments.

In one embodiment, when a restored image is generated, the decryption device (1100) can transmit the restored image to a playback device such as a display.

FIG. 16 is a block diagram illustrating the configuration of a terminal device including an encoding device (200) or a decoding device (1100) according to one embodiment.

Referring to FIG. 16, a terminal device (1600) according to one embodiment may include a tuner unit (1640), a processor (1610), a display unit (1620), a communication unit (1650), a detection unit (1630), an input/output unit (1670), a video processing unit (1680), an audio processing unit (1685), an audio output unit (1660), a memory (1690), and a power supply unit (1695).

A tuner unit (1640) according to one embodiment can select only the frequency of a channel to be received among many radio wave components by amplifying, mixing, resonating, etc. a broadcast signal received wired or wirelessly. The broadcast signal includes audio, video, and additional information (e.g., EPG (Electronic Program Guide)).

The tuner unit (1640) can receive broadcast signals from various sources, such as terrestrial broadcasting, cable broadcasting, satellite broadcasting, and Internet broadcasting. The tuner unit (1640) can also receive broadcast signals from sources, such as analog broadcasting or digital broadcasting.

The detection unit (1630) detects the user's voice, the user's image, or the user's interaction, and may include a microphone (1631), a camera unit (1632), and a light receiving unit (1633).

The microphone (1631) receives the user's uttered voice. The microphone (1631) can convert the received voice into an electric signal and output it to the processor (1610). The user's voice may include, for example, a voice corresponding to a menu or function of the terminal device (1600).

The camera unit (1632) can receive images (e.g., consecutive frames).

The optical receiver (1633) receives an optical signal (including a control signal) from an external control device through an optical window (not shown) of a bezel of the display unit (1620). The optical receiver (1633) can receive an optical signal corresponding to a user input (e.g., touch, pressing, touch gesture, voice, or motion) from the control device. A control signal can be extracted from the received optical signal under the control of the processor (1610).

The input/output unit (1670) receives video (e.g., moving images, etc.), audio (e.g., voice, music, etc.), and additional information (e.g., EPG, etc.) from the outside of the terminal device (1600) under the control of the processor (1610). The input/output interface may include any one of HDMI (High-Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), DVI (Digital Visual Interface), component jack, and PC port.

The processor (1610) controls the overall operation of the terminal device (1600) and the signal flow between the internal components of the terminal device (1600), and performs a function of processing data. The processor (1610) can execute an OS (Operation System) and various applications stored in the memory (1690) when there is a user input or a preset and stored condition is satisfied.

The processor (1610) may include a RAM that is used as a storage area for storing signals or data input from the outside of the terminal device (1600) or corresponding to various tasks (e.g., image noise removal tasks) performed in the terminal device (1600), a ROM that stores a control program for controlling the terminal device (1600), and a processor.

The video processing unit (1680) performs processing on video data received by the terminal device (1600). The video processing unit (1680) can perform various image processing such as encoding, decoding, scaling, noise filtering, frame rate conversion, and resolution conversion on the video data.

In one embodiment, the encoding process of the aforementioned encoding device (200) or the decoding process of the decoding device (1100) may be performed by the video processing unit (1680).

The audio processing unit (1685) performs processing on audio data. The audio processing unit (1685) may perform various processing such as decoding, amplification, and noise filtering on audio data. Meanwhile, the audio processing unit (1685) may be equipped with a plurality of audio processing modules to process audio corresponding to a plurality of contents.

The audio output unit (1660) outputs audio included in a broadcast signal received through the tuner unit (1640) under the control of the processor (1610). The audio output unit (1660) can output audio (e.g., voice, sound) input through the communication unit (1650) or the input/output unit (1670). In addition, the audio output unit (1660) can output audio stored in the memory (1690) under the control of the processor (1610). The audio output unit (1660) can include at least one of a speaker, a headphone output terminal, or a S/PDIF (Sony/Philips Digital Interface: output terminal).

The power supply unit (1695) supplies power input from an external power source to components inside the terminal device (1600) under the control of the processor (1610). In addition, the power supply unit (1695) can supply power output from one or more batteries (not shown) located inside the terminal device (1600) to the internal components under the control of the processor (1610).

The memory (1690) can store various data, programs or applications for driving and controlling the terminal device (1600) under the control of the processor (1610). The memory (1690) can include a broadcast reception module, a channel control module, a volume control module, a communication control module, a voice recognition module, a motion recognition module, an optical reception module, a display control module, an audio control module, an external input control module, a power control module, a power control module for an external device connected wirelessly (for example, Bluetooth), a voice database (DB), or a motion database (DB). The modules and the database of the memory (1690) can be implemented in the form of software to perform a mirroring function, a broadcast reception control function, a channel control function, a volume control function, a communication control function, a voice recognition function, a motion recognition function, an optical reception control function, a display control function, an audio control function, an external input control function, a power control function or a power control function for an external device connected wirelessly (for example, Bluetooth) in the terminal device (1600). The processor (1610) can perform each function using the software stored in the memory (1690).

Meanwhile, the block diagram of the terminal device (1600) illustrated in FIG. 16 is a block diagram for one embodiment. Each component of the block diagram may be integrated, added, or omitted according to the specifications of the terminal device (1600) actually implemented. That is, two or more components may be combined into one component, or one component may be subdivided into two or more components, as needed. In addition, the functions performed by each block are for explaining embodiments, and the specific operations or devices thereof do not limit the scope of the present invention.

One embodiment aims to minimize the delay between the process of encoding an image and the process of decoding the image.

Additionally, one embodiment aims to minimize delay that may occur during the output process of a bitstream generated through parallel encoding of an image.

In addition, one embodiment aims to provide a codec suitable for applications requiring low latency characteristics, such as wireless TV, cloud gaming, AR (Augmented Reality) and VR (Virtual Reality).

The technical problems to be achieved through the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary skill in the technical field to which the present disclosure belongs from the description below.

A method of encoding an image by an encoding device (200) according to one embodiment may include a step of identifying a plurality of segments from an image.

A method of encoding an image by an encoding device (200) according to one embodiment may include a step of encoding a plurality of segments in parallel.

A method of encoding an image by an encoding device (200) according to one embodiment may include a step of storing substreams generated through parallel encoding in a plurality of buffers (250).

A method of encoding an image by an encoding device (200) according to one embodiment may include a step of obtaining substream packets having a predetermined size from a plurality of buffers (250) when a current state corresponds to a preset output condition while the substreams are stored in a plurality of buffers (250).

A method of encoding an image by an encoding device (200) according to one embodiment may include a step of arranging substream packets in a predetermined order and outputting a bitstream in which the substream packets are arranged.

According to a method for encoding an image according to one embodiment, a time delay that may occur during an image encoding process can be minimized.

In one embodiment, the preset output condition may be set based on at least one of the total size of the substreams stored in the plurality of buffers (250), the ratio between the capacity of the plurality of buffers (250) and the total size of the substreams stored in the plurality of buffers (250), the elapsed time of parallel encoding, or the progress of encoding for each of the plurality of segments.

According to one embodiment, various output conditions can be set, so that the amount of delay occurring during the encoding process of an image can be variously adjusted.

In one embodiment, the predetermined order corresponds to an encoding order of a plurality of segments, and if the encoding order of the first segment is earlier than the encoding order of the second segment, the sub-stream packet corresponding to the first segment may be arranged before the sub-stream packet corresponding to the second segment.

In one embodiment, since the substream packets are arranged in a predetermined order, it is easy to determine which segment each substream packet corresponds to.

In one embodiment, the outputting step may include: generating a substream packet of a predetermined size, comprising bits of a predetermined value, when a size of a substream stored in a first buffer among the plurality of buffers (250) is less than a predetermined size; and arranging the generated substream packets in a predetermined order.

In one embodiment, a constant bitrate can be maintained by generating garbage packets even if there is not enough data stored in the buffer.

In one embodiment, the step of outputting the bitstream may include the step of including in the bitstream information that the generated substream packets are not subject to decryption.

In one embodiment, decoding of certain substream packets may be skipped, thereby preventing segments from being incorrectly restored.

In one embodiment, the plurality of segments may be encoded in parallel by the plurality of cores (230a, 230b, 230c), or the plurality of segments may be assigned to the plurality of threads (235a, 235b, 235c), and the plurality of threads (235a, 235b, 235c) may be encoded in parallel by at least one core (230).

In one embodiment, rapid encoding of images may be possible through various types of parallel encoding.

In one embodiment, each of the plurality of segments may include a slice, a tile, or a maximum coding unit row.

According to one embodiment, parallel encoding is performed based on slices, tiles, or maximum encoding unit rows specified in a video standard, so that it can be utilized in various applications.

A method of decoding an image by a decoding device (1100) according to one embodiment may include a step of identifying a plurality of segments within the image.

A method of decoding an image by a decoding device (1100) according to one embodiment may include a step of obtaining substream packets of a predetermined size arranged in a predetermined order from a bitstream. In one embodiment, the substream packets may be generated through parallel encoding of a plurality of segments.

A method of decoding an image by a decoding device (1100) according to one embodiment may include a step of decoding substream packets to restore a plurality of segments.

A method of decoding an image by a decoding device (1100) according to one embodiment may include a step of generating a restored image including a plurality of restored segments.

In one embodiment, the substream packets may correspond to each of the plurality of segments in a predetermined order.

According to a method for decoding an image according to one embodiment, time delay that may occur during an image encoding process and an image decoding process can be minimized.

In one embodiment, if the bitstream includes information that a first substream packet among the substream packets is not a target for decoding, decoding of the first substream packet may be skipped.

In one embodiment, decoding of certain substream packets may be skipped, thereby preventing segments from being incorrectly restored.

In one embodiment, the method of decoding an image further includes the step of obtaining information on the number of cores or threads used for parallel encoding from a bitstream, and the substream packets may be arranged in a predetermined order for each number of cores or threads.

In one embodiment, since the substream packets are arranged in a predetermined order, it is easy to determine which segment each substream packet corresponds to.

In one embodiment, when the number of cores or threads is n (where n is a natural number greater than 1) and the number of multiple segments is m (where m is a natural number greater than or equal to n), the n substream packets are arranged in a predetermined order, and each of the n substream packets can correspond to each of n segments among the m segments in the predetermined order.

An encoding device (200) according to one embodiment may include a control unit (210) that identifies a plurality of segments from an image.

An encoding device (200) according to one embodiment may include at least one core (230) that encodes a plurality of segments in parallel.

An encoding device (200) according to one embodiment may include a plurality of buffers (250).

In one embodiment, the control unit (210) can store substreams generated through parallel encoding in multiple buffers (250).

In one embodiment, the control unit (210) may obtain substream packets having a predetermined size from the plurality of buffers (250) when the current state corresponds to a preset output condition while the substreams are stored in the plurality of buffers (250).

In one embodiment, the control unit (210) can arrange the substream packets in a predetermined order and output a bitstream in which the substream packets are arranged.

According to an encoding device (200) according to one embodiment, a time delay that may occur during an image encoding process can be minimized.

In one embodiment, the control unit (210) may, when the size of a substream stored in a first buffer among the plurality of buffers (250) is less than a predetermined size, generate a substream packet of a predetermined size, consisting of bits of a predetermined value, and arrange the generated substream packets in a predetermined order.

In one embodiment, a constant bitrate can be maintained by generating garbage packets even if there is not enough data stored in the buffer.

A decoding device (1100) according to one embodiment may include a control unit (1110) that identifies a plurality of segments in an image and obtains substream packets of a predetermined size arranged in a predetermined order from a bitstream.

A decryption device (1100) according to one embodiment may include at least one core (1130) that decrypts substream packets to restore a plurality of segments.

In one embodiment, the substream packets are generated through parallel encoding of a plurality of segments, and may correspond to each of the plurality of segments in a predetermined order.

In one embodiment, the control unit (1110) can generate a restored image including a plurality of restored segments.

According to a decoding device (1100) according to one embodiment, time delay that may occur during an image encoding process and an image decoding process can be minimized.

One embodiment can minimize the delay between the process of encoding an image and the process of decoding the image.

Additionally, one embodiment can minimize delay that may occur during the output process of a bitstream generated through parallel encoding of an image.

Additionally, one embodiment may provide a codec suitable for applications requiring low latency characteristics, such as wireless TV, cloud gaming, AR (Augmented Reality), and VR (Virtual Reality).

The effects obtainable from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by a person skilled in the art to which the present disclosure belongs from the description below.

Meanwhile, the embodiments of the present disclosure described above can be written as a program that can be executed on a computer, and the written program can be stored in a storage medium that can be read by a machine.

A storage medium that can be read by the device may be provided in the form of a non-transitory storage medium. Here, the term 'non-transitory storage medium' means only that it is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently in the storage medium and cases where data is stored temporarily. For example, a 'non-transitory storage medium' may include a buffer in which data is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be at least temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or an intermediary server.

Above, the technical idea of the present disclosure has been described in detail with reference to preferred embodiments, but the technical idea of the present disclosure is not limited to the above embodiments, and various modifications and changes are possible by a person having ordinary knowledge in the relevant field within the scope of the technical idea of the present disclosure.

Claims (15)

In a method of encoding an image using an encoding device (200),
A step of identifying multiple segments from an image;
A step of parallel encoding the above plurality of segments;
A step of storing substreams generated through the above parallel encoding in multiple buffers (250);
While the above substreams are stored in the plurality of buffers (250), if the current state corresponds to a preset output condition, a step of obtaining substream packets having a predetermined size from the plurality of buffers (250); and
A method for encoding a video, comprising the steps of arranging the substream packets in a predetermined order and outputting a bitstream in which the substream packets are arranged.
In the first paragraph,
The above preset output conditions are:
A method for encoding a video, wherein the method is set based on at least one of the total size of the substreams stored in the plurality of buffers (250), the ratio between the capacity of the plurality of buffers (250) and the total size of the substreams stored in the plurality of buffers (250), the elapsed time of the parallel encoding, or the progress of encoding for each of the plurality of segments.
In any one of the first and second paragraphs,
The above predetermined order is,
Corresponding to the encoding order of the above multiple segments,
A method for encoding a video, wherein if the encoding order of the first segment is earlier than the encoding order of the second segment, the substream packet corresponding to the first segment is arranged earlier than the substream packet corresponding to the second segment.
In any one of claims 1 to 3,
The above output step is,
When the size of the substream stored in the first buffer among the plurality of buffers (250) is less than the predetermined size, a step of generating a substream packet of the predetermined size, which is composed of bits of a predetermined value; and
A method for encoding a video, comprising the step of arranging the generated substream packets in the predetermined order.
In any one of claims 1 to 4,
The step of outputting the above bitstream is:
A method for encoding a video, comprising the step of including information in the bitstream that the generated substream packet is not a target of decoding.
In any one of claims 1 to 5,
A method for encoding a video, wherein the plurality of segments are encoded in parallel by a plurality of cores (230a, 230b, 230c), or the plurality of segments are allocated to a plurality of threads (235a, 235b, 235c), and the plurality of threads (235a, 235b, 235c) are encoded in parallel by at least one core (230).
In any one of claims 1 to 6,
Each of the above multiple segments,
A method of encoding an image, comprising slices, tiles or rows of maximum coding units.
A computer-readable recording medium having recorded thereon a program for performing on a computer the method of encoding an image according to any one of claims 1 to 7.
In a method of decoding an image using a decoding device (1100),
A step of identifying multiple segments within an image;
A step of obtaining substream packets of a predetermined size arranged in a predetermined order from a bitstream, the substream packets being generated through parallel encoding of the plurality of segments;
a step of decrypting the above substream packets to restore the plurality of segments; and
A step of generating a restored image including the restored plurality of segments,
A method for decoding a video, wherein the substream packets correspond to each of the plurality of segments in the predetermined order.
In Article 9,
A method for decoding a video, wherein if the bitstream includes information that a first substream packet among the above substream packets is not a target of decoding, decoding of the first substream packet is skipped.
In any one of the clauses 9 to 10,
The method of decrypting the above video is as follows:
Further comprising a step of obtaining information on the number of cores or threads used for the parallel encoding from the bitstream,
A method for decoding a video, wherein the substream packets are arranged in a predetermined order for each number of cores or threads.
In any one of the clauses 9 to 11,
A method for decoding a video, wherein when the number of the cores or threads is n (n is a natural number greater than 1) and the number of the plurality of segments is m (m is a natural number greater than or equal to n), n substream packets are arranged in the predetermined order, and each of the n substream packets corresponds to each of n segments among the m segments in the predetermined order.
A control unit (210) for identifying multiple segments from an image;
At least one core (230) for parallel encoding of the plurality of segments; and
Including multiple buffers (250),
The above control unit (210)
The substreams generated through the above parallel encoding are stored in the plurality of buffers (250),
While the above substreams are stored in the plurality of buffers (250), if the current state corresponds to a preset output condition, substream packets having a predetermined size are obtained from the plurality of buffers (250),
An encoding device that arranges the above substream packets in a predetermined order and outputs a bitstream in which the above substream packets are arranged.
In Article 13,
The above control unit (210)
An encoding device that generates a substream packet of the predetermined size, which is composed of bits of a predetermined value, when the size of a substream stored in a first buffer among the plurality of buffers (250) is less than the predetermined size, and arranges the generated substream packets in the predetermined order.
A control unit (1110) for identifying multiple segments in an image and obtaining substream packets of a predetermined size arranged in a predetermined order from a bitstream; and
At least one core (1130) for decrypting the above substream packets and restoring the plurality of segments,
The above substream packets are generated through parallel encoding of the above multiple segments,
The above substream packets correspond to each of the above plurality of segments in the above predetermined order,
The above control unit (1110) is a decoding device that generates a restored image including the restored plurality of segments.

Images