JP2007166406A - Image processing method, image processing apparatus, computer program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image communication method for promptly presenting an important region by setting an ROI in an image in advance and transmitting image data. In such a conventional method, the ROI is first processed on the receiving side. In addition, an operation different from the normal decoding method is required, and it is difficult to dynamically change the important area.
An ROI is created in a pseudo manner by using a tile part of JPEG2000. As a result, the receiving side can automatically display the important area first by a decoding method similar to that of ordinary JPEG2000. Furthermore, since the tile part can be easily created, the ROI can be dynamically changed according to the client.
[Selection] Figure 1

Description

  The present invention relates to image data creation technology.

  In recent years, digital cameras that have spread rapidly have an enormous number of pixels, and those with about 2 to 5 million pixels have become common. Furthermore, when it comes to professional digital cameras, there are 10 million pixel class cameras.

  Furthermore, an increasing number of users enjoy uploading images taken with a digital camera to an online album service. While the size of the image data increases, it takes time to download such a high-definition image on a thin data line such as a mobile communication environment. In raster display in which detailed images are displayed in order from the upper left to the lower right, it takes a very long time until the user can determine whether the downloaded image is a necessary image.

  Therefore, there is progressive display as an image display method for shortening the time until the user grasps the entire image. Progressive display means that the entire image is first displayed at a low resolution and then displayed in detail gradually.

  Typical examples include progressive JPEG and interlaced GIF. By using this method, the user can determine whether the image is a necessary image at a relatively early stage. However, with this method, if there is an important area in the center part or the lower half of the screen, the entire image still needs to be displayed in order to see the details of the important area. If the details of the image affect the necessity / unnecessity of the image, it cannot be determined unless the entire image is displayed.

  In order to cope with such a problem, a method for enlarging and displaying only a designated portion of an image and a method for transmitting an important region first have been proposed.

  A method for enlarging and displaying only a specified portion of an image can be realized by using JPEG2000 Part 9: Interactivity tools, APIs and protocols (hereinafter, JPIP) standardized by ISO. If you use JPIP and JPEG2000, you can see an enlarged image of the partial area. However, in the case of such a partial display, it is difficult to grasp which part of the whole is located. In addition, when JPIP is used, the received data must be cached and the data structure unique to JPIP must be analyzed, so that the downloaded image cannot be displayed only with a normal JPEG2000 decoder.

  The following documents are known as methods for transmitting image data of an important area before other areas.

  In Reference 1, the position of the important area is transmitted together with the image data, and the image data transmission order is low resolution of the important area, low resolution other than the important area, high resolution of the important area, and high resolution other than the important area. This is the order. With this method, it is easy to grasp the position of the important area in the entire image, and even if the important area is in the center or lower half of the image, the time until confirmation of details can be shortened. it can. However, it is necessary to match the position information with the rendering position of the received data. That is, there remains a problem that the image cannot be displayed simply by performing normal decoding processing on the received data, such as progressive JPEG or interlaced GIF.

Another method for transmitting the important area first is the method using JPEG2000 ROI. In JPEG2000, when data is sent from the head of image data by bit-shifting the ROI-designated area, the bit-shifted ROI area is transmitted earlier than other data. With this method, as in JP 2003-333593, the position of the important area in the entire image can be easily grasped, and even if the important area is in the center or lower half of the image, the details of The time until confirmation can be shortened. However, in order to change the ROI, re-encoding is required, and when the ROI is dynamically changed, a large load is imposed on the server side.
JP 2003-333593 A

  In the present invention, an important region can be dynamically changed only by rearranging data without re-encoding in response to a user request, and the received data is simply decoded from the top. Thus, the transmission image data that can display the important area earlier than the other areas is created.

  Further, special processing is not required for rendering, and transmission image data that can display an important area at an earlier stage than other areas is created by an operation similar to normal decoding.

  For example, according to the image processing method of the present invention, a series of hierarchically encoded data that is divided into tiles and has one or more layers is stored in one or more logical groups. An important area specifying step for specifying an important area, an important tile specifying process for specifying a tile including the important area specified by the important area specifying process, and a plurality of physical encoded data constituting each tile. And a group output unit that outputs the encoded data of each tile to the group specified by the group belonging step, together with a group identification unit that identifies each group for each group. The tiles identified by the important tile identification step in the first group of tiles are the other tiles. Characterized by also storing the encoded data of many hierarchy.

  According to the above configuration, images can be displayed in high definition sequentially from the important partial areas. Therefore, before the display of the entire image is finished, a high-resolution image is displayed for the important part.

Example 1
Hereinafter, it demonstrates concretely using drawing.

  FIG. 1 is a diagram showing an outline of a system for realizing the present embodiment. The CPU 101 controls the operation of the entire system and executes a program stored in the primary storage 102. The primary storage 102 is mainly a memory, and reads and stores programs stored in the secondary storage 103.

  The secondary storage 103 corresponds to, for example, a hard disk. Generally, the capacity of the primary storage is smaller than the capacity of the secondary storage, and programs and data that cannot be stored in the primary storage are stored in the secondary storage. Further, data that must be stored for a long time is also stored in the secondary storage. In this embodiment, the program is stored in the secondary storage 103, read into the primary storage 102 when the program is executed, and the CPU 101 executes the execution process.

  Examples of the input device 104 include a mouse and a keyboard. Used to send an interrupt signal to a program or the like. Examples of the output device 105 include a monitor and a printer. Data such as images is transmitted / received via the network interface 106. Various other configurations of the apparatus configuration method are conceivable, but the description thereof is omitted because it is not the main point of the present invention.

  FIG. 2 is a diagram showing an outline of a system for realizing the present embodiment. 201 represents a user 1, 202 represents a user 2, and has the system described in FIG. A plurality of users such as the user 1 or the user 2 can communicate with the server 204 via the network 203 regardless of wired or wireless.

  The server 204 has a large-capacity storage device 205 that stores image files. The large-capacity storage device 205 corresponds to, for example, a hard disk. This hard disk stores a large number of image data encoded by the JPEG2000 encoding method. The users 201 and 202 receive JPEG2000 files from the JPEG2000 image data stored in the server 204 and display them on the display device.

  Next, general JPEG2000 encoded data will be described with reference to FIG. The JPEG2000 encoded data has data describing the encoding conditions of the entire image, such as the resolution level number and the layer number, called main header data 301, at the top, and tile data is arranged behind the data.

  The tile data has tile header data called a tile part header (Tile-part header) 302, and has encoded unit data called packet 303 constituting the encoded data of the tile behind the tile data. Since each tile is encoded independently, it can be decoded in tile units.

  FIG. 3 shows a configuration of a JPEG2000 file in which packet data is recorded in accordance with Layer-resolution level-component-position progression (LRCP).

  When conforming to LRCP, it is recorded in the order of layer / resolution / component / position.

  This way of arranging data is called progress order.

  The position described here is a precinct in JPEG2000 encoded data.

  If the progression order is LRCP as shown in FIG. 3, each layer stores data in ascending order of resolution number. The layer number corresponds to the S / N ratio with respect to the original image of the image to be restored. The smaller the layer number, the worse the S / N ratio.

  The maximum values of resolution number, layer number, and component number in one JPEG2000 file are preset by the encoder, encoded according to the parameters, and the information is stored in the encoded data.

  In FIG. 3, to simplify the explanation of the structure of JPEG2000, tiles are represented as a single chunk of data, but in reality, each tile is divided into a plurality of tile-parts. be able to. Figure 4 shows the relationship between tiles and tile parts. In FIG. 3, as shown in FIG. 4 (a), all tile data are grouped together using one tile part header 400.

  The tile data shown in Fig. 4 (a) is divided into three parts, 410, 411, 412, and tile parts headers 420, 421, 422 are attached to the tiles as shown in Fig. 4 (b). Part division is possible. When the tile data is divided, it is divided in units of packets. That is, tile data cannot be divided in the middle of a packet.

  FIG. 4 (c) shows the structure of the tile part header. As shown in FIG. 4 (c), the tile part header describes the tile number, the data length of the tile part, the tile part number, and the total number of tile parts of the tile.

  Also, in JPEG2000 encoded data, tile part 0 for all tiles must be arranged in tile order and immediately after the main header. However, for the other tile parts, the tile parts related to each tile need to be arranged in ascending order, but the individual tile parts do not need to appear in succession. An example of the order of tile parts is shown in FIG.

<How to determine the ROI when registering images on the server>
In this embodiment, when storing a JPEG2000 image on the large-capacity hard disk 205 of the server 204, the image provider designates the ROI, and the image can be received and decoded by the client prior to other areas. Convert to data.

  An image registered by the image provider in the large-capacity hard disk 205 is encoded as JPEG2000 encoded data under the following decoding conditions.

Image size: 1600 × 1200 [pixel]
Tile size: 256 x 256 [pixel]
Number of resolution levels: 4
resolution level 0… 200 × 150 [pixel]
resolution level 1… 400 × 300 [pixel]
resolution level 2… 800 × 600 [pixel]
resolution level 3… 1600 × 1200 [pixel]
Number of layers: 1
Number of positions: 1 position per tile (ie no position division)
Number of components: 3
FIG. 6 shows the tile division of the image in this embodiment. Assume that the image provider designates the church included in the area 601 in FIG. 6 as the ROI. In this case, the area 601 is converted into a file that can be downloaded by the client before other areas.

  The image file conversion method will be described with reference to FIG.

First, in step S701, an image to be converted is acquired. In step S702, the main header of the image acquired in step S701 is analyzed, and the resolution level number Ro and the tile total number Ttotal are acquired. In this embodiment, Ro = 4, Ttotal = 35 (= 7 × 5). In step S703, the ROI area designated by the image provider is acquired.
Various methods can be considered, such as a method of specifying on the screen using a mouse or a method of specifying with a numerical value, but since it is not the main point of this case, a detailed description is omitted. In the present embodiment, it is assumed that the church portion included in the area 601 in FIG. 6 is designated as the ROI.

  In step S704, the tile number group Troi including the ROI area acquired in step S703 is converted. The conversion of the tile number can be easily obtained by converting the coordinates specified in step S703 into a value at the maximum resolution and dividing by the tile size.

  In this embodiment, Troi = {7, 8, 14, 15, 21, 22}. In step S705, initialization is performed by substituting 0 into the tile part counter TPn. In step S706, 0 is assigned to the tile counter T to be initialized. In step S707, tile part data is created. Detailed description of the creation of the tile part will be given later with reference to FIG.

  In step S708, 1 is added to the tile counter T. In step S709, the tile counter T is compared with the total number of tiles Ttotal. If they are the same, it is determined that the tile part TPn has been created for all the tiles, and the process proceeds to step S710. Proceed to

  In the case of the present embodiment, when steps S707 and S708 are repeated 35 times, T = 35, and the process proceeds to step S710. In step S710, 1 is added to the tile part counter TPn. In step S711, the tile part counter TPn is compared with the resolution level number Ro.If TPn is less than or equal to Ro, the process returns to step S706.If TPn is greater than Ro, all the TPn has been created. Exit.

  In the case of the present embodiment, when step S706 to step S710 are repeated five times, TPn = 5> 4 = Ro, and this flow is finished. That is, for each tile, as many tile parts as the resolution level number Ro are created.

  Next, a method for creating a tile part will be described with reference to FIG.

  In step S801, it is determined whether or not the tile counter T is included in the Troi acquired in step S703. If it is included in Troi, the process proceeds to step S802, and if not included, the process proceeds to step S806.

  In the case of this embodiment, Troi = {7, 8, 14, 15, 21, 22}. Therefore, if T is any value of 7, 8, 14, 15, 21, 22, the process proceeds to step S802. In step S802, it is checked whether the value of the tile part counter TPn is 0. If it is 0, the process proceeds to step S803, and if it is not 0, the process proceeds to step S804. In step S803, tile part 0 is created using data of resolution level 0 to (Ro-2).

  In this embodiment, the process of step S803 is performed when T = 7 and TPn = 0. Extract data of tile 7 resolution level 0, 1, 2 from the encoded data, obtain the data length, and assign tile number 7 to Isot 431 and resolution level 0, 1, 2 to Psot 432 in FIG. The tile part data length (= 14), the tile part number 0 in Tpsot 433, and the total number of tile parts (Ro + 1) = 5 are written in Tnsot 434 to create a tile header.

  Then, the data of the resolution level 0, 1, 2 of the tile 7 extracted before the tile header is continued. Therefore, only the resolution level 3 of the tile 7 remains.

  If the value of the tile part counter TPn is not 0 in step S802, the process proceeds to step S804. In step S804, it is determined whether the value of the tile part counter TPn is 1. If TPn = 1, the process proceeds to step S805, and if TPn is not 1, the process proceeds to step S807.

  In this embodiment, when T = 7 and TPn = 1, the process proceeds to step S804. In this case, only data of resolution level 3 that is not included in tile part 0 of tile 7 becomes data of tile part 1.

  At this time, as described above, tile number 7 is set in Isot 431, data length of resolution level 3 + data length of tile header is set in Psot 432, tile part number 1 is set in Tpsot 433, and the total number of tile parts is set in Tnsot 434. Write (Ro + 1) = 5, create a tile header, followed by resolution level 3 data. Therefore, all data of tile Troi included in ROI is divided into tile part 0 and tile part 1.

  If the tile counter T is not included in the tile group Troi in step S801, the process proceeds to step S806. In step S806, if the tile part counter TPn is 0, the process proceeds to step S807, and if not 0, the process proceeds to step S808.

  That is, the process of step S807 is performed when the tile part number is 2 or more among tiles included in Troi and when the tile part number is 0 among tiles not included in Troi. In step S807, only tile part headers of tile T and tile part TPn are created.

  In the present embodiment, the process of step S807 is performed when the tile part 2 is tile part 2, 3, or 4 of the tile 7 included in the ROI, or when the tile part 0 is tile 0 that is outside the ROI. If it is tile part 2 of tile 7, tile number 7 is assigned to Isot 431, data length 14 of the tile header is assigned to Psot 432, tile part number 2 is assigned to Tpsot 433, and the total number of tile parts (Ro + 1) Write 5 and create a tile header.

  If TPn is not 0 in step S806, the process proceeds to step S808. In step S808, a tile part TPn is created from the data of resolution level (TPn-1).

  In the case of the present embodiment, step S808 is passed when tile 0 and tile part 1 are set. Therefore, obtain the data length of resolution level 0 of tile 0, tile number 0 in Isot 431, data length of resolution level 0 + data length of tile header (= 14) in Psot 432, tile part number in Tpsot 433 1 is written in Tnsot 434 and the total number of tile parts (Ro + 1) = 5, a tile header is created, and data of resolution level 0 is continued after the tile header.

  Therefore, tiles that are not included in the ROI do not contain data in tile part 0, and data is stored for each resolution level after tile part 1.

  As described above, the data of this embodiment is arranged in the tile part order as shown in FIG. 5 (a), and the resolution data of the tiles included in the ROI is divided into each tile part as shown in FIG. The resolution data of the tile not included is divided into tile parts as shown in FIG. 9 (b).

  When such data is received from the beginning of the file and decoded for each tile part, the display on the client side is as follows.

1. An image with a resolution level one level below the highest resolution included in the ROI
2. Non-ROI area resolution level 0 image and ROI highest resolution image
3. Progressive display of high-definition images in areas other than ROI Therefore, the client can see some high-definition images in the important areas at an early stage of downloading, and in the next stage, a rough view of the entire image. Can also be seen. Moreover, the encoded data can be decoded using a standard JPEG 2000 decoder.

(Example 2)
In the first embodiment, when the image provider registers an image on the large-capacity hard disk 205 on the server side, the ROI is created at the discretion of the image provider. This ROI is dynamically generated according to the client. May be.

  For example, a keyword is assigned to a partial area of an image, and in the image search system, the search keyword of the user is compared with the keyword of the partial area of the image, the ROI area is dynamically changed, and the data is rearranged immediately before transmission. You can go.

  For example, when searching with the keyword “church”, if the region 601 is ROI, if the name of the person is a keyword, the region 602 containing the person is ROI, and if searching with the keyword “flower” In this case, the data may be rearranged and transmitted using the area 603 as the ROI.

  In the first embodiment, one ROI area is used, but a plurality of ROI areas may be designated. For example, it is possible to designate two locations, ROI, of the region 601 including the church and the region 603 including tulips in FIG.

  In the first embodiment, the data is divided at the resolution level, but may be divided at the layer. In other words, tile part 0 of tiles included in the ROI area includes only data with a certain level of image quality, tile layer 1 includes all layer data, and data is stored in tile part 0 of tiles other than the ROI area. The layer data is stored in order from tile part 1.

  According to the embodiment described above, images can be displayed so as to become high definition sequentially from the important partial areas. Therefore, before the display of the entire image is finished, a high-resolution image is displayed for the important part.

  In addition, since areas other than the important partial areas are displayed roughly at an early stage, the entire image can be grasped next to the important partial areas. Finally, an image is displayed at a high resolution other than the important partial area.

  Furthermore, the data of important partial areas is created using JPEG2000 tile parts, and all tile part 0 data is arranged immediately after the main header, so only normal decoding is performed and rendering is done in tile order. And it is displayed from the important area naturally.

  Furthermore, the generation of the important area data by the tile part can be easily realized only by rearranging the data without re-encoding the image data. Therefore, it is possible to easily change the ROI dynamically according to the client request.

Configuration diagram of computer system Server / client configuration diagram Configuration diagram of JPEG2000 code data Relationship diagram between tiles and tile parts Illustration of tile part order Diagram of image tile division in the first embodiment Diagram of image conversion flow in Example 1 Diagram of tile part creation flow in Embodiment 1 Example of resolution data division in P1

Claims (13)

An image processing method in which a series of hierarchically encoded data divided into tiles and having one or more layers is divided and stored in one or more logical groups,
An important area identification process for identifying an important area;
An important tile specifying step for specifying a tile including the important region specified by the important region specifying step;
Belonging group specifying step of distributing a plurality of physical encoded data constituting each tile into a plurality of groups,
A code output step for outputting the encoded data of each tile to the group specified by the belonging group specifying step together with group identification means for identifying each group for each group,
An image processing method characterized in that, among the leading groups of each tile, encoded data of a higher layer is stored in a tile specified by the important tile specifying step than in other tiles.   The image processing method according to claim 1, wherein the group identification unit is header information for grouping grouped code data.   The image processing method according to claim 1, wherein the hierarchically encoded data has a hierarchy in a resolution direction.   The image processing method according to claim 1, wherein the hierarchically encoded data has a hierarchy in an image quality direction.   The encoded data stored in the leading group of the tile specified by the important tile specifying step is encoded data having a higher resolution hierarchy than the leading group of other tiles. The image processing method as described.   The encoded data stored in the leading group of the tile specified by the important tile specifying step is encoded data having a higher image quality layer than the leading group of other tiles. The image processing method as described.   The image processing method according to claim 1, wherein the area specified by the important area specifying step is specified from the incidental information of the area.   The image processing method according to claim 1, wherein the hierarchically encoded data is JPEG2000.   The image processing method according to claim 7, wherein the physically encoded data is a JPEG2000 packet.   The image processing method according to claim 7, wherein the group identification unit is a Tile-part header. An image processing apparatus that is divided into tiles and stores a series of hierarchically encoded data having one or more layers divided into one or more logical groups,
An important area identification means for identifying an important area;
An important tile specifying means for specifying a tile including an important area specified by the important area specifying means;
Belonging group specifying means for allocating a plurality of physical encoded data constituting each tile to a plurality of groups,
Along with a group identifying means for identifying each group for each group, a code output means for outputting the encoded data of each tile to the group specified by the belonging group specifying means,
An image processing apparatus characterized in that, among the head group of each tile, encoded data of a higher hierarchy is stored in a tile specified by the important tile specifying means than in other tiles.   A computer program for causing a computer to execute the image processing method according to claim 1.   A storage medium storing the computer program of claim 12 in a state readable by a computer.

Images