JP7767783B2 - Image processing device and program - Google Patents

Description

The present invention relates to an image processing device and a program.

In the past, when selling condominiums and other properties, model rooms such as prefabricated buildings were constructed for potential buyers to view. Potential buyers would actually visit the model rooms to get an idea of what the rooms would look like.
In recent years, efforts have been made to provide potential buyers with a similar experience in their own homes by creating virtual environments similar to model rooms and allowing them to view the property using a PC (personal computer).

It is technically difficult to run something made up of 3DCG (3-Dimensional Computer Graphics) in a web browser. Therefore, in the above-mentioned efforts, in order to display a virtual space in a web browser, for example, the virtual space is displayed in the web browser as a 3D panoramic image (for example, Patent Document 1).

Japanese Patent Application Laid-Open No. 2018-147272

When it comes to panoramic images, the problem is determining which room and which position to use as the viewpoint for generating the panoramic image. With the technology described in Patent Document 1, the position for generating the panoramic image must be set manually, which is time-consuming.

Therefore, the present invention aims to provide an image processing device and program that can more easily generate panoramic images.

The present invention solves the above problems by the following means.
The first invention is an image processing device comprising an image acquisition means for acquiring a three-dimensional image of an object, a plan view acquisition means for acquiring a plan view showing the inside of the object from above, an area detection means for detecting an area from the plan view acquired by the plan view acquisition means, an area selection means for selecting a target area from the area detected by the area detection means, a fixed point setting means for setting a fixed point for the target area selected by the area selection means, and a panoramic image generation means for generating a panoramic image at the fixed point set by the fixed point setting means from the three-dimensional image of the object.
A second invention is the image processing device according to the first invention, wherein the region selection means selects the region whose size is equal to or greater than a threshold as the target region.
A third invention is the image processing device of the first or second invention, wherein the area selection means selects the target area based on information relating to the use of each area.
A fourth invention is an image processing device in which, in any of the first to third inventions, the fixed point setting means sets the fixed point based on a fixed point setting rule based on the area and aspect ratio of the region.
A fifth invention is an image processing device that is any of the first to fourth inventions, and is provided with an object image receiving means that receives an image related to the object, and the image acquisition means acquires a three-dimensional image of the object generated from the plan view when the image received by the object image receiving means is the plan view.
A sixth invention is an image processing device that is any of the image processing devices of the first to fourth inventions, and that includes an object image receiving means that receives an image related to the object, and a plan view generation means that, when the image received by the object image receiving means is a three-dimensional image of the object, generates the plan view from the three-dimensional image of the object.
The seventh invention is an image processing device that is any of the first to sixth inventions, and is equipped with a captured image generation means that generates a plurality of captured images of the three-dimensional image of the object, the three-dimensional image of the object being captured from positions moved circumferentially on the circumference of a horizontal concentric circle centered on the three-dimensional image of the object acquired by the image acquisition means, and a composite image generation means that generates a composite image by combining the plurality of captured images generated by the captured image generation means in a circumferential order.
An eighth aspect of the present invention is a program for causing a computer to function as any one of the image processing devices according to the first to seventh aspects of the present invention.

The present invention provides an image processing device and program that can more easily generate panoramic images.

1 is a schematic diagram of an overall image processing system according to an embodiment of the present invention and a functional block diagram of an image processing apparatus; 10 is a flowchart showing an image generation process of the image processing apparatus according to the present embodiment. 1A and 1B are diagrams illustrating examples of images received by the image processing apparatus according to the present embodiment. 10 is a flowchart showing an interior image generation process of the image processing device according to the present embodiment. 10A to 10C are diagrams illustrating a specific example of interior image generation processing in the image processing device according to the present embodiment. 10 is a diagram illustrating a fixed point rule storage unit and a setting example of fixed points of the image processing apparatus according to the present embodiment. FIG. 10A to 10C are diagrams illustrating a specific example of interior image generation processing in the image processing device according to the present embodiment. 10 is a flowchart showing an exterior image generation process of the image processing device according to the present embodiment. 10A to 10C are diagrams illustrating a specific example of exterior image generation processing in the image processing device according to the present embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, this is merely an example, and the technical scope of the present invention is not limited to this example.
(Embodiment)
<Image Processing System 100>
FIG. 1 is a schematic diagram of an image processing system 100 according to this embodiment and a functional block diagram of an image processing device 1. As shown in FIG.

The image processing system 100 shown in FIG. 1 is a system including an image processing device 1, an image input terminal 3, a user terminal 4, and a web server 5.
In the image processing system 100, the image input terminal 3 transmits image data for creating a panoramic image to the image processing device 1. The image processing device 1 then identifies the image data, generates the image data necessary to generate the panoramic image, automatically generates the panoramic image, and stores the generated image in a predetermined storage device of the web server 5. In this way, the user terminal 4 accesses the web server 5, causing the user terminal 4 to display the panoramic image.

The image processing device 1, the image input terminal 3, and the web server 5 are communicatively connected via a communication network N1. The user terminal 4 and the web server 5 are communicatively connected via a communication network N2. The communication networks N1 and N2 are, for example, internet lines, and may be wired or wireless. Although only one image input terminal 3 and one user terminal 4 are shown in FIG. 1, a plurality of them may of course be provided.
In the following, the image processing device 1 will be described as generating a panoramic image suitable for viewing, taking as an example a model room of a house (object).

<Image processing device 1>
The image processing device 1 is a device that performs image processing on image data received from the image input terminal 3 to generate a panoramic image of the interior and a panoramic image of the exterior.
The image processing device 1 is, for example, a server, but may also be a PC or the like.
Furthermore, there is no limit to the number of pieces of hardware that constitute the image processing device 1. As needed, the image processing device 1 may be configured with one or more pieces of hardware. Furthermore, the image processing device 1 may be, for example, a cloud.

The image processing device 1 includes a control unit 10 , a storage unit 20 , and a communication interface unit 29 .
The control unit 10 is a central processing unit (CPU) that controls the entire image processing device 1. The control unit 10 works in cooperation with the above-mentioned hardware to execute various functions by appropriately reading and executing an operating system (OS) and application programs stored in the storage unit 20.

The control unit 10 includes an image receiving unit 11 (object image receiving means), a 2D (2-Dimensions) image acquisition unit 12 (floor plan acquisition means, floor plan generation means), a 3D (3-Dimensions) image acquisition unit 13 (image acquisition means), an area detection processing unit 14 (area detection means, area selection means), a fixed point setting unit 15 (fixed point setting means), a panoramic image generation unit 16 (panoramic image generation means), a captured image generation unit 18 (captured image generation means), and a composite image generation unit 19 (composite image generation means).

The image receiving unit 11 receives image data of a house to be subjected to image processing from the image input terminal 3. The image data received from the image input terminal 3 may be, for example, image data representing a 2D image showing a floor plan such as a floor plan or a sketch, or may be 3DCG (three-dimensional space image) data representing a 3D image (three-dimensional image).
When the image data accepted by the image accepting unit 11 is 3DCG data, the 2D image acquiring unit 12 acquires a 2D image by generating image data of a 2D image from the 3DCG data. The 2D image acquiring unit 12 can generate a 2D image from the 3DCG data, for example, as an image of the interior of the 3D image viewed from above to below.

When the image data received by the image receiving unit 11 is image data of a 2D image, the 3D image obtaining unit 13 obtains a 3D image by generating 3DCG data from the image data of the 2D image.
Here, 3DCG data can be generated from image data of a 2D image using, for example, known software. For example, 3DCG data can be created by inputting image data of a 2D image as input data into known software for generating simple 3DCG (three-dimensional spatial images) (hereinafter, this software will also be referred to as "simple CG generation software").

The region detection processing unit 14 detects a region from the 2D image. More specifically, the region detection processing unit 14 extracts silhouette lines from the 2D image and detects a region based on the extracted silhouette lines. Note that the detection of a region by the region detection processing unit 14 is not limited to the above, and various known methods can be used for detection. The shape of the region detected by the region detection processing unit 14 may be, for example, a rectangle, a circle, a polygon, or any other shape.
The area detection processing unit 14 then selects a target area (target area) for creating a panoramic image from the detected area. Here, the target area refers to, for example, an area whose size is equal to or larger than a threshold value.

The area detection processing unit 14 may also select the target area based on, for example, information related to the area's purpose. The area detection processing unit 14 may obtain information related to the area's purpose by, for example, object recognition processing. For example, if the 2D image includes a design indicating a toilet, the area detection processing unit 14 may obtain "toilet" as information related to the area's purpose. Furthermore, if information related to the area's purpose is previously stored in the image data of the 2D image, the area detection processing unit 14 may obtain information related to the area's purpose from that data.

The fixed point setting unit 15 sets fixed points in the target area selected by the area detection processing unit 14. Here, the fixed point refers to the center position for generating a panoramic image. The fixed point setting unit 15 determines the number and positions of fixed points to be set in the target area selected by the area detection processing unit 14, for example, based on fixed point setting rules stored in the fixed point rule storage unit 22, and sets the determined fixed points in the target area. The fixed point setting rules define the number of fixed points and the installation positions of the fixed points based on the area and aspect ratio of the target area. An example of the fixed point setting rule regarding the number of fixed points is that the number of fixed points is one for every specific area (e.g., 10 ^m2 ). Furthermore, an example of the fixed point setting rule regarding the positions of the fixed points is that the fixed points are arranged in a direction with a higher aspect ratio of the target area.

The panoramic image generation unit 16 generates a panoramic image of the fixed point set by the fixed point setting unit 15 from the 3DCG data. The panoramic image generated by the panoramic image generation unit 16 is an image of a 360-degree indoor scene centered on the fixed point within the target area, and is an interior image.

The captured image generating unit 18 generates a plurality of captured images by capturing 3D images from a plurality of positions moved in the circumferential direction on the circumference of horizontal concentric circles centered on the 3D image.
The composite image generating unit 19 generates a composite image by sequentially combining the multiple captured images generated by the captured image generating unit 18 in the circumferential direction. The generated composite image is an image of the exterior of the house as seen from 360 degrees.

The storage unit 20 is a storage area such as a hard disk or semiconductor memory element for storing programs, data, etc. required for the control unit 10 to execute various processes.
The storage unit 20 includes a program storage unit 21 and a fixed point rule storage unit 22 .
The program storage unit 21 is a storage area for storing various programs. The program storage unit 21 stores an image processing program 21 a. The image processing program 21 a is a program for performing various functions executed by the control unit 10 of the image processing device 1.
Furthermore, the program storage unit 21 may have simple CG generation software installed.

The fixed point rule storage unit 22 is a storage area for storing fixed point setting rules, which are rules regarding the number and positions of fixed points to be set in a target area.
The communication interface unit 29 is an interface for communicating with the image input terminal 3 and the Web server 5, etc., via the communication network N1.
Here, a computer refers to an information processing device that includes a control unit, a storage device, etc., and the image processing device 1 is an information processing device that includes a control unit, a storage device, etc., and is included in the concept of a computer.

<Image input terminal 3>
The image input terminal 3 is a terminal used by, for example, a person in charge of a home builder who wishes to create a panoramic image, and is, for example, a stationary terminal such as a PC as shown in Fig. 1. The image input terminal 3 may also be a portable terminal that also has the functions of a computer, such as a tablet.
Although not shown, the image input terminal 3 includes at least a control unit, a storage unit, an input unit, an output unit, and a communication interface unit.

<User Terminal 4>
The user terminal 4 is a terminal owned by a prospective purchaser who wishes to purchase a home, and is, for example, a mobile terminal such as a tablet as shown in Fig. 1. The user terminal 4 may also be various other terminals such as a smartphone or a notebook PC.
Although not shown, the user terminal 4 includes at least a control unit, a storage unit, a touch panel display (or an input unit and an output unit), and a communication interface unit.
A web browser is also installed on the user terminal 4. The user terminal 4 can then launch the web browser and communicate with the web server 5 using the web, thereby obtaining web pages including image data from the web server 5.

<Web Server 5>
The web server 5 is a server that stores web pages including image data generated by the image processing device 1. Although not shown, the web server 5 includes at least a control unit, a storage unit, and a communication interface unit.

<Processing of image processing device 1>
Next, the processing performed by the image processing device 1 will be described with reference to a flowchart.
FIG. 2 is a flowchart showing the image generation process of the image processing device 1 according to this embodiment.
FIG. 3 is a diagram showing an example of an image accepted by the image processing apparatus 1 according to this embodiment.
FIG. 4 is a flowchart showing the interior image generation process of the image processing device 1 according to this embodiment.
FIG. 5 is a diagram showing a specific example of the interior image generation process of the image processing device 1 according to this embodiment.
FIG. 6 is a diagram showing an example of the fixed point rule storage unit 22 and the setting of fixed points in the image processing apparatus 1 according to this embodiment.
FIG. 7 is a diagram showing a specific example of the interior image generation process of the image processing device 1 according to this embodiment.
FIG. 8 is a flowchart showing the exterior image generation process of the image processing device 1 according to this embodiment.
FIG. 9 is a diagram showing a specific example of the exterior image generation process of the image processing device 1 according to this embodiment.

2 (hereinafter, "step S" will be simply referred to as "S") 11, the control unit 10 (image receiving unit 11) of the image processing device 1 receives image data that is the target for generating a panoramic image from the image input terminal 3. The control unit 10 receives, as image data, image data of a 2D image or 3DCG data that is image data of a 3D image.
Fig. 3A is a 2D image 50 showing a floor plan of a house, which is an example of a 2D image, and Fig. 3B is a 3D image 55 showing an exterior view of the house in 3DCG data, which is an example of a 3D image.

In S12, the control unit 10 determines whether the image data received from the image input terminal 3 is image data for a 2D image. If it is image data for a 2D image (S12: YES), the control unit 10 proceeds to S13. On the other hand, if it is not image data for a 2D image (S12: NO), the control unit 10 proceeds to S14.

In S13, the control unit 10 (3D image acquisition unit 13) generates 3DCG data from the image data of the 2D image. Here, the control unit 10 may generate the 3DCG data from the image data of the 2D image using known simple CG generation software. Alternatively, the control unit 10 may transmit the image data of the 2D image to another device (not shown), generate 3DCG data from the image data of the 2D image using known simple CG generation software in the other device, and acquire the 3DCG data from the other device. Thereafter, the control unit 10 proceeds to S15.
On the other hand, in S14, the control unit 10 (2D image acquisition unit 12) generates image data of a 2D image from the 3DCG data.
In S15, the control unit 10 performs interior image processing.

Here, the interior image processing will be described with reference to FIG.
In S21 of FIG. 4, the control unit 10 (area detection processing unit 14) detects an area from the 2D image.
First, as shown in Fig. 5A, the control unit 10 extracts silhouette lines 51 from a 2D image 50. In Fig. 5A, some of the silhouette lines 51 are indicated by symbols.
Next, the control unit 10 detects regions based on the extracted silhouette lines, as shown in the 2D image 60 of Fig. 5(B) In Fig. 5(B) , rectangular regions 61 to 66, which are part of the detected regions, are indicated by symbols.

In S22 of FIG. 4, the control unit 10 (area detection processing unit 14) selects a target area from the detected area.
The control unit 10 selects, from the detected regions, regions whose size is equal to or larger than a threshold value as target areas. Fig. 5(C) shows a 2D image 70 after the target areas have been selected, which includes target areas 71 to 75. The target areas 71 to 75 correspond to the rectangular regions 61 to 65 in Fig. 5(B).

In S23 of FIG. 4, the control unit 10 ( fixed point setting unit 15 ) sets a fixed point in the selected target area based on the fixed point setting rule in the fixed point rule storage unit 22.
The fixed point setting rules include rules regarding the number of fixed points and rules regarding the setting locations. The rules regarding the number of fixed points are based on the size (area) of the target area, and the larger the target area, the more fixed points there are.

Next, the rules regarding the installation location will be described with reference to FIG.
FIG. 6A shows an example of fixed point setting rules related to setting positions stored in the fixed point rule storage unit 22.
Figure 6(A) shows the rules for setting the fixed points depending on the number of fixed points. The rules shown in Figure 6(A) are an example of rules for when the rectangular area is somewhat long vertically. When the number of fixed points is 1, the fixed point is set in the center of the rectangular area. When the number of fixed points is 2, the fixed points are set in a row at regular intervals along the long side of the rectangular area. When the number of fixed points is 3, the fixed points are set so that when connected by lines, they form a triangle.
In FIG. 6B, when the target area 31 is horizontally long and there are two fixed points, the two-fixed-point rule 32 is applied to the target area 31, and fixed points are set as in the area after application 33.

Figure 7(A) shows a 2D image 80 after fixed points have been set. Target areas 81 to 85 in the 2D image 80 correspond to target areas 71 to 75 in Figure 5(C). Target areas 81 to 85 include fixed point marks 81a to 85a and 82b, respectively.

4, the control unit 10 (panoramic image generating unit 16) generates a panoramic image at each fixed point from the 3DCG data representing the 3D image. After that, the control unit 10 proceeds to S16 in FIG.
In S16 of FIG. 2, the control unit 10 performs exterior image processing.

Here, the exterior image processing will be described with reference to FIG.
In S31 of FIG. 8, the control unit 10 (captured image generating unit 18) generates a captured image by capturing an object in a 3D image from a horizontal position.
In S32, the control unit 10 (captured image generation unit 18) generates a plurality of captured images by capturing 3D images from a plurality of positions moved circumferentially on the circumference of a horizontal concentric circle centered on the 3D image.
In S33, the control unit 10 (composite image generating unit 19) generates a composite image by combining the generated images in order in the circumferential direction. After that, the control unit 10 proceeds to S17 in FIG.

Here, the exterior image processing will be specifically described with reference to FIG.
First, the control unit 10 prepares a 3D image 90 for imaging.
The control unit 10 (captured image generation unit 18) then generates captured images 91, 92, 93, ... obtained by capturing the 3D image 90 from cameras C1, C2, C3, ... at multiple positions moved in the circumferential direction on the circumference of horizontal concentric circles centered on the 3D image 90. Note that the cameras C1, C2, C3, ... are virtual cameras used to determine the angles of the images to be acquired as captured images.
Then, the control unit 10 (composite image generating unit 19) generates a composite image by combining the photographed images 91, 92, 93, . . . in this order.
2, the control unit 10 stores the generated image in a predetermined storage area of the web server 5. Thereafter, the control unit 10 ends this process.

FIG. 7B shows an example of an output of a panoramic image 41 on the user terminal 4 .
When the user terminal connects to the web server 5 and performs an operation to output the corresponding web page, the control unit of the web server 5 transmits data of the panoramic image 41 to the user terminal 4. The panoramic image 41 includes a plan view area 42. The plan view area includes a display area 43 p, which is the target area in which the panoramic image 41 is displayed, so that the display area 43 can be seen.
Here, the control unit 10 may set the most viewed direction in advance so that the most viewed direction is displayed first when a target area for which a panoramic image is desired to be displayed is selected on the user terminal 4. In this case, the control unit 10 can determine the most viewed direction by storing a usage log based on operations from the user terminal 4.

As described above, the image processing device 1 of this embodiment has the following advantages.
(1) From the area included in the 2D image of the house that is the subject of image processing, a target area for generating a panoramic image is selected and a fixed point is set. In addition, a panoramic image at the set fixed point is generated using 3D CG data that is a 3D image of the image data of the 2D image.
Therefore, since the time-consuming task of setting fixed points when generating a panoramic image is automated, panoramic images can be generated more easily.

(2) The target area for generating a panoramic image is selected based on the size of the area, so for example, in a house, a panoramic image of a relatively large room such as the living room can be generated. This allows a panoramic image of an area that meets the user's needs to be generated.
(3) The target area for generating a panoramic image is selected based on information related to the use of the area. For example, in a house, even a small area such as a bathroom or toilet can be used to generate a panoramic image as needed.

(4) Fixed points are set based on fixed point setting rules that are based on the area and aspect ratio of the region. For example, if the area is large, multiple fixed points can be set. Furthermore, the position at which the fixed points are set is determined based on the shape of the region. Therefore, it is possible to set fixed points that are optimal for the size and shape of the region.

(5) If the image data is of a 2D image, image data of a 3D image is acquired, and if the image data is 3DCG data, a 2D image is generated. Thus, in response to receiving image data of either a 2D image or a 3D image of an object, a panoramic image of the interior can be generated that allows the interior of the object to be viewed by rotating 360 degrees.

(6) 3D images of multiple objects are captured from positions moving circumferentially around a horizontal concentric circle centered on the 3D image of the object, generating 3D images of the object, and then combining the multiple captured images in the circumferential direction to generate a composite image. This allows for the generation of a panoramic exterior image that allows the outside of the object to be viewed from the periphery.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the embodiments are merely a list of the most favorable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments. The above-described embodiments and the modified forms described below can also be used in appropriate combinations, but detailed explanations will be omitted.

(Modified form)
(1) In the present embodiment, the image processing device and the web server are different devices, but the present invention is not limited to this. If the information processing device has the functionality of a web server, a single device can perform operations from image generation to image viewing on a user terminal.
In addition, in the present embodiment, the image input terminal and the image processing device are described as separate devices, but this is not limiting. For example, the image processing device may be a standalone image processing device that includes an input unit, an output unit, etc.
Furthermore, in the present embodiment, a web browser is installed on the user terminal and the user connects to a web server via the web browser. However, this is not limiting. For example, an application that can be used offline may be installed on the user terminal, and a panoramic image may be displayed when the application is launched. In this case, the web server may be a server that downloads panoramic images to the user terminal.

(2) In the present embodiment, the fixed point setting rules are stored in the fixed point rule storage unit. However, the present invention is not limited to this. The fixed point setting rules may be stored as logic in an image processing program, for example.
(3) In the present embodiment, the image data of either a 3D image or a 2D image is received and processed, but this is not limiting. If image data of both a 3D image and a 2D image is available, both sets of image data may be received and processed. In this case, processing to acquire the other set of image data is not required.

(4) In this embodiment, an image of a house has been described as an example, but the present invention is not limited to this. For example, an image of a car may also be used. In the case of a car, each area including the seat can be extracted, and fixed points can be set based on the size and aspect ratio of the area.

REFERENCE SIGNS LIST 1 Image processing device 3 Image input terminal 4 User terminal 5 Web server 10 Control unit 11 Image reception unit 12 2D image acquisition unit 13 3D image acquisition unit 14 Area detection processing unit 15 Fixed point setting unit 16 Panoramic image generation unit 18 Captured image generation unit 19 Synthesized image generation unit 20 Storage unit 21a Image processing program 22 Fixed point rule storage unit 50, 60, 70, 80 2D image 55, 90 3D image 100 Image processing system

Claims (7)

image acquisition means for acquiring a three-dimensional image of an object;
a plan view acquisition means for acquiring a plan view showing the inside of the object from above;
an area detection means for detecting an area from the plan view acquired by the plan view acquisition means;
an area selection means for selecting a target area from the areas detected by the area detection means;
a fixed point setting means for setting a fixed point for the target area selected by the area selection means based on a fixed point setting rule based on the area and aspect ratio of the area ;
a panoramic image generating means for generating a panoramic image at the fixed point set by the fixed point setting means from a three-dimensional image of the object;
An image processing device comprising: 2. The image processing device according to claim 1,
The area selection means selects the area whose size is equal to or larger than a threshold as the target area. 3. The image processing device according to claim 1,
The area selection means selects the target area based on information relating to the use of each area. 4. The image processing device according to claim 1,
an object image receiving means for receiving an image of the object;
The image acquisition means, when the image accepted by the object image acceptance means is the plan view, acquires a three-dimensional image of the object generated from the plan view. 4. The image processing device according to claim 1,
an object image receiving means for receiving an image relating to the object;
a plan view generating means for generating the plan view from the three-dimensional image of the object when the image received by the object image receiving means is a three-dimensional image of the object;
An image processing device comprising: 6. The image processing device according to claim 1,
a photographed image generating means for generating a plurality of photographed images of the three-dimensional image of the object, the photographed images being obtained by photographing the three-dimensional image of the object from positions moved in a circumferential direction on the circumference of a horizontal concentric circle centered on the three-dimensional image of the object acquired by the image acquiring means;
a composite image generating means for generating a composite image by sequentially combining the plurality of photographed images generated by the photographed image generating means in a circumferential direction;
An image processing device comprising: A program for causing a computer to function as the image processing device according to any one of claims 1 to 6 .