JP6950548B2 - Transmission program, method and device, and image synthesis program, method and device - Google Patents

Description

The present invention relates to transmission programs, methods and devices, and image synthesis programs, methods and devices.

In recent years, an increasing number of companies are considering and introducing telework represented by working from home. When telework is introduced, information sharing is important because multiple people will work in different places.

As an information sharing technology, a technology for sharing digital data by using, for example, an electronic conference system is known. In addition, when sharing analog data such as paper and whiteboard, it is necessary to share images taken of paper and whiteboard.

JP-A-2010-34739 Japanese Unexamined Patent Publication No. 2006-162692 Japanese Unexamined Patent Publication No. 2008-117235

When images are taken and shared, for example, when one image cannot cover the entire whiteboard, it is conceivable to combine a plurality of images.

As a technique for synthesizing a plurality of images, a technique is known in which a plurality of images are analyzed, local features are detected, and each image is combined based on the degree of overlap of the images. , There is a risk of imposing a heavy load on the processing device that performs image composition.

In one aspect, it is an object of the present invention to provide a transmission program, method and apparatus capable of reducing a processing load during image composition, and an image composition program, method and apparatus.

In one embodiment, the transmission program refers to a storage unit that receives designation of a part of the subject included in the first image captured by the imaging device and stores the position of the part of the subject in association with the part. When the plane including the position associated with the designated portion is specified and the second image newly captured by the imaging device is acquired, the specified plane and the acquired second image are obtained. Based on the orientation of the imaging device at the time of imaging, information indicating a range corresponding to the acquired second image in the plane is generated, and the acquired second image and the generated information are used. Is a transmission program for causing a computer to execute a process of transmitting an image to a specific destination.

The processing load at the time of image composition can be reduced.

It is a figure which shows roughly the structure of the information sharing system which concerns on one Embodiment. It is a figure which shows the hardware configuration of a server. FIG. 3A is a diagram showing the configuration of the HMD, and FIG. 3B is a diagram showing the hardware configuration of the control device of FIG. 3A. It is a functional block diagram of a server, a main side HMD and a remote side HMD. It is a flowchart which shows the processing of the main side HMD. It is a figure (the 1) for demonstrating the process of FIG. It is a figure (the 2) for demonstrating the process of FIG. 8 (a) and 8 (b) are diagrams (No. 3) for explaining the process of FIG. It is a figure (the 4) for demonstrating the process of FIG. FIG. 5 is a diagram (No. 5) for explaining the process of FIG. FIG. 6 is a diagram (No. 6) for explaining the process of FIG. FIG. 7 is a diagram (No. 7) for explaining the process of FIG. It is a figure which shows the two-dimensional array P, H. It is a figure which shows the display example in the remote side HMD. FIG. 8 is a diagram (No. 8) for explaining the process of FIG. It is a functional block diagram of the main side HMD and the remote side HMD which concerns on a modification. It is a flowchart which shows the process of the main side HMD which concerns on a modification.

Hereinafter, one embodiment of the information sharing system will be described in detail with reference to FIGS. 1 to 15. FIG. 1 schematically shows the configuration of the information sharing system 100 according to the embodiment.

The information sharing system 100 is a system used when, for example, a user working in an office and a user working at home or the like by telework hold a conference by voice call. For example, in the information sharing system 100, information sharing between users is realized by a user working in an office transmitting information described on a whiteboard or the like to a user working at home or the like.

As shown in FIG. 1, the information sharing system 100 includes a server 10 and a plurality of head-mounted displays (HMDs) 70. The server 10 and the HMD 70 are connected to a network 80 such as the Internet.

(Server 10)
The server 10 is a device that executes image composition processing and the like when sharing information described on a whiteboard or the like in a plurality of HMD 70s. FIG. 2 shows the hardware configuration of the server 10. As shown in FIG. 2, the server 10 includes a CPU (Central Processing Unit) 90, a ROM (Read Only Memory) 92, a RAM (Random Access Memory) 94, a storage unit (here, an HDD (Hard Disk Drive), and an SSD (Solid). State Drive), flash memory, etc.) 96, network interface 97, portable storage medium drive 99, and the like. Each component of the server 10 is connected to the bus 98. In the server 10, the functions of each part shown in FIG. 4 are realized by the CPU 90 executing the program stored in the ROM 92 or the HDD 96 or the program read from the portable storage medium 91 by the portable storage medium drive 99. NS. The functions of each part in FIG. 4 may be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The details of each part of FIG. 4 will be described later.

(HMD70)
The HMD 70 is a device that can be worn by the user on the head. As the HMD 70, for example, a monocular transmissive HMD can be used. As the HMD 70, for example, various HMDs for both eyes or immersive type may be used.

FIG. 3A shows the hardware configuration of the HMD 70. As shown in FIG. 3A, the HMD 70 includes an acceleration sensor 12, a gyro sensor 14, a geomagnetic sensor 15, a camera 16, a distance image camera 17, a microphone 18, an input device 20, and a display device 22. A voice output device 24, a communication device 26, and a control device 30 are provided.

The acceleration sensor 12 is a sensor that detects the acceleration of the HMD 70. As the acceleration sensor 12, for example, a three-axis acceleration sensor such as a piezoresistive type or a capacitance type can be used. The measurement result by the acceleration sensor 12 is transmitted to the control device 30.

The gyro sensor 14 is a sensor that detects the angle (posture), angular velocity, or angular acceleration of the HMD 70. As the gyro sensor 14, for example, a vibration type gyro sensor can be used. The measurement result by the gyro sensor 14 is transmitted to the control device 30.

The geomagnetic sensor 15 is, for example, a three-axis geomagnetic sensor, and by detecting the geomagnetism, it measures the direction in which the HMD 70 is facing. The measurement result by the geomagnetic sensor 15 is transmitted to the control device 30.

The HMD 70 may include an acceleration sensor 12, a gyro sensor 14, a geomagnetic sensor 15, a directional sensor, a GPS (Global Positioning System), and the like.

The camera 16 is provided near the user's eyes and captures (imaging) the direction in which the user wearing the HMD 70 is facing. The image taken by the camera 16 is transmitted to the control device 30. In the present embodiment, the angle of view of the camera 16 is fixed at a predetermined angle of view.

The distance image camera 17 uses, for example, a TOF (Time of Flight) distance measuring technique to generate a distance image showing the distance to a three-dimensional object. For example, when the user wearing the HMD 70 is in a room as shown in FIG. 6 (such as a conference room in a company), 3D mapping data as shown in FIG. 7 is generated as a distance image. In this embodiment, it is assumed that the range captured by the distance image camera 17 and the range captured by the camera 16 are adjusted to be the same.

The microphone 18 collects the user's voice and surrounding sounds. The input device 20 is a touch pad, a button, or the like, and includes a power button, a home button, a volume control button, and the like.

The display device 22 is a display device for displaying various types of information. The display device 22 is, for example, a device that projects information onto a half mirror. That is, the display device 22 is a transparent display device that allows the user to see through the external landscape together with the information. The display device 22 may be an immersive type, a video transmission type, a retinal projection type, or the like.

The audio output device 24 is a speaker or earphone, and outputs audio information under the instruction of the control device 30. The user activates the call application on the HMD70 and communicates with other terminals (including the HMD70) to make a call (conference call, etc.) with the user of the other terminal using the microphone 18 and the voice output device 24. It can be carried out.

The communication device 26 exchanges information with the server 10 and other HMD 70s via the network 80.

The control device 30 comprehensively controls each part of the HMD 70. In particular, in the present embodiment, the control device 30 is for sharing an image of a predetermined shared area (for example, an area where a whiteboard, paper, etc.) exists among a plurality of users when a telephone conference or the like is held. It has a function. FIG. 3B shows the hardware configuration of the control device 30. As shown in FIG. 3B, the control device 30 includes a CPU 190, a ROM 192, a RAM 194, a storage unit (here, SSD, flash memory, etc.) 196, an input / output interface 197, a portable storage medium drive 199, and the like. ing. Each component of the control device 30 is connected to the bus 198. Each device other than the control device 30 shown in FIG. 3A is connected to the input / output interface 197. In the control device 30, the CPU 190 executes a program (including a transmission program) stored in the ROM 192 or the HDD 196, or a program read from the portable storage medium 191 by the portable storage medium drive 199, so that FIG. 4 shows. The functions of each part shown are realized. The functions of each part in FIG. 4 may be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). Here, FIG. 4 shows a functional block diagram of the main HMD70a and the remote HMD70b. In the HMD 70, the CPU 190 of the control device 30 executes a program to function as the main HMD 70a or as the remote HMD 70b. Whether the HMD70 functions as the main HMD70a or the remote HMD70b depends on the user's setting. The main HMD70a has a function as a transmission device for transmitting an image taken and information about the photographed image to the server 10.

(Function of main side HMD70a)
When the CPU 190 executes a program, the main HMD70a has an origin acquisition unit 31, a 3D mapping data acquisition unit 32, a reception unit and a shared area setting unit 33 as a specific unit, a line-of-sight vector acquisition unit 34, and a blur as a determination unit. It functions as a detection unit 35, a shooting necessity determination unit 36 as a determination unit, an image acquisition unit 37, a photographed range data acquisition unit 38, a positional relationship specifying unit 39 as a generation unit, and a transmission unit 40.

The origin acquisition unit 31 acquires the position of the main HMD70a (which is the position of the camera 16 in the present embodiment) as the origin at the timing when the power of the main HMD70a is turned on.

The 3D mapping data acquisition unit 32 acquires 3D mapping data (see FIG. 7) as the first image generated by the distance image camera 17.

The shared area setting unit 33 sets an area (for example, a drawing range of a whiteboard) specified by the user among the subjects included in the 3D mapping data as a shared area. The shared area setting unit 33 obtains information on the shared area (information on four vertices of the rectangle when the shared area is a rectangular area) based on the origin and 3D mapping data acquired by the origin acquisition unit 31. It is specified and stored in the shared area DB 29 as a storage unit. Further, the shared area setting unit 33 specifies a plane including the shared area, and stores the information of the specified plane in the shared area DB 29.

The line-of-sight vector acquisition unit 34 acquires the user's line-of-sight vector based on the measurement results of the origin and the geomagnetic sensor 15 acquired by the origin acquisition unit 31. In the present embodiment, the user's line-of-sight vector is a vector indicating the direction in which the HMD 70 is facing (the direction in which the camera 16 shoots), and is a vector connecting the origin and the center of the angle of view of the camera 16. ..

The blur detection unit 35 acquires the line-of-sight vector acquired by the line-of-sight vector acquisition unit 34 before and after shooting by the camera 16, and detects the blur of the user's head (main side HMD70a) at the time of shooting based on the acquired line-of-sight vector. .. The blur detection unit 35 outputs the detected blur information to the positional relationship specifying unit 39. The blur detection unit 35 refers to the information stored in the shared area DB 29 when detecting the blur.

The shooting necessity determination unit 36 includes the direction in which the camera 16 of the HMD 70 is facing (line-of-sight vector), information on the shared area stored in the shared area DB 29, and the shooting range acquired by the shooting range data acquisition unit 38. Based on the data, it is determined whether or not to perform shooting using the camera 16. The photographed range data is data indicating a range (unphotographed area) that has not been photographed so far in the shared area. The details of the photographed range data and the details of the determination method of the imaging necessity determination unit 36 will be described later.

The image acquisition unit 37 acquires an image (second image) taken by the camera 16 and outputs the image (second image) to the transmission unit 40.

The photographed range data acquisition unit 38 acquires the photographed range data transmitted from the server 10 and transmits the photographed range data to the photographing necessity determination unit 36.

The positional relationship specifying unit 39 specifies the positional relationship between the shared area and the range (shooting range) photographed by the camera 16 on the plane including the shared area. The information specified by the positional relationship specifying unit 39 is notified to the transmitting unit 40. The positional relationship specifying unit 39 specifies the positional relationship when the blur detected by the blur detecting unit 35 is within the permissible range.

The transmission unit 40 transmits the image acquired by the image acquisition unit 37 and the information on the positional relationship between the shared area and the shooting range specified by the positional relationship specifying unit 39 to the server 10.

(Function of remote side HMD70b)
The remote-side HMD70b functions as a composite image receiving unit 41 and a composite image display unit 42 when the CPU 190 executes a program.

The composite image receiving unit 41 receives the image of the shared area synthesized by the server 10 and transmits it to the composite image display unit 42.

The composite image display unit 42 receives the composite image from the composite image reception unit 41 and displays it on the display device 22 of the remote HMD 70b. In this case, the window 102 for displaying the composite image of the shared area as shown in FIG. 14 is displayed at a predetermined position in the field of view of the user who wears the remote side HMD70b.

(Function of server 10)
The server 10 functions as a receiving unit 51, an image synthesizing unit 54, a captured range data creating unit 52, a captured range data transmitting unit 53, and a composite image transmitting unit 55 when the CPU 90 executes a program.

The receiving unit 51 receives the image transmitted from the main HMD 70a and the information on the positional relationship between the shared area and the shooting range, and transmits the information to the image synthesizing unit 54.

The image synthesizing unit 54 executes image synthesizing based on the information on the positional relationship between the shared area and the photographing range received from the main side HMD 70a by the receiving unit 51 and the photographed range data.

The captured range data creation unit 52 reflects the range in which the images are combined by the image synthesizing unit 54 in the captured range data. The captured range data transmission unit 53 transmits the captured range data to the image synthesizing unit 54 and the captured range data acquisition unit 38 of the main HMD 70a.

The composite image transmission unit 55 appropriately transmits the image synthesized by the image synthesis unit 54 to the remote HMD 70b.

(About the processing of HMD70a on the main side)
Next, the processing of the main HMD 70a will be described in detail with reference to other drawings according to the flowchart of FIG. As an example, the process of FIG. 5 is assumed to be a process started when the power of the main HMD70a is turned on.

First, in step S10, the origin acquisition unit 31 acquires the origin. In the present embodiment, the origin is assumed to be the position of a predetermined point on the main side HMD70a (the position of the camera 16 in the present embodiment).

Next, in step S12, the 3D mapping data acquisition unit 32 acquires 3D mapping data as shown in FIG. 7 from the distance image camera 17. The 3D mapping data is data based on the origin acquired in step S10.

Next, in step S14, the shared area setting unit 33 waits until there is a request from the user to specify the shared area. The user inputs a request for designating the shared area by performing a predetermined input (for example, an input using an input device 20 or a microphone 18). When there is a predetermined input from the user, the shared area setting unit 33 proceeds to step S16.

When the process proceeds to step S16, the shared area setting unit 33 executes the shared area setting process. Specifically, the following processing is executed. In the present embodiment, the user specifies a rectangular area as the shared area.

When the user specifies the shared area, the center of the field of view of the main HMD70a is sequentially aligned with the four vertices of the shared area. When a mark such as a marker is attached to the center of the visual field of the main HMD70a, the user may adjust the position of the mark to the position of each apex.

When the shared area setting unit 33 receives the designation of the shared area (designation of each vertex) by the user, the shared area setting unit 33 calculates the intersection of the line-of-sight vector when the user specifies each vertex and the 3D mapping data. Here, the line-of-sight vector is a vector (vector indicated by a broken line arrow in FIG. 8A) that passes through a predetermined point (position of the camera 16) P of the main HMD70a and extends in the direction in which the main HMD70a faces, and is a geomagnetic sensor. It is acquired by the line-of-sight vector acquisition unit 34 based on the measurement result of 15 or the like. FIG. 8A shows the four intersections A, B, C, and D obtained by this calculation. In FIG. 8A, the coordinates of the predetermined point P are (Xp, Yp, Zp), the coordinates of the intersection A are (Xa, Ya, Za), the coordinates of the intersection B are (Xb, Yb, Zb), and the intersection. The coordinates of C are (Xc, Yc, Zc), and the coordinates of the intersection D are (Xd, Yd, Zd).

Next, the shared area setting unit 33 calculates the center points E of the four points (A, B, C, D). In this case, the shared area setting unit 33 sets the average of the coordinates of the four points as the coordinates of the center point E. In FIG. 8A, the coordinates of the center point E are (Xe, Ye, Ze).

Next, the shared area setting unit 33 obtains a plane f including the center point E and spreads in the vicinity of the four points A, B, C, and D, and stores the information of the plane f in the shared area DB 29.

Next, the shared area setting unit 33 is centered on the center point E on the plane f, the width is the average length between the lengths between the line segments AD and the lengths between the line segments BC, and the height is the line segment AB. The rectangle r, which is the average length between the length between the lines and the length between the line segments CD, is obtained. In FIG. 8B, the rectangle r is indicated by a thick broken line.

Next, the shared area setting unit 33 sets the four vertices of the rectangle r as vertices A', B', C', and D'. In FIG. 8B, the coordinates of the vertex A'are (Xa', Ya', Za'), the coordinates of the vertex B'are (Xb', Yb', Zb'), and the coordinates of the vertex C'are (Xa', Yb', Zb'). Xc', Yc', Zc'), and the coordinates of the vertex D'are (Xd', Yd', Zd').

Then, the shared area setting unit 33 sets a rectangle r having four vertices A'to D'as the shared area r, and stores the coordinates of the vertices A'to D'in the shared area DB 29.

As a result, the setting process of the shared area in step S16 is completed. In the present embodiment, the fact that the shared area setting unit 33 obtains the plane f means that the plane including the position (each vertex) associated with the portion designated by the user is specified.

Next, when the process proceeds to step S18, the line-of-sight vector acquisition unit 34 acquires the line-of-sight vector. It can be said that the line-of-sight vector acquired here is the line-of-sight vector before image acquisition.

Next, in step S19, the shooting necessity determination unit 36 confirms the shooting necessity. Specifically, the shooting necessity determination unit 36 executes the following processing.

First, the shooting necessity determination unit 36 acquires the line-of-sight vector Pv (see FIG. 9) from the line-of-sight vector acquisition unit 34. Next, as shown in FIG. 9, the photographing necessity determination unit 36 obtains the position of the intersection Ph of the line-of-sight vector Pv and the plane f.

Next, the shooting necessity determination unit 36 confirms whether or not the intersection Ph satisfies both of the following conditions (1) and (2).
Condition (1): The intersection Ph is in the shared area r.
Condition (2): The intersection Ph is not in the area already photographed.
Then, the shooting necessity determination unit 36 determines that shooting is necessary when both the conditions (1) and (2) are satisfied.

When determining whether or not the condition (1) is satisfied, the shooting necessity determination unit 36 refers to the shared area DB 29 and acquires the information of the shared area r. The details of the condition (2) will be described later because the condition (2 is a condition that does not need to be determined at the time of the first shooting.

The conditions are not limited to the above (1) and (2). That is, the number of conditions may be increased or decreased.

Returning to FIG. 5, in the next step S20, the shooting necessity determination unit 36 determines whether or not shooting is necessary as a result of the processing in step S19. If the determination in step S20 is denied, the process returns to step S18, but if the determination is affirmed, the process proceeds to step S21.

When the process proceeds to step S21, the image acquisition unit 37 issues an instruction to the camera 16 to take an image, and acquires the taken image. In addition, the line-of-sight vector acquisition unit 34 acquires the line-of-sight vector at the timing when the image is taken.

Next, in step S22, the line-of-sight vector acquisition unit 34 acquires the line-of-sight vector. It can be said that the line-of-sight vector acquired here is the line-of-sight vector after the image is acquired.

Next, in step S26, the blur detection unit 35 calculates the degree of blur. In this case, the blur detection unit 35 first identifies the intersection (P1) of the line-of-sight vector before shooting and the plane f, as shown in FIG. The blur detection unit 35 acquires the information on the plane f from the shared area DB 29. Next, the blur detection unit 35 identifies the intersection (P2) of the line-of-sight vector after photographing and the plane f. Then, the blur detection unit 35 obtains a vector (V) connecting the intersection P1 and the intersection P2, and sets the magnitude (scalar amount) of the vector V as the degree of blur.

In step S28, the blur detection unit 35 determines whether or not the degree of blur is within the permissible range. Here, the blur detection unit 35 determines that an image with a large blur has been captured when the magnitude (scalar amount) of the vector V is not within a predetermined allowable range. If the degree of blurring is out of the permissible range, the captured data is not processed (not transmitted to the server 10). As a result, it is possible to prevent an image having a large blur from being used for compositing. If the determination in step S28 is affirmed, the process proceeds to step S30, and if it is denied, the process returns to step S18.

In step S30, the positional relationship specifying unit 39 specifies the positional relationship between the shared area r and the photographing range on the plane f, and the transmitting unit 40 transmits data.

In step S30, as shown in FIG. 11, the positional relationship specifying unit 39 identifies a quadrangular pyramid representing a captured range from the line-of-sight vector Pv and the angle of view of the camera 16. Specifically, a square weight whose directions of the vectors Av, Bv, Cv, and Dv extending from the camera 16 toward the four corners of the shooting range of the camera 16 are the directions of the side sides of the square weight (however, the position of the bottom surface is specified. Not).

Next, the positional relationship specifying unit 39 specifies the range included in the specified square pyramid in the plane f as the bottom surface of the square pyramid, and the bottom surface as the photographing range Cr.

Next, as shown in FIG. 12, the positional relationship specifying unit 39 sets the upper left point Ca of the imaging range Cr as the origin (coordinates (0,0)) and the coordinates of the lower right point Cc of the imaging range Cr as (1). , 1) Define the two-dimensional coordinate system. The two-dimensional coordinate system of FIG. 12 is a two-dimensional coordinate system arranged on the plane f. Of the vertices of the photographing range Cr, the coordinates of the points Cb and Cd other than the points Ca and Cc are (0,1) and (1,0), respectively.

Next, the positional relationship specifying unit 39 identifies four vertices (ra, rb, rc, rd) of the rectangular shared area r on the defined coordinate system. Then, the positional relationship specifying unit 39 outputs information on the specified four vertices (that is, information on the positional relationship between the shared area r and the photographing range Cr) to the transmitting unit 40. The transmission unit 40 transmits the captured image and the information on the positional relationship between the shared area r and the photographing range Cr to the server 10. It can be said that the information on the positional relationship between the shared area r and the shooting range Cr is information indicating the range shot in the shot image of the plane f.

(Processing of server 10)
Here, the processing of the server 10 that has received the captured image and the information of the four vertices (ra, rb, rc, rd) of the shared area r from the main HMD 70a will be described.

The server 10 performs a compositing process of images provided to the remote HMD70b and creates captured range data. First, as shown in FIG. 13, the server 10 creates a two-dimensional array (bitmap) P for storing the synthesized image, and also creates a two-dimensional array (bitmap) as management data having the same size as the two-dimensional array P. ) Create H. Here, the two-dimensional array H is photographed range data used for determining whether or not an image has been synthesized (photographed) for each element (bit) of the two-dimensional array P. The two-dimensional array H has a numerical value of 0 to 255 as an element, 0 represents unsynthesized, and 1 to 255 represents synthesized. Further, the numerical values of 1 to 255 indicate that the larger the value, the clearer the combined image. It is assumed that the initial value of each element of the two-dimensional array H is 0. Here, the sharpness can be calculated based on the size of the captured range in the captured image. Specifically, the smaller the captured range in the captured image (that is, the larger the area of the shared area r obtained from the coordinates in FIG. 12), the clearer the image, and the larger the value.

In the server 10, when the receiving unit 51 receives the captured image and the coordinates of the four vertices (ra, rb, rc, rd) of the shared area r, the image synthesizing unit 54 receives the captured image from the coordinates of the four vertices. The portion of the shared area r that overlaps with the shared area r (the portion of the shared area r that was captured in the captured image) is cut out. Since the captured image is not necessarily an image captured from the front of the shared area r, the image synthesizing unit 54 performs trapezoidal correction or the like on the received captured image to obtain an image captured from the front of the shared area r. After correction, it is assumed that the portion overlapping with the shared area r is cut out.

Next, the image synthesizing unit 54 compares the numerical value of the sharpness of each element (bit) of the overlapping portion of the captured image with the numerical value of the sharpness of the corresponding element (bit) of the two-dimensional array H, and compares the numerical value of the sharpness of the corresponding element (bit) of the captured image. For the element with a larger numerical value, the element of the captured image is transferred to the corresponding element of the two-dimensional array P.

Further, the captured range data creation unit 52 acquires the numerical value of the sharpness of the elements of the captured image transferred to the two-dimensional array P by the image synthesizing unit 54, and sets the numerical value of each element of the two-dimensional array H.

The composite image transmission unit 55 transmits the two-dimensional array P to the composite image reception unit 41 of the remote HMD 70b each time the image (two-dimensional array P) synthesized by the image composition unit 54 is updated. In the remote side HMD 70b, as shown in FIG. 14, the composite image display unit 42 displays the window 102 including the received composite image (two-dimensional array P) on the display device 22.

Further, the photographed range data transmission unit 53 transmits the two-dimensional array H to the photographed range data acquisition unit 38 of the main HMD 70a every time the two-dimensional array H (photographed range data) is updated.

Returning to FIG. 5, in the main side HMD70a, after step S30 is executed, the process proceeds to step S32, and the captured range data acquisition unit 38 is transmitted from the server 10 (photographed range data transmission unit 53). Receives the captured range data.

Next, in step S34, the positional relationship specifying unit 39 refers to the captured range data and determines whether or not the image composition is completed. That is, the positional relationship specifying unit 39 determines whether or not the number of all elements of the photographed range data (two-dimensional array H) is other than 0, and if the number of all elements is other than 0, step S34. The judgment is affirmed, and the process proceeds to step S36. On the other hand, when the number of at least one element of the two-dimensional array H is 0, the determination in step S34 is denied and the process returns to step S18. When returning to step S18, the processing / determination of steps S18 to S34 is repeatedly executed until the determination in step S34 is affirmed.

Here, in step S19, which is performed after receiving the photographed range data from the server 10 in step S32, the imaging necessity determination unit 36 determines the above-mentioned condition (2) "The intersection Ph is not in the already photographed area." Judge whether or not the above is satisfied. In this case, the photographing necessity determination unit 36 replaces the shared area r with the received photographed range data (two-dimensional array H), and as shown in FIG. 15, the intersection of the line-of-sight vector Pv and the plane f (the line-of-sight vector and the line-of-sight vector). (Intersection with shared area r) Ph is obtained. Then, the shooting necessity determination unit 36 acquires the value of the element of the two-dimensional array H corresponding to the intersection Ph, and if the acquired value is 0, the condition (2) “In the region where the intersection Ph has already been taken” It is judged that "No." is satisfied. By making such a judgment, unnecessary shooting can be reduced, so that shooting efficiency can be improved.

When the determination in step S34 is affirmed and the process proceeds to step S36, the positional relationship specifying unit 39 waits until a new image composition request is input. When a new image composition request is input by the user (either the user of the main HMD70a or the user of the remote HMD70b), the process proceeds to step S38, and the positional relationship specifying unit 39 uses the captured range data (two-dimensional). The sequence H) is discarded, and the process returns to step S18. At the stage where the process of step S38 is executed on the main side HMD70a, it is assumed that the captured range data (two-dimensional array H) is also discarded on the server 10. After that, the processes after step S18 described above are repeatedly executed.

The process of FIG. 5 ends when a termination request from the user is received or when the power of the main HMD70a or the remote HMD70b is turned off.

It should be noted that the remote-side HMD70b can draw a rectangle, a straight line, a curved line, or the like with respect to the composite image of the shared area displayed in the window 102 of FIG. For example, when the control device 30 selects a figure to be drawn using the icon in the window 102 and detects that the user has moved a finger on the window 102, the control device 30 detects that the user has moved the finger, and the figure (character or the like) corresponding to the movement of the finger may be used. Is drawn in the window 102. The control device 30 may transmit the information of the drawn figure to the main HMD 70a. In this case, the control device 30 of the main HMD 70a may display the graphic on the display device 22 by superimposing it on the whiteboard in the user's field of view based on the received information. As a result, the figure drawn by the user of the remote HMD 70b can be shared among the users.

As is clear from the above description, in the present embodiment, the distance image camera 17 that generates the 3D mapping data (first image) used when setting the shared area and the image to be transmitted to the server 10 (the first image). The function as a photographing device is realized including the camera 16 that captures the image of 2).

As described in detail above, according to the present embodiment, the shared area setting unit 33 accepts the designation of each vertex of the shared area r and stores the coordinates of each vertex of the shared area r in the shared area DB 29. , The plane f including each vertex is specified and stored in the shared area DB 29. Further, when the image acquisition unit 37 acquires the captured image, the positional relationship specifying unit 39 obtains information (shared area r) indicating the captured range of the plane f based on the plane f and the line-of-sight vector Pv at the time of shooting. Information on the positional relationship of the shooting range Cr) is generated. Then, the transmission unit 40 transmits the captured image and the generated information to the server 10. Therefore, the server 10 synthesizes the captured images based on the information on the positional relationship between the shared area r and the photographing range Cr (information indicating the photographing range in the plane f) to generate an image in the shared area r. Can be done. Thereby, for example, the processing load of the server 10 at the time of image composition can be reduced as compared with the case where the feature of the image is detected by image analysis, the overlapping condition of a plurality of images is specified and the images are combined. can.

Further, in the present embodiment, the blur detection unit 35 determines whether or not the movement (blur) of the camera 16 when the image acquired by the image acquisition unit 37 is taken is included in the allowable range. Then, the transmission unit 40 transmits the captured image and the information on the positional relationship between the shared area r and the photographing range Cr to the server 10 when it is included in the allowable range. As a result, the captured image with less blur and the information corresponding to the captured image can be transmitted to the server 10, so that the server 10 can generate a clear image as a composite image.

Further, in the present embodiment, the shooting necessity determination unit 36 determines the shooting necessity based on the photographed range data and the direction (line-of-sight vector) of the camera 16, so that the image is limited to the images required for compositing. It is possible to take pictures and reduce unnecessary shooting.

In the above embodiment, the case where one shared area r is set has been described, but the present invention is not limited to this, and a plurality of shared areas r may be set. In this case, by repeatedly executing the processes after step S18 in FIG. 5 for each shared area, a composite image of each shared area may be generated and displayed side by side on the display device 22 of the remote HMD 70b. ..

In the above embodiment, the case where the shared area is rectangular has been described, but the present invention is not limited to this. The common area may be a triangle, a pentagon, a hexagon, and so on.

In the above embodiment, there may be a plurality of remote-side HMD70bs. In this case, the server 10 may transmit the composite image of the shared area r to a plurality of remote side HMDs 70b.

In the above embodiment, after the image composition is completed (after the determination in step S34 in FIG. 5 is affirmed), the case where the image composition is restarted in response to the user's input in step S36 has been described. It is not limited to. For example, image composition may be performed at predetermined timing intervals. Specifically, a new image composition may be started when the image composition of one image is completed. Further, a new image composition may be started when a predetermined time has elapsed from the completion of the image composition of one image.

In the above embodiment, the case where the image composition is executed on the server 10 has been described, but the present invention is not limited to this. For example, the process executed on the server 10 in the above embodiment (image composition process or captured range data creation process) may be executed on the main HMD 70a. That is, as shown in FIG. 16, the main side HMD70a is an image as a compositing unit that performs image composition based on a captured image specified by the positional relationship specifying unit 39 and information on the positional relationship between the shared area r and the photographing range Cr. It may be an image compositing device having a function of a compositing unit 54'or a photographed range data creating unit 52' as a creating unit for creating the photographed range data. In this case, as shown in FIG. 17, the control device 30 may execute the processes of steps S30'and S32'instead of steps S30 and S32 of FIG. In step S30', the image synthesizing unit 54'of the control device 30 performs image synthesizing based on the information on the positional relationship between the shared area r and the photographing range Cr. Further, in step S32', the photographed range data creation unit 52'of the control device 30 creates the photographed range data (two-dimensional array H). In step S30', the transmission unit 40 transmits the image synthesized by the image synthesis unit 54'to the composite image reception unit 41 of the remote HMD 70b.

Further, the process executed by the server 10 in the above embodiment may be executed on the remote side HMD 70b. That is, the control device 30 of the remote side HMD 70b may have a function of an image synthesizing unit that performs image composition and a photographed range data creating unit that creates captured range data.

The remote side HMD70b may be a terminal such as a PC. In this case, by displaying the composite image of the shared area on the display device of the terminal, it is possible to share the information of the shared area between the users.

The above processing function can be realized by a computer. In that case, a program that describes the processing content of the function that the processing device should have is provided. By executing the program on a computer, the above processing function is realized on the computer. The program describing the processing content can be recorded on a computer-readable recording medium (however, the carrier wave is excluded).

When a program is distributed, it is sold in the form of a portable recording medium such as a DVD (Digital Versatile Disc) or a CD-ROM (Compact Disc Read Only Memory) on which the program is recorded. It is also possible to store the program in the storage device of the server computer and transfer the program from the server computer to another computer via the network.

The computer that executes the program stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in its own storage device. Then, the computer reads the program from its own storage device and executes the processing according to the program. The computer can also read the program directly from the portable recording medium and execute the processing according to the program. In addition, the computer can sequentially execute processing according to the received program each time the program is transferred from the server computer.

The embodiments described above are examples of preferred embodiments of the present invention. However, the present invention is not limited to this, and various modifications can be made without departing from the gist of the present invention.

Regarding the above description of the embodiment, the following additional notes will be further disclosed.
(Appendix 1) Accepting the designation of the part of the subject included in the first image captured by the imaging device,
With reference to a storage unit that stores the position of the part of the subject in association with the part, a plane including the position associated with the designated part is specified.
When the second image newly captured by the image pickup device was acquired, it was acquired out of the planes based on the specified plane and the orientation of the image pickup device at the time of capturing the acquired second image. Generate information indicating the range corresponding to the second image,
The acquired second image and the generated information are transmitted to a specific destination.
A sending program that lets a computer perform processing.
(Appendix 2) It is determined whether or not the movement of the imaging device at the time of capturing the acquired second image is included in the permissible range.
If the determination result is affirmative, the acquired second image and the generated information are transmitted.
The transmission program according to Appendix 1, wherein the computer further executes the process.
(Appendix 3) Based on the management data for managing the range of the subject imaged by the image pickup device and the orientation of the image pickup device, it is determined whether or not to perform the image pickup by the image pickup device.
The transmission program according to Appendix 1 or 2, wherein the computer further executes the process.
(Appendix 4) Accepting the designation of the part of the subject included in the first image captured by the imaging device,
With reference to a storage unit that stores the position of the part of the subject in association with the part, a plane including the position associated with the designated part is specified.
When the second image newly captured by the image pickup device was acquired, it was acquired out of the planes based on the specified plane and the orientation of the image pickup device at the time of capturing the acquired second image. Generate information indicating the range corresponding to the second image,
The acquired second image and the generated information are transmitted to a specific destination.
A transmission method characterized in that the processing is performed by a computer.
(Appendix 5) A reception unit that accepts the designation of the part of the subject included in the first image captured by the imaging device, and
With reference to a storage unit that stores the position of the portion of the subject in association with the portion, a specific portion that specifies a plane including the position associated with the designated portion, and a specific portion.
When the second image newly captured by the image pickup device was acquired, it was acquired out of the planes based on the specified plane and the orientation of the image pickup device at the time of capturing the acquired second image. A generation unit that generates information indicating a range corresponding to the second image, and a generation unit.
A transmission unit that transmits the acquired second image and the generated information to a specific destination.
A transmitter equipped with.
(Appendix 6) Further provided with a determination unit for determining whether or not the movement of the imaging device at the time of capturing the acquired second image is included in the permissible range.
The transmission device according to Appendix 5, wherein the transmission unit transmits the acquired second image and the generated information when the determination result of the determination unit is positive.
(Appendix 7) Further, a determination unit for determining whether or not to perform imaging by the imaging device is further provided based on the management data for managing the range of the subject imaged by the imaging device and the orientation of the imaging device. The transmitter according to Appendix 5 or 6.
(Appendix 8) Accepting the designation of the part of the subject included in the first image captured by the imaging device,
With reference to a storage unit that stores the position of the part of the subject in association with the part, a plane including the position associated with the designated part is specified.
When the second image newly captured by the image pickup device was acquired, it was acquired out of the planes based on the specified plane and the orientation of the image pickup device at the time of capturing the acquired second image. Generate information indicating the range corresponding to the second image,
When a plurality of the second images are acquired, the plurality of the second images are combined based on the information indicating the range corresponding to the plurality of the second images.
An image synthesis program that allows a computer to perform processing.
(Appendix 9) Create management data for managing the range of the subject imaged by the imaging device, and create management data.
Based on the management data and the orientation of the imaging device, it is determined whether or not to perform imaging by the imaging device.
The image synthesis program according to Appendix 8, wherein the processing is further executed by a computer.
(Appendix 10) Accepting the designation of the part of the subject included in the first image captured by the imaging device,
With reference to a storage unit that stores the position of the part of the subject in association with the part, a plane including the position associated with the designated part is specified.
When the second image newly captured by the image pickup device was acquired, it was acquired out of the planes based on the specified plane and the orientation of the image pickup device at the time of capturing the acquired second image. Generate information indicating the range corresponding to the second image,
When a plurality of the second images are acquired, the plurality of the second images are combined based on the information indicating the range corresponding to the plurality of the second images.
An image composition method characterized in that processing is performed by a computer.
(Appendix 11) A reception unit that accepts designation of a part of the subject included in the first image captured by the imaging device, and a reception unit.
With reference to a storage unit that stores the position of the portion of the subject in association with the portion, a specific portion that specifies a plane including the position associated with the designated portion, and a specific portion.
When the second image newly captured by the image pickup device was acquired, it was acquired out of the planes based on the specified plane and the orientation of the image pickup device at the time of capturing the acquired second image. A generation unit that generates information indicating a range corresponding to the second image, and a generation unit.
When a plurality of the second images are acquired, a compositing unit that synthesizes the plurality of the second images based on information indicating a range corresponding to the plurality of the second images, and a compositing unit.
An image synthesizer comprising.
(Appendix 12) A creation unit that creates management data that manages the range of the subject imaged by the imaging device, and a creation unit.
The image synthesizer according to Appendix 11, further comprising a determination unit for determining whether or not to perform imaging by the image pickup device based on the management data and the orientation of the image pickup device.

16 Camera (part of imaging device)
17 Distance image camera (part of imaging device)
29 Shared area DB (storage unit)
33 Shared area setting department (reception department, specific department)
35 Blur detection unit (judgment unit)
36 Shooting Necessity Judgment Department (Judgment Department)
39 Positional relationship identification part (generation part)
40 Transmitter 52'Captured range data creation unit (creation unit)
54'Image Synthesis Department (Composite Department)
70a Main side HMD (transmitter, image synthesizer)

Claims (9)

Accepts the designation of the part of the subject included in the first image captured by the image pickup device,
With reference to a storage unit that stores the position of the part of the subject in association with the part, a plane including the position associated with the designated part is specified.
When the second image newly captured by the image pickup device was acquired, it was acquired out of the planes based on the specified plane and the orientation of the image pickup device at the time of capturing the acquired second image. Generate information indicating the range corresponding to the second image,
The acquired second image and the generated information are transmitted to a specific destination.
A sending program that lets a computer perform processing. It is determined whether or not the movement of the imaging device at the time of capturing the acquired second image is included in the permissible range.
If the determination result is affirmative, the acquired second image and the generated information are transmitted.
The transmission program according to claim 1, wherein the computer further executes the process. Based on the management data for managing the range of the subject captured by the imaging device and the orientation of the imaging device, it is determined whether or not to perform imaging by the imaging device.
The transmission program according to claim 1 or 2, wherein the computer further executes the process. Accepts the designation of the part of the subject included in the first image captured by the image pickup device,
With reference to a storage unit that stores the position of the part of the subject in association with the part, a plane including the position associated with the designated part is specified.
When the second image newly captured by the image pickup device was acquired, it was acquired out of the planes based on the specified plane and the orientation of the image pickup device at the time of capturing the acquired second image. Generate information indicating the range corresponding to the second image,
The acquired second image and the generated information are transmitted to a specific destination.
A transmission method characterized in that the processing is performed by a computer. A reception unit that accepts the designation of the part of the subject included in the first image captured by the image pickup device, and
With reference to a storage unit that stores the position of the portion of the subject in association with the portion, a specific portion that specifies a plane including the position associated with the designated portion, and a specific portion.
When the second image newly captured by the image pickup device was acquired, it was acquired out of the planes based on the specified plane and the orientation of the image pickup device at the time of capturing the acquired second image. A generation unit that generates information indicating a range corresponding to the second image, and a generation unit.
A transmission unit that transmits the acquired second image and the generated information to a specific destination.
A transmitter equipped with. Accepts the designation of the part of the subject included in the first image captured by the image pickup device,
With reference to a storage unit that stores the position of the part of the subject in association with the part, a plane including the position associated with the designated part is specified.
When the second image newly captured by the image pickup device was acquired, it was acquired out of the planes based on the specified plane and the orientation of the image pickup device at the time of capturing the acquired second image. Generate information indicating the range corresponding to the second image,
When a plurality of the second images are acquired, the plurality of the second images are combined based on the information indicating the range corresponding to the plurality of the second images.
An image synthesis program that allows a computer to perform processing. Create management data that manages the range of the subject captured by the imaging device, and create management data.
Based on the management data and the orientation of the imaging device, it is determined whether or not to perform imaging by the imaging device.
The image synthesis program according to claim 6, wherein the processing is further executed by a computer. Accepts the designation of the part of the subject included in the first image captured by the image pickup device,
With reference to a storage unit that stores the position of the part of the subject in association with the part, a plane including the position associated with the designated part is specified.
When the second image newly captured by the image pickup device was acquired, it was acquired out of the planes based on the specified plane and the orientation of the image pickup device at the time of capturing the acquired second image. Generate information indicating the range corresponding to the second image,
When a plurality of the second images are acquired, the plurality of the second images are combined based on the information indicating the range corresponding to the plurality of the second images.
An image composition method characterized in that processing is performed by a computer. A reception unit that accepts the designation of the part of the subject included in the first image captured by the image pickup device, and
With reference to a storage unit that stores the position of the portion of the subject in association with the portion, a specific portion that specifies a plane including the position associated with the designated portion, and a specific portion.
When the second image newly captured by the image pickup device was acquired, it was acquired out of the planes based on the specified plane and the orientation of the image pickup device at the time of capturing the acquired second image. A generation unit that generates information indicating a range corresponding to the second image, and a generation unit.
When a plurality of the second images are acquired, a compositing unit that synthesizes the plurality of the second images based on information indicating a range corresponding to the plurality of the second images, and a compositing unit.
An image synthesizer comprising.

Images