JP7000935B2 - Information processing equipment, information processing system, information processing method and program - Google Patents

Description

The present invention relates to an information processing apparatus, an information processing system, an information processing method and a program.

Check whether the equipment and facilities installed in the factory or office are installed in an appropriate state, and if a failure occurs in these equipment, etc., promptly maintain these equipment, etc. In order to do so, it is required to intuitively grasp the state of these target devices and the like. Therefore, for example, the current image of the target device or the like is acquired, the acquired image is compared with the state in which the target device or the like should be by a computer or the like, and the comparison result is given to the worker or the like who installs the target device or the like. By providing it, the worker or the like can intuitively grasp the state of the target device or the like. As another method, various sensors are attached to the target device or the like in advance, sensor data acquired by the various sensors is modeled according to the actual condition of the target device or the like, and the model is analyzed. Therefore, it is possible to use a method of verifying inadequacies in the above-mentioned target device or the like and providing the verification result to the worker or the like.

For example, in Patent Document 1 below, the mounting of the component is supported by comparing the virtual model of the mounted body to which the component is attached and the virtual model of the component with the three-dimensional image of the work space (real space) acquired by the depth sensor. The work support system to be done is disclosed. Further, Patent Document 2 below discloses a measurement system that displays sensor data acquired by various sensors together with comments (for example, problems detected by the sensors).

Japanese Unexamined Patent Publication No. 2017-191351 Japanese Unexamined Patent Publication No. 2015-040775

However, from the viewpoint of preventing confidentiality leakage to persons other than those concerned, it may not be possible to take out the image of the target device or the like taken in the factory or the like. In addition, it is difficult to grasp the current state of the target device, etc. due to insufficient number of sensors to be mounted on the target device, etc., or the sensors not being mounted at appropriate positions. In some cases.

Therefore, the present invention has been made in view of the above problems, and an object of the present invention is a novel invention capable of intuitively grasping the state of a target device or the like while preventing confidentiality leakage. Moreover, it is an object of providing the improved information processing apparatus, information processing system, information processing method and program.

In order to solve the above problems, according to a certain viewpoint of the present invention, a sensor unit that acquires depth information of an object in real space and a storage that stores shape information regarding the three-dimensional shape of the object. Based on the unit, the positional relationship information calculation unit that calculates the positional relationship information indicating the positional relationship of the sensor unit with respect to the object in the real space, and the positional relationship of the display unit with respect to the object, and the depth information. An image acquisition unit that acquires a real depth image and further acquires a shape depth image based on the shape information converted based on the positional relationship of the sensor unit with respect to the object, the actual depth image, and the above. The difference calculation unit that calculates the difference information that is the difference from the shape depth image and the calculated difference information are converted based on the positional relationship of the display unit with respect to the object, and the actual image of the object or the object is converted. An information processing apparatus is provided that includes the display unit that is superimposed on the captured image of the object and displayed to the user.

The information processing apparatus further includes an effect processing unit that performs effect processing on the difference information in order to emphasize and display the difference information on the real image of the object or the captured image of the object. You may prepare.

The effect processing unit may process the difference information so that the difference information is displayed with a color, brightness, or transparency different from the real image of the object or the captured image of the object. ..

The effect processing unit may process the difference information so that the difference information is displayed in a wire frame.

The sensor unit irradiates the object with irradiation light, receives the reflected light reflected by the object, and detects the phase difference or time difference between the irradiation light and the reflected light. , The depth sensor that acquires the depth information may be used.

The sensor unit may be a stereo camera that acquires the depth information by photographing the object with a plurality of cameras.

The display unit may be a smart eyeglass that is attached to the user's head and displays the difference information superimposed on the real image of the object.

The display unit may be a head-mounted display that is attached to the user's head and displays the difference information superimposed on the captured image of the object.

The information processing device may further include an image pickup unit that images the object.

The information processing apparatus includes at least one of the acceleration sensor and the geomagnetic sensor mounted on the object or the user, and the positional relationship information calculation unit is at least one of the acceleration sensor and the geomagnetic sensor. The positional relationship information may be calculated based on the sensor data obtained from one.

The image acquisition unit is a plane projection unit that projects the converted shape information and the depth information onto a plane, and the difference calculation unit is a shape depth image that is the shape information projected on the plane and a plane. The difference information, which is the difference from the actual depth image, which is the projected depth information, may be calculated.

The difference calculation unit may perform processing on the region where the depth is infinity in the depth information, assuming that no difference has occurred.

Further, in order to solve the above-mentioned problems, according to another viewpoint of the present invention, a sensor unit that acquires depth information of an object in real space and shape information regarding the three-dimensional shape of the object are obtained. The storage unit for storing, the positional relationship information calculation unit for calculating the positional relationship of the sensor unit with respect to the object in the real space, and the positional relationship information indicating the positional relationship of the display unit with respect to the object, and the depth . An image acquisition unit that acquires an actual depth image based on information and further acquires a shape depth image based on the shape information converted based on the positional relationship of the sensor unit with respect to the object, and the actual depth image. The difference calculation unit that calculates the difference information, which is the difference from the shape depth image , and the calculated difference information are converted based on the positional relationship of the display unit with respect to the object, and the object is converted. An information processing system including the display unit that superimposes and displays the real image of the object or the captured image of the object is provided.

Further, in order to solve the above-mentioned problem, according to still another viewpoint of the present invention, the depth information of the object in the real space is acquired from the sensor unit, and the sensor for the object in the real space is obtained. 3 _ Shape information related to a three-dimensional shape is converted based on the positional relationship of the sensor unit with respect to the object, a shape depth image is acquired based on the converted shape information, and the actual depth image and the shape depth image are obtained. The difference information, which is the difference between the object and the subject, is calculated, and the calculated difference information is converted based on the positional relationship of the display unit with respect to the object, and the actual image of the object or the captured image of the object is captured. An information processing method including superimposing on the display unit and displaying the image on the display unit is provided.

Further, in order to solve the above-mentioned problems, according to still another viewpoint of the present invention, a function of acquiring depth information of an object in the real space from a sensor unit by a computer and the object in the real space. A function of calculating the positional relationship of the sensor unit with respect to the object and the positional relationship of the display unit with respect to the object, and an actual depth image acquired based on the depth information, and further stored in advance. A function of converting shape information regarding the three-dimensional shape of the object based on the positional relationship of the sensor unit with respect to the object and acquiring a shape depth image based on the converted shape information, and the actual state. The function of calculating the difference information which is the difference between the depth image and the shape depth image and the calculated difference information are converted based on the positional relationship of the display unit with respect to the object, and the object is converted. A program is provided that realizes a function of superimposing on a real image of the object or an captured image of the object and displaying the image on the display unit .

As described above, according to the present invention, there is provided an information processing device, an information processing system, an information processing method and a program capable of intuitively grasping the state of a target device or the like while preventing confidentiality leakage. be able to.

It is explanatory drawing for demonstrating the outline of the information processing system 1 which concerns on 1st Embodiment of this invention. It is explanatory drawing for demonstrating an example of HMD 32 which concerns on 1st Embodiment of this invention. It is explanatory drawing for demonstrating an example of use of the smart eyeglass 30 which concerns on 1st Embodiment of this invention. It is a block diagram of the processing apparatus 40 which concerns on 1st Embodiment of this invention. It is a flowchart of the information processing method which concerns on 1st Embodiment of this invention. It is explanatory drawing for demonstrating the example of the real image 800 of the real space in 1st Embodiment of this invention. It is explanatory drawing for demonstrating the example of the 3D information 810 of the object 10 in 1st Embodiment of this invention. It is explanatory drawing for demonstrating the processing example in the information processing method which concerns on 1st Embodiment of this invention. It is explanatory drawing for demonstrating an example of superposed image 822 superposed with the difference image 820 which concerns on 1st Embodiment of this invention. It is a block diagram of the processing apparatus 40, 50 which concerns on the 2nd Embodiment of this invention. It is a sequence diagram of the information processing method which concerns on the 2nd Embodiment of this invention. It is explanatory drawing for demonstrating an example of the display image 840 which concerns on the 2nd Embodiment of this invention. It is a block diagram which showed the hardware configuration example of the information processing apparatus 900 which concerns on embodiment of this invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

<< Background to the embodiment of the present invention >>
First, prior to the specific description of the embodiment of the present invention, the background of the present inventor's formation of the embodiment of the present invention will be described.

For example, as explained above, it is possible to check whether the equipment and facilities installed in the factory or business establishment are installed in an appropriate state, or when a failure occurs in these equipment, etc. In order to quickly carry out maintenance of equipment, etc., it is required to intuitively grasp the state of these equipment, etc. Therefore, for example, the current image of the target device or the like is acquired, the acquired image is compared with the state in which the target device or the like should be by a computer or the like, and the comparison result is given to the worker or the like who installs the target device or the like. By providing it, the worker or the like can intuitively grasp the state of the target device or the like.

However, confidentiality leaks to non-related parties in factories with manufacturing lines, in the buildings of financial institutions that handle customer property, and in data centers with multiple servers that store customer information. From the viewpoint of preventing the above, it may be restricted to take out the image taken inside these places to the outside. Therefore, it may not be possible to take out the images of the devices installed in these places, so it is not possible to grasp the current state of the target devices by using the images as described above. It can be difficult.

Therefore, in order to grasp the current state of the target device, it is conceivable to use the following method. For example, as one method, various sensors (for example, a reed switch for detecting the opening / closing of the device) are attached to the target device in advance, and the sensor data acquired by these various sensors is used in the target device or the like. Modeling according to the actual situation can be mentioned. Then, by analyzing the model, it is possible to verify the deficiency in the target device and provide the verification result to the worker or the like, so that the current state of the target device can be grasped. However, even with such a method, the number of sensors to be mounted on the target device is insufficient, or the appropriate sensor is not mounted at an appropriate position. It may not be possible to obtain a model that appropriately shows the state of the target device. In such a case, since an appropriate model cannot be obtained, it is not possible to grasp the state of the target device or the like.

Based on such a situation, the present inventor has been diligently studying to establish a technique capable of intuitively grasping the state of the target device or the like while preventing the leakage of confidential information. In conducting such a study, the present inventor came up with the idea of using a depth sensor to acquire the state of the target device.

The depth sensor irradiates the target object with pulsed laser light or the like (irradiation light) and detects the light reflected on the surface of the target object (reflected light) to obtain the irradiation light and the reflected light. It is a sensor capable of acquiring depth information which is depth information such as the distance from the depth sensor to the surface of the target object, that is, the unevenness of the surface of the target object, by the phase difference or the time difference of the above. Specifically, the depth information can be a three-dimensional coordinate information group which is a collection of relative three-dimensional coordinate information of each point on the surface of the target object when the position of the depth sensor is used as a reference. Further, by using the above-mentioned three-dimensional coordinate information group and projecting it on a plane with a gradation according to the depth (distance) of each point on the surface of the target object, a depth image of the target object (for example, FIG. 8). Actual depth image 802) can be obtained. It can be said that the depth image is a planar image capable of grasping the contour of the target object and the shape of the surface by expressing the contour of the target object and the uneven shape of the surface by the gradation.

That is, it can be said that the depth information that can be acquired by the depth sensor is information about the relative distance from the depth sensor to the target object and the shape of the surface of the target object. Therefore, it is difficult for the depth information obtained by the depth sensor to include, for example, information such as characters and patterns printed or displayed on the surface of the target object unless they are expressed by unevenness. .. Further, although it depends on the accuracy of the depth sensor, it is generally difficult for the depth sensor to obtain information on fine shapes such as fine irregularities on a uniform surface.

In view of such characteristics of the depth sensor, the present inventor may be able to create a technology that can intuitively grasp the current state of the target device while preventing confidentiality leakage by utilizing the characteristics of the depth sensor. I wondered if there was one. For example, in factories, financial institutions, data centers, etc., confidential information that should not be leaked is other equipment, etc. that exist around the target equipment installed in these places, or the surface of the target equipment. It is often the text information displayed in. Therefore, the present inventor has come up with the idea of using the depth information acquired by the depth sensor as the information for grasping the state of the target device. Since the depth information does not include the above-mentioned confidential information due to the characteristics of the depth sensor, the present inventor thought that the use of the depth sensor could prevent the leakage of confidential information. ..

That is, in the embodiment of the present invention conceived by the present inventor, the depth information of the target device obtained by the depth sensor should be modeled (depth imaging) according to the actual condition of the target device, and the model and the target device should be present. It compares with the ideal state and provides the comparison result (difference) to the workers and the like. According to such an embodiment of the present invention, it is possible to intuitively grasp the current state of the target device while preventing confidentiality leakage. Hereinafter, such embodiments of the present invention will be sequentially described in detail.

Each embodiment of the present invention described below is used to support the installation and maintenance of equipment and facilities in factories, financial institutions, data sensors, factories and business establishments where various infrastructure facilities are installed, and the like. be able to. In the following description, a case where the user 20 grasps the state of the object 10 which is the target device and performs maintenance work on the object 10 will be described as an example inside the factory.

<< First Embodiment >>
<Overview of Information Processing System 1>
First, an outline of the information processing system 1 according to the first embodiment of the present invention will be described with reference to FIGS. 1 to 3. Note that FIG. 1 is an explanatory diagram for explaining an outline of the information processing system 1 according to the present embodiment. FIG. 2 is an explanatory diagram for explaining an example of an HMD (Head Mounted Display) 32 according to the present embodiment. Further, FIG. 3 is an explanatory diagram for explaining an example of using the smart eyeglass 30 according to the present embodiment.

As shown in FIG. 1, the information processing system 1 according to the present embodiment includes an object-side sensor 12 mounted on the object 10, a depth sensor (sensor unit) 22, a camera (imaging unit) 24, and the like. It mainly includes an acceleration sensor 26, a geomagnetic sensor 28, a smart eyeglass 30 mounted on the head of the user 20, and a processing device 40. In FIG. 1, for the sake of clarity, the depth sensor 22 is shown in duplicate, such as the depth sensor 22 alone and the depth sensor 22 integrated with the sart eyeglass 30. The outline of each apparatus included in the information processing system 1 according to the present embodiment will be described below.

(Object side sensor 12)
The object-side sensor 12 is a sensor mounted on equipment, equipment, or the like installed inside a factory, which is the object 10 in the present embodiment. Specifically, the object-side sensor 12 can include a group of sensors for detecting the state of the object 10 such as a reed switch for detecting the opening / closing of the object device which is the object 10. In the present embodiment, the object-side sensor 12 may be a sensor previously installed in the device which is the object 10, or is attached to the device which is the object 10. It may be a sensor and is not particularly limited.

In the present embodiment, for example, when the object side sensor 12 is not installed so as to be able to detect the state of the object 10 (for example, the object side sensor 12 is not designed or installed). Even if it is not installed due to such factors), the desired state of the object 10 can be detected by using the information processing system 1 described below. Further, the object-side sensor 12 can transmit the sensor data acquired by itself to the processing device 40 described later via the communication network 70. Then, the processing device 40 can display the state of the object 10 by using the sensor data from the object side sensor 12.

(Depth sensor 22)
The depth sensor 22 is a sensor using Light Detection and Ringing and Laser Imaging Detection and Ranking (LIDAR). In other words, the depth sensor 22 is a sensor that can measure the distance from the depth sensor 22 in the real space to each point on the surface of the target object (object 10 in the present embodiment) by using light. Is. Specifically, the depth sensor 22 irradiates the object 10 with pulsed laser light or the like having a predetermined period (irradiation light), and emits light reflected on the surface of the object 10 (reflected light). By receiving light, depth information such as the contour of the object 10 and the unevenness of the surface of the object 10 can be acquired by the phase difference or the time difference between the irradiation light and the reflected light.

Then, as described above, the depth information is a collection of relative three-dimensional coordinate information of each point on the surface of the object 10 when the position of the depth sensor 22 is used as a reference. It can be a group of information. Therefore, by using the above-mentioned three-dimensional coordinate information group and projecting the object 10 on a plane with a gradation according to the depth (distance) of each point on the surface of the object 10, a depth image of the object 10 (for example, , The actual depth image 802 of FIG. 8) can be obtained.

As described above, such a depth sensor 22 expresses confidential information such as information such as characters and patterns printed or displayed on the surface of the object 10 by unevenness. Unless it's something, it's difficult to get. Therefore, in the present embodiment, by using the depth information obtained by the depth sensor 22, it is possible to intuitively grasp the current state of the object 10 while preventing confidentiality leakage.

Further, in the present embodiment, it is preferable that the depth information used for grasping the state of the object 10 does not include the depth information about the devices and the like existing around the object 10. .. By doing so, in the present embodiment, it is possible to prevent leakage of confidential information such as information on devices and the like existing around the object 10.

Further, the depth sensor 22 can transmit the acquired sensor data (depth information) to the processing device 40 described later via the communication network 70 described later.

It is preferable that the depth sensor 22 has a fixed relative positional relationship with the smart eyeglass 30 described later, and information on the fixed relative positional relationship is acquired in advance. For example, as shown in FIGS. 1 and 3, it is preferable that the depth sensor 22 and the smart eyeglass 30 are integrated. However, the present embodiment is not limited to this, and uses a position information sensor (not shown) that can acquire the position information of the depth sensor 22 and the smart eyeglass 30, and the depth sensor 22 and the smart. Information regarding the relative positional relationship of the eyeglass 30 may be acquired.

Further, in the present embodiment, instead of the depth sensor 22, the parallax between a plurality of cameras can be used to acquire information on the distance to a target object such as the depth sensor 22 from a plurality of cameras. A stereo camera (not shown) may be used. However, when a stereo camera is used, a detailed image of the inside of the factory can be obtained. Therefore, the image of the stereo camera used to grasp the state of the object 10 is confidential information. It is preferable to perform processing such as applying a mask so that an image of a device or the like existing around the object 10 is not included.

(Camera 24)
The camera 24 is an image pickup device that images an object 10, a user 20, a depth sensor 22, a smart eyeglass 30, and the like, and collects light reflected by the object 10 to form an optical image on the image pickup surface. An image can be acquired by converting the optical image formed on the imaging surface into an electrical image signal. Specifically, the camera 24 may have an image pickup optical system such as a photographing lens and a zoom lens for condensing light, and an image pickup element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Sensor). can. Further, the camera 24 can also transmit the image data acquired by the camera 24 to the processing device 40 via the communication network 70 described later. The processing device 40 may calculate positional relationship information indicating the positional relationship between the object 10 and the depth sensor 22 or the smart eyeglass 30 using the image data from the camera 24.

As with the stereo camera (not shown) described above, when the camera 24 is used, a detailed image of the inside of the factory can be obtained. Therefore, the image data of the camera 24 is a subject which is confidential information. It is preferable to perform processing such as applying a mask so that an image of a device or the like existing around the object 10 is not included.

(Accelerometer 26)
The acceleration sensor 26 is realized by, for example, a three-axis acceleration sensor, and is attached to an object 10, a user 20, a depth sensor 22, a smart eyeglass 30, or the like, and detects acceleration due to the movement of the object 10, or the like. The acceleration sensor 26 can also transmit the sensor data acquired by the acceleration sensor 26 to the processing device 40 via the communication network 70. The processing device 40 may use the sensor data from the acceleration sensor 26 to calculate positional relationship information indicating the positional relationship between the object 10 and the depth sensor 22 and the smart eyeglass 30. Further, the acceleration sensor 26 may include a gyro sensor (not shown) mounted on the object 10, the user 20, the depth sensor 22, the smart eyeglass 30, and the like, and the acceleration sensor 26 may include, for example, 3. It is realized by an axial gyro sensor and detects the angular velocity (rotational speed) due to the movement of the object 10 or the like.

(Geomagnetic sensor 28)
The geomagnetic sensor 28 is realized by, for example, a three-axis geomagnetic sensor (compass), is attached to an object 10, a user 20, a depth sensor 22, a smart eyeglass 30, etc., is attached to an object 10, etc. Orientation) is detected. Here, the absolute azimuth refers to the azimuth in the world coordinate system (north, south, east, and west) in the real space. The geomagnetic sensor 28 can also transmit the sensor data acquired by the geomagnetic sensor 28 to the processing device 40 via the communication network 70. The processing device 40 may use the sensor data from the geomagnetic sensor 28 to calculate positional relationship information indicating the positional relationship between the object 10 and the depth sensor 22 or the smart eyeglass 30.

The depth sensor 22, the camera 24, the acceleration sensor 26, and the geomagnetic sensor 28 described above are a fixing device (not shown) located near the object 10 or a moving device such as a robot that can move in the factory (not shown). It may be attached to (not shown). Further, the camera 24, the acceleration sensor 26 and the geomagnetic sensor 28 may be configured as a device integrated with the smart eyeglass 30 mounted on the head of the user 20.

(Smart eyeglass 30)
As shown in FIG. 1, the smart eyeglass 30 is a spectacle-shaped HMD worn on the head of the user 20. The smart eyeglass 30 has a display unit (not shown) corresponding to the spectacle lens portion located in front of the user 20 when worn, and the display unit is a transmissive display capable of visually recognizing the outside of the spectacle lens portion. It will be realized. Since the display unit is a transmissive display, the smart eyeglass 30 is used as a real image 800 (for example, see FIG. 8) of an object (object 10 in the present embodiment) in the real space around the user 20. The images can be superimposed and displayed toward the user 20. In such a transmissive display, for example, a virtual image optical system including a transparent light guide portion using a half mirror or a transparent light guide plate is held in front of the user 20 so that the virtual image optical system is inside the virtual image optical system. An image can be formed and the image can be displayed to the user 20. In the following description, an HMD having a display unit of a transmissive display will be referred to as a "smart eyeglass". Further, the smart eyeglass 30 can receive information from the processing device 40, which will be described later, via the communication network 70. As described above, as shown in FIGS. 1 and 3, the smart eyeglass 30 is preferably an integral device with the depth sensor 22.

Further, in the present embodiment, instead of the smart eyeglass 30, an HMD 32 using a non-transmissive display as a display unit (not shown) may be used. In such a case, as shown in FIG. 2, the HMD 32 can be attached to the head of the user 20 and has an outward-facing camera 34 that captures an image in the real space in the direction visually recognized by the user 20 as an imaging range. .. Then, the HMD 32 can superimpose the image of the object (for example, the object 10) in the real space captured by the outward camera 34 on the image and display another image toward the user 20. .. In FIG. 2, for the sake of clarity, the depth sensor 22 provided to be integrated with the HMD 32 is not shown.

In this embodiment, the forms of the smart eyeglass 30 and the HMD 32 are not limited to the forms shown in FIGS. 1 to 3. For example, the HMD 32 may be a headband type or a helmet type, and in the present embodiment, if the HMD 32 has a display unit (not shown) located in front of the user 20 when worn, the form thereof is particularly high. Not limited.

Further, in the following description, a case where a smart eyeglass 30 having a transmissive display is used will be described as an example.

(Processing device 40)
The processing device 40 can be a notebook PC (Personal Computer), a tablet PC, a smartphone, or the like, and can process the sensor data obtained by the depth sensor 22 or the like described above, or display it on the smart eyeglass 30 described above. You can generate an image. In the present embodiment, in order to prevent confidentiality leakage, it is preferable that the processing device 40 is managed so that it cannot be taken out from a designated place (for example, in a factory). Further, the processing device 40 can transmit and receive information to and from the depth sensor 22 and the smart eyeglass 30 described above via the communication network 70. The detailed configuration of the processing device 40 will be described later.

(Communication network 70)
The communication network 70 is a network used for transmitting and receiving information in one or more factories and the like. The communication network 70 can be wired or wireless, and may be, for example, a public line network such as a telephone line or a satellite communication network, or various LANs (Local Area Network) including the Internet, a dedicated line, and Ethernet (registered trademark). Alternatively, it is composed of an arbitrary communication network such as WAN (Wide Area Network). Further, the dedicated line may be composed of an IP-VPN (Internet Protocol-Virtual Private Network) or the like. In the present embodiment, it is preferable that the communication network 70 is provided with some kind of security means (for example, Secure Sockets Layer (SSL) or the like).

The information processing system 1 according to the present embodiment may not include the camera 24, the acceleration sensor 26, and the geomagnetic sensor 28.

Further, in the present embodiment, the depth sensor 22, the camera 24, the acceleration sensor 26, the geomagnetic sensor 28, the smart eyeglass 30, and the processing device 40 may be configured as a part or all of them as an integrated device. .. For example, when the depth sensor 22, the smart eyeglass 30, and the processing device 40 are configured as an integrated device, it can be a device capable of performing information processing standalone. Further, the above-mentioned processing device 40 may be constructed by a system including a plurality of devices, which is premised on connection to a network such as cloud computing.

<Detailed configuration of processing device 40>
The outline configuration of the information processing system 1 according to the present embodiment has been described above. Next, the detailed configuration of the processing apparatus 40 according to the present embodiment will be described with reference to FIG. FIG. 4 is a block diagram of the processing apparatus 40 according to the present embodiment.

Specifically, the processing device 40 is realized by a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. As shown in FIG. 4, the processing device 40 includes a position / orientation calculation unit (positional relationship information calculation unit) 400, a plane projection unit 402, a difference calculation unit (difference calculation unit) 404, a drawing calculation unit 406, and a storage unit 408. Have. Further, the processing device 40 has a communication unit, although not shown in FIG. 4. Hereinafter, each block of the processing apparatus 40 according to the present embodiment will be described.

(Position / Posture Calculation Unit 400)
The position / orientation calculation unit 400 calculates the positional relationship information indicating the relative distance and posture (direction) of the depth sensor 22 and the smart eyeglass 30 with respect to the object 10 in the real space, and the calculated positional relationship information will be described later. Output to the plane projection unit 402.

Specifically, the position / attitude calculation unit 400 can acquire the relative distance of the depth sensor 22 to the object 10 by acquiring the sensor data from the depth sensor 22 and acquiring the depth information included in the sensor data. .. Further, as described above, the relative positional relationship between the depth sensor 22 and the smart eyeglass 30 is fixed. Therefore, the position / posture calculation unit 400 uses the known information on the relative distance between the depth sensor 22 and the smart eyeglass 30 to obtain the smart relative distance to the object 10 from the relative distance of the depth sensor 22 to the object 10. It is also possible to obtain the relative distance of the eyeglass 30.

Further, the position / orientation calculation unit 400 extracts the feature points of the object 10 (for example, the contour line of the characteristic portion) from the depth information, and stores the feature points in the storage unit 408 described later by using the extracted feature points. From the obtained DataBase (DB) 410, the shape information which is the information of the three-dimensional shape corresponding to the object 10 (for example, the object 10 acquired in advance in the state where the object 10 should be (ideal state)) It is possible to acquire the three-dimensional coordinate information group) of each point on the surface. Then, the position / orientation calculation unit 400 acquires information on the posture (orientation) of the depth sensor 22 with respect to the object 10 by comparing the acquired depth information with the shape information of the object 10 in the ideal state. Can be done. Further, since the relative positional relationship between the depth sensor 22 and the smart eyeglass 30 is fixed, the position / orientation calculation unit 400 determines the posture (orientation) of the depth sensor 22 with respect to the object 10 as described above. Information on the posture (orientation) of the smart eyeglass 30 with respect to the object 10 can also be obtained from the above information. In the present embodiment, instead of the method using the feature points as described above, the user 20 selects in advance the shape information which is the information of the three-dimensional shape corresponding to the object 10. The position / orientation calculation unit 400 may be able to acquire the shape information from the DB 410.

Further, in the present embodiment, the position / orientation calculation unit 400 acquires information on the posture (orientation) of the depth sensor 22 and the smart eyeglass 30 with respect to the object 10 by using the sensor data from the object side sensor 12. You may. Further, the position / orientation calculation unit 400 uses the image data or sensor data from the above-mentioned camera 24, acceleration sensor 26, geomagnetic sensor 28, etc. to the attitude (orientation) of the depth sensor 22 and the smart eyeglass 30 with respect to the object 10. The accuracy of the estimation may be improved.

In the present embodiment, the object 10 may be provided with a marker that serves as a feature point in the depth information, or the object 10 may be provided with a communication tag, an infrared irradiation device, or the like. In this case, the position / orientation calculation unit 400 acquires information on the posture (orientation) of the depth sensor 22 and the smart eyeglass 30 with respect to the object 10 by using a marker, a communication tag, or the like provided on the object 10. You may.

(Plane projection unit 402)
The plane projection unit 402 projects the depth information of the object 10 obtained by the depth sensor 22 and the three-dimensional information corresponding to the object 10 in the ideal state acquired from the DB 410 onto the plane. Each depth image (for example, the actual depth image 802 and the ideal depth image 812 of FIG. 8) can be obtained. Then, the plane projection unit 402 outputs each acquired depth image to the difference calculation unit 404, which will be described later.

Specifically, the plane projection unit 402 uses a three-dimensional coordinate information group which is a collection of relative three-dimensional coordinate information of each point on the surface of the object 10 included in the depth information of the object 10. Then, by projecting the object 10 on a plane with a gradation according to the depth of each point on the surface of the object 10, the depth image of the object 10 in the real space, that is, the actual depth image 802 (see FIG. 8). Can be obtained.

In the present embodiment, the plane projection unit 402 acquires the sensor data obtained by the object-side sensor 12 mounted on the object 10, and reflects the acquired sensor data. Depth image 802 (see FIG. 8) may be generated.

Further, the plane projection unit 402 uses the positional relationship information obtained by the position / orientation calculation unit 400 described above to obtain three-dimensional information 810 (see FIG. 8) corresponding to the object 10 in the ideal state acquired from the DB 410. ) Is normalized to match the actual depth image 802 (see FIG. 8). Further, the plane projection unit 402 projects the normalized three-dimensional information 810 onto the plane with a gradation according to the depth of each point on the surface of the object 10 in the normalized three-dimensional information 810. , The depth image of the object 10 in the ideal state, that is, the ideal depth image 812 (see FIG. 8) can be obtained. More specifically, in the normalization, using the positional relationship information, the relative distance and orientation (posture) of the object 10 in the ideal state to the depth sensor 22 is the depth sensor 22 with respect to the actual object 10. The 3D coordinate information group of each point on the surface of the object 10 in the ideal state included in the 3D information 810 is converted so as to match the relative distance and the direction (posture). Then, the plane projection unit 402 projects the converted three-dimensional coordinate information group onto the plane with a gradation according to the depth of each point on the surface of the object 10 in the same manner as the actual depth image 802. An ideal depth image 812 can be obtained.

For example, since the front surface of the object 10 faces the depth sensor 22, it is assumed that the actual depth image 802 in which the front surface of the object 10 faces straight to the front of the image is obtained. Further, for example, in the ideal depth image 812 obtained by projecting the three-dimensional information 810 corresponding to the object 10 in the ideal state as it is onto a plane, the front surface of the object 10 is tilted diagonally. do. In such a case, even if the actual depth image 802 and the ideal depth image 812 are compared, the orientation (posture) of the object 10 is different from each other in these depth images, so that the current state of the object 10, that is, , It is not possible to accurately grasp the current state of the object 10 with respect to the ideal state. Therefore, in the present embodiment, the plane projection unit 402 uses the above-mentioned positional relationship information to obtain the three-dimensional information 810 corresponding to the object 10 in the ideal state, as in the actual depth image 802. By converting the front surface of the object 10 so that it faces straight ahead of the image, the orientation (posture) of the object 10 in the actual depth image 802 and the ideal depth image 812 is made the same. Then, in the present embodiment, by comparing the actual depth image 802 and the ideal depth image 812 including the images of the objects 10 in the same orientation (posture), the current state of the objects 10 is obtained. Will be able to grasp accurately.

Further, for example, when the distance from the depth sensor 22 to the object 10 is long, the area occupied by the object 10 in the depth image becomes narrow, and when the distance from the depth sensor 22 to the object 10 is short, the area occupied by the object 10 becomes narrow. The area occupied by the object 10 in the depth image becomes large. Therefore, in the present embodiment, similarly to the normalization of the posture (orientation) described above, the plane projection unit 402 normalizes the three-dimensional information 810 corresponding to the object 10 in the ideal state with respect to the relative distance. Therefore, the size occupied by the object 10 in the actual depth image 802 (see FIG. 8) and the ideal depth image 812 (see FIG. 8) is matched, and the ideal depth image 812 comparable to the actual depth image 802 is acquired. ..

(Difference calculation unit 404)
The difference calculation unit 404 includes an actual depth image 802 (see FIG. 8) obtained by the above-mentioned plane projection unit 402 and an ideal depth image 812 (see FIG. 8) obtained by conversion based on the positional relationship information. Is calculated as the difference information (difference image 820, see FIG. 8), and the calculation result is output to the drawing calculation unit 406 described later. Specifically, the difference calculation unit 404 compares the actual depth image 802 and the ideal depth image 812, and extracts a portion different from each other as the difference image 820. At this time, it is preferable that the difference calculation unit 404 processes the region where the depth is infinity in the actual depth image 802 obtained from the depth information, assuming that no difference has occurred. Further, if the difference calculation unit 404 can determine that the actual depth image 802 and the ideal depth image 812 are different from each other but are not occupied by the object 10, the difference is generated in the different parts. It is preferable to treat it as absent (not generated). By doing so, in the present embodiment, it is possible to prevent information about the device or the like existing around the object 10 which is confidential information from being leaked.

In the present embodiment, as described above, the difference calculation unit 404 is not limited to calculating the difference between the actual depth image 802 and the ideal depth image 812 obtained by projecting onto a plane. do not have. For example, the difference calculation unit 404 is obtained by converting the depth information (three-dimensional coordinate information group) obtained by the depth sensor 2 and the three-dimensional information 810 of the object 10 in the ideal state based on the positional relationship information. The difference from the three-dimensional information 810 (three-dimensional coordinate information group) may be calculated. Then, in such a case, the processing apparatus 40 may acquire the above-mentioned difference image 820 by projecting the calculated difference onto the plane by the above-mentioned plane projection unit 402.

(Drawing calculation unit 406)
The drawing calculation unit 406 converts the difference information (difference image 820, see FIG. 8) obtained by the difference calculation unit 404 described above into a form suitable for the display of the smart eyeglass 30, and converts the converted information into a form. Send to smart eyeglass 30. Then, the smart eyeglass 30 can display the difference image 820 by superimposing the received information on the real image 800 (see FIG. 8) of the object 10.

Specifically, the drawing calculation unit 406 uses the above-mentioned positional relationship information, that is, the positional relationship information indicating the relative distance and posture (orientation) of the smart eyeglass 30 with respect to the object 10, and the actual smart eye. The size, posture (orientation), and display position occupied by the object 10 included in the difference image 820 (see FIG. 8) are converted so as to match the relative distance and posture (orientation) of the object 10 with respect to the glass 30. do. By doing so, the difference image 820 displayed by the smart eyeglass 30 can be displayed superimposed on the real image 800 (see FIG. 8) of the object 10 in the real space. Then, in the present embodiment, since the difference image 820 is superimposed on the real image 800 of the object 10 in the real space, the user 20 is in the current state of the object 10 and the ideal state. It is possible to intuitively grasp the difference between the objects 10, that is, the deficiencies of the object 10.

In the present embodiment, the drawing calculation unit 406 displays the difference image 820 (see FIG. 8) with respect to the real image 800 (or the captured image of the object 10) of the object 10 (see FIG. 8). In order to emphasize and display the difference image 820, it may function as an effect processing unit that performs effect processing on the difference image 820. For example, the drawing calculation unit 406 may process the difference image 820 so that the difference image 820 is displayed with a color, brightness, or transparency different from that of the real image 800 of the object 10. At this time, the drawing calculation unit 406 may use the color information or the like of the object 10 stored in the DB 410 in advance. Further, the drawing calculation unit 406 displays the difference image 820 in the form of a wire frame 830 (see FIG. 9) (wire frame display) in which the portion of the object 10 related to the difference is represented only by the contour line. May be processed.

(Memory unit 408)
The storage unit 408 is realized by a storage device such as a ROM and a RAM, and stores shape information (three-dimensional coordinate information group) regarding the three-dimensional shape of the object 10 as a DB 410. Specifically, the DB 410 has 3D information 810 (FIG.) which is a 3D coordinate information group of each point on the surface of the object 10 in the state where the object 10 should be (ideal state) acquired in advance. 8) is stored. The storage unit 408 may not be provided inside the processing device 40, and may be configured as a device separate from the processing device 40, for example.

The processing device 40 is not limited to including only the block shown in FIG. 4, and for example, in the present embodiment, the communication module for transmitting and receiving information between the depth sensor 22 and the like described above. It may include a communication unit which is.

<Information processing method>
The detailed configuration of the processing apparatus 40 according to the present embodiment has been described above. Next, the information processing method according to the present embodiment will be described with reference to FIGS. 3 and 5 to 9. FIG. 5 is a flowchart of the information processing method according to the present embodiment. FIG. 6 is an explanatory diagram for explaining an example of the real image 800 in the real space in the present embodiment. FIG. 7 is an explanatory diagram for explaining an example of the three-dimensional information 810 of the object 10 in the present embodiment. FIG. 8 is an explanatory diagram for explaining a processing example in the information processing method according to the present embodiment. Further, FIG. 9 is an explanatory diagram for explaining an example of the superimposed image 822 on which the difference image 820 according to the present embodiment is superimposed.

Specifically, the information processing method according to the present embodiment can include a plurality of steps from step S101 to step S113 as shown in the flowchart of FIG. The details of each step of the information processing method according to the present embodiment will be described below.

First, as shown in FIG. 3, the user 20 wears the smart eyeglass 30 integrated with the depth sensor 22, enters the factory, and takes a posture facing the object 10. By doing so, the depth sensor 22 acquires sensor data including depth information in the real space including the object 10.

In the example described below, the object 10 installed in the factory (real space) is in a state in which the drawer 10a arranged at the lower part of the object 10 is pulled out as shown in FIG. It is assumed that the object 10 in which the drawer 10a is stored is in a state different from the state (ideal state) in which it should be. Further, as shown in FIG. 6, in the following example, it is assumed that an object 60 that is not a target of processing in the present embodiment exists around the object 10. Further, in the following example, the drawer 10a arranged in the lower part of the object 10 as shown in FIG. 7 as the state (ideal state) in which the object 10 should be in the DB 410 of the storage unit 408. It is assumed that the three-dimensional information 810 (see FIG. 8), which is a three-dimensional coordinate information group of each point on the surface of the object 10 in the state of being stored in the object 10, is stored.

(Step S101)
The processing device 40 acquires sensor data from the depth sensor 22. At this time, it is assumed that the user 20 can see the real image 800 as shown in the upper left side of FIG. Then, the processing device 40 uses a three-dimensional coordinate information group which is a collection of relative three-dimensional coordinate information of each point on the surface of the object 10 and the object 60 included in the depth information of the sensor data of the depth sensor 22. Then, a gradation is added according to the depth of each point on the surface of the object 10 and the object 60, and the object is projected onto a plane. More specifically, the processing apparatus 40 darkens the depth information, for example, the points on the surface of the object 10 and the object 60, which are close to the depth sensor 22 in the real space, while in the real space. The actual depth image 802 as shown in FIG. 8 is obtained by projecting the points on the surface of the object 10 and the object 60 so as to be thin so that the relative distance to the depth sensor 22 is long. In the actual depth image 802 illustrated in FIG. 8, the contour of the object and the boundary between different gradations are shown by the contour line and the boundary line for the sake of clarity, but in the actual actual depth image 802, the gradation is shown. Only the outline and shape of the object are shown.

(Step S103)
The processing device 40 uses the sensor data (depth information) acquired by the depth sensor 22 to indicate the relative distance and posture (direction) of the depth sensor 22 and the smart eyeglass 30 with respect to the object 10 in the real space. Calculate positional relationship information. Specifically, the processing apparatus 40 extracts the feature points of the object 10 from the depth information of the sensor data, and by using the extracted feature points, the subject is subjected to the object from the DB 410 stored in the storage unit 408, for example. By extracting the three-dimensional information 810 corresponding to the object 10 and comparing the feature points with the three-dimensional information 810, information on the posture (orientation) of the depth sensor 22 with respect to the object 10 is acquired. Further, as described above, since the relative positional relationship between the depth sensor 22 and the smart eyeglass 30 is fixed, the processing device 40 has a relative distance and posture of the depth sensor 22 with respect to the object 10. From the (orientation) information, information on the relative distance and posture (orientation) of the smart eyeglass 30 with respect to the object 10 can also be acquired.

Further, in the present embodiment, the processing device 40 uses the image data or sensor data from the above-mentioned camera 24, acceleration sensor 26, geomagnetic sensor 28, etc. to the depth sensor 22 and the smart eyeglass 30 with respect to the object 10. The accuracy of posture (orientation) estimation may be improved. Alternatively, in the present embodiment, the object 10 or the like may be provided with a marker that serves as a feature point in the depth information, or the object 10 or the like may be provided with a communication tag, an infrared irradiation device, or the like. good. In this case, the processing device 40 acquires information on the posture (orientation) of the depth sensor 22 and the smart eyeglass 30 with respect to the object 10 by using a marker, a communication tag, or the like provided on the object 10 or the like. be able to.

(Step S105)
The processing apparatus 40 uses the positional relationship information obtained in step S103 described above to obtain the object in the ideal state included in the three-dimensional information 810 corresponding to the object 10 in the ideal state acquired from the DB 410. The relative distance and orientation (direction) of the object 10 included in the three-dimensional information 810 to the depth sensor 22 of the three-dimensional coordinate information group of each point on the surface of the surface 10 is relative to the actual object 10. Convert (normalize) to match the distance and orientation (orientation). Further, the processing apparatus 40 projects the normalized three-dimensional information 810 onto a plane with a gradation according to the depth of each point on the surface of the object 10 in the normalized three-dimensional information 810. The ideal depth image 812 shown in FIG. 8 is created. In the ideal depth image 812 illustrated in FIG. 8, the contour of the object and the boundary between different gradations are shown by the contour line and the boundary line for the sake of clarity, but in the actual ideal depth image 812, the gradation is shown. Only the outline and shape of the object are shown.

(Step S107)
The processing apparatus 40 calculates the difference image 820 shown in FIG. 8, which is the difference between the actual depth image 802 obtained in the above step S101 and the ideal depth image 812 obtained in the above step S105. Specifically, the processing apparatus 40 compares the actual depth image 802 and the ideal depth image 812, and extracts a portion different from each other as the difference image 820. In the difference image 820, for example, an image having a large difference may be displayed brightly, and an image having no difference may be displayed dark.

At this time, as described above, the processing apparatus 40 processes the region where the depth is infinity in the actual depth image 802 obtained from the depth information, assuming that no difference has occurred. Further, even if the processing device 40 has a different portion between the actual depth image 802 and the ideal depth image 812, if it can be determined that the region is not occupied by the object 10, no difference is generated in the different portion. Treat as a thing. Therefore, in the difference image 820 in FIG. 8, although the difference is generated between the actual depth image 802 and the ideal depth image 812 due to the presence of the object 60, the object 60 is not a target of processing in the present embodiment. The difference due to the existence of the object 60 is processed as not being a difference. By doing so, in the present embodiment, it is possible to avoid leakage of confidential information about the device or the like existing around the object 10.

(Step S109)
The processing device 40 uses the positional relationship information indicating the relative distance and posture (orientation) of the smart eyeglass 30 with respect to the object 10 acquired in step S103 described above, and the object with respect to the actual smart eyeglass 30. Display (drawing) for converting the size, posture (direction) and display position occupied by the object 10 included in the difference image 820 so as to match the relative distance and posture (orientation) of the object 10 and superimposing the display. Figure) is created. By doing so, in the present embodiment, the difference image 820 displayed by the smart eyeglass 30 can be displayed superimposed on the real image 800 on the object 10 in the real space.

(Step S111)
The processing device 40 performs effect processing on the difference image 820 in order to emphasize and display the difference image 820 on the real image 800 of the object 10. For example, the processing device 40 performs effect processing on the difference image 820 so as to be displayed in the form of a wire frame 830 (see FIG. 9) in which the portion of the object 10 related to the difference is represented only by the contour line. .. In this embodiment, the implementation of step S111 may be omitted.

(Step S113)
The processing device 40 controls the smart eyeglass 30 to superimpose and display the difference image 820 processed as described above on the real image 800 of the object 10 in the real space. For example, the processing device 40 displays the superimposed image 822 as shown in FIG. 8 on the smart eyeglass 30. In the present embodiment, by superimposing the difference image 820 on the real image 800 of the object 10, the user 20 can see the difference between the current state of the object 10 and the object 10 in the ideal state. That is, the defect of the object 10 can be intuitively grasped.

Further, when the effect processing is performed, for example, as shown in FIG. 9, the portion of the drawer 10a, which is the difference between the real depth image 802 and the ideal depth image 812, is represented by the wire frame 830, and the wire frame is represented. The 830 is displayed superimposed on the real image 800 of the object 10 in the real space. In the present embodiment, by drawing the difference image 820 with the wire frame 830, the difference image 820 can be displayed without obscuring the real image 800 or the captured image of the object 10, so that the user 20 can display the object. The state of 10 can be grasped more easily.

When an HMD 32 using a non-transmissive display is used instead of the smart eyeglass 30, the HMD 32 is an image captured by the object 10 in the real space captured by the outward camera 34 of the HMD 32. The difference image 820 is displayed by superimposing on the image.

As described above, in the first embodiment of the present invention, the difference image 820, which is the difference between the actual object 10 and the object 10 in the ideal state, is superimposed on the real image 800 of the object 10. Since it can be displayed, the user 20 can intuitively display the difference between the current state of the object 10 and the object 10 in the ideal state, that is, the state of the object 10 such as a defect of the object 10. Can be grasped. Therefore, according to the present embodiment, for example, the number of object-side sensors 12 to be attached to the object 10 is insufficient, or the object-side sensor 12 is not attached at an appropriate position. Even in this case, the state of the object 10 can be grasped. As a result, for example, it becomes easy to identify the shortage of the object side sensor 12 and detect the installation omission, and further, it becomes easy to detect the failure of the object side sensor 12.

Further, in the present embodiment, the depth information of the object 10 acquired by the depth sensor 22 is used as the information for grasping the state of the object 10. Therefore, according to the present embodiment, since the depth information does not include confidential information such as character information displayed on the surface of the object 10 due to the feature of the depth sensor 22, such confidential information is external. It is possible to prevent leakage to.

<< Second Embodiment >>
In the first embodiment of the present invention described above, a case where the user 20 grasps the state of the object 10 which is the target device and performs maintenance work inside the factory has been described as an example. However, depending on the content of the maintenance work of the object 10, the user 20 in the factory may need support from outside the factory, and other users outside the factory can grasp the state of the object 10. You may be required to do so. Further, when another user outside the factory grasps the state of the object 10, the efficiency of maintenance work may be improved (working time may be shortened, etc.). Therefore, a second embodiment of the present invention will be described, which can intuitively grasp the current state of the object 10 which is the target device while preventing confidentiality leakage even in such a case. do.

<Detailed configuration of processing device 50>
The information processing system 1 according to the present embodiment further includes a processing device 50 in the information processing system 1 according to the first embodiment of the present invention shown in FIG. Therefore, the detailed configuration of the processing device 50 will be described with reference to FIG. FIG. 10 is a block diagram of the processing devices 40 and 50 according to the second embodiment of the present invention. In this embodiment, the detailed configuration of each device included in the information processing system 1 other than the processing device 50 is the same as that of the first embodiment described above, and thus the description thereof will be omitted here.

The processing device 50 is realized by a desktop PC, a notebook PC, a tablet terminal, a smartphone, or the like, and unlike the processing device 40, the processing device 50 is installed in a place other than the factory where the object 10 exists, that is, is covered. It is assumed that it is installed at a place away from the object 10. Further, the processing device 50 can send and receive information to and from the processing device 40 described above via the communication network 70.

Specifically, the processing device 50 is realized by a CPU, RAM, ROM, a display device, and the like, similarly to the processing device 40. As shown in FIG. 10, the processing apparatus 50 mainly includes a plane projection unit 502, a difference calculation unit 504, a drawing calculation unit 506, and a display 508. Hereinafter, each block of the processing apparatus 50 according to the present embodiment will be described. The plane projection unit 502, the difference calculation unit 504, and the drawing calculation unit 506 of the processing device 50 have the same names as the plane projection unit 402, the difference calculation unit 404, and the drawing calculation unit 406 of the processing device 40. Since it has a function, the description of these blocks is omitted here, and only the display 508 will be described.

(Display 508)
The display 508 is a display device that displays the difference image 820 superimposed on the captured image of the object 10 to other users (users other than the user 20) existing outside the factory, and is, for example, a liquid crystal display. It is realized by a display device such as a (LCD) device and an OLED (Organic Light Emitting Diode) device.

The processing device 50 is not limited to including only the block shown in FIG. 10, and for example, in the present embodiment, communication for transmitting / receiving information to / from the processing device 40 described above. It may include a communication unit which is a module.

<Information processing method>
The detailed configuration of the processing apparatus 50 according to the present embodiment has been described above. Next, the information processing method according to the present embodiment will be described with reference to FIGS. 11 and 12. FIG. 11 is a sequence diagram of the information processing method according to the present embodiment, and FIG. 12 is an explanatory diagram for explaining an example of the display image 840 according to the present embodiment.

Specifically, the information processing method according to the present embodiment can include a plurality of steps from step S201 to step S219, as shown in the sequence diagram of FIG. The details of each step of the information processing method according to the present embodiment will be described below.

(Step S201)
Similar to the information processing method according to the first embodiment described above, the user 20 wears the smart eyeglass 30 integrated with the depth sensor 22, enters the factory, and takes a posture facing the object 10. .. By doing so, the depth sensor 22 senses depth information in the real space including the object 10.

In addition, also in the example described below, the object 10 installed in the factory (real space) is in a state where the drawer 10a arranged at the lower part of the object 10 is pulled out as shown in FIG. It is assumed that the object 10 in which the drawer 10a is stored is in a state different from the state (ideal state) in which it should be. Further, also in the following example, it is assumed that an object 60 that is not a target of processing in the present embodiment exists around the object 10. Further, also in the following example, the drawer 10a arranged in the lower part of the object 10 as shown in FIG. 7 as the state (ideal state) in which the object 10 should be in the DB 410 of the storage unit 408. It is assumed that the three-dimensional information 810 (see FIG. 8), which is a three-dimensional coordinate information group of each point on the surface of the object 10 in the state of being stored in the object 10, is stored.

(Step S203)
Since the step S203 is the same as the step S101 of the first embodiment shown in FIG. 5, the description thereof is omitted here.

(Step S205)
Since the step S205 is the same as the step S103 of the first embodiment shown in FIG. 5, detailed description thereof will be omitted, but the processing apparatus 40 has a depth sensor 22 and a depth sensor 22 for the object 10 in the real space. The positional relationship information indicating the relative distance and posture (direction) of the smart eyeglass 30 is calculated. Further, the processing device 40 transmits the calculated positional relationship information to the processing device 50 via the communication network 70.

(Step S207)
The processing device 50 acquires sensor data including depth information from the depth sensor 22 via the communication network 70, and the relative three-dimensional coordinates of each point on the surface of the object 10 and the object 60 included in the depth information. Using a three-dimensional coordinate information group that is a collection of information, a gradient is added according to the depth of each point on the surface of the object 10 and the object 60 and projected onto a plane, and the actual depth is as shown in FIG. Create image 802.

Further, the processing device 50 acquires captured image data of the object 10 captured by the camera 24 from the camera 24 (or, when the HMD 32 is used, the outward camera 34 of the HMD 32). In the present embodiment, in the processing device 50, the image data of the object 10 captured by the camera 24 includes equipment (for example, an object 60) existing around the object 10 which is confidential information. It is preferable to perform processing such as applying a mask so that the image of the image is not included.

(Step S209)
The processing device 50 is the relative distance and posture (direction) of the depth sensor 22 and the smart eyeglass 30 with respect to the object 10 calculated by the processing device 40 in the above-mentioned step S205 from the processing device 40 via the communication network 70. ) Is acquired.

(Step S211)
Since the step S211 is the same as the step S105 of the first embodiment shown in FIG. 5, except that the step S211 is executed by the processing apparatus 50 using the positional relationship information acquired by the processing apparatus 40, it will be described here. Is omitted.

(Step S213)
Since the step S213 is the same as the step S107 of the first embodiment shown in FIG. 5 except that it is executed by the processing apparatus 50, the description thereof is omitted here.

(Step S215)
Since the step S215 is the same as the step S109 of the first embodiment shown in FIG. 5 except that the step S215 is executed by the processing apparatus 50, the description thereof is omitted here.

(Step S217)
Since the step S217 is the same as the step S111 of the first embodiment shown in FIG. 5 except that the step S217 is executed by the processing apparatus 50, the description thereof is omitted here.

(Step S219)
The processing device 50 controls the display 508 to superimpose the difference image 820 on the captured image of the object 10 captured by the camera 24 (or, when the HMD 32 is used, the outward camera 34 of the HMD 32). And display. For example, the processing device 50 displays the display image 840 as shown in FIG. 12 on the display 508. In the display image 840, it is preferable that the difference image 820 of the portion of the drawer 10a, which is the difference between the actual depth image 802 and the ideal depth image 812, is emphasized and displayed with respect to the captured image of the object 10. .. In the present embodiment, by superimposing and displaying such a difference image 820 on the captured image of the object 10, the user 20 can use the object 10 in the current state and the ideal state of the object 10. The difference between the two, that is, the deficiency of the object 10 can be intuitively grasped.

In the present embodiment, the display image 840 is masked so as not to include an image of a device (for example, an object 60) existing around the object 10 which is confidential information. It is preferable to display after processing. Therefore, in the display image 840 shown in FIG. 12, the object 60 existing around the object 10 is not displayed.

As described above, in the second embodiment of the present invention, similarly to the first embodiment described above, the difference image 820 which is the difference between the actual object 10 and the object 10 in the ideal state is obtained. Since the image of the object 10 can be superimposed and displayed, the user outside the factory can see the difference between the current state of the object 10 and the object 10 in the ideal state, that is, the object. It is possible to intuitively grasp the state of the object 10 such as a defect of the object 10.

Further, also in this embodiment, the depth information of the object 10 acquired by the depth sensor 22 is used as the information for grasping the state of the object 10. Therefore, according to the present embodiment, since the depth information does not include confidential information such as character information displayed on the surface of the object 10 due to the feature of the depth sensor 22, such confidential information is external. It is possible to prevent leakage to. In addition, in the present embodiment, the display image 840 displayed to the user outside the factory includes equipment (for example, object 60) existing around the object 10 which is confidential information. Since the image is displayed after being processed so as not to be included, the confidential information, which is the information of the device existing around the object 10, is not leaked to the user existing outside the factory.

In the above description, it has been described that the processing device 50 projects a plane to acquire a depth image or a difference image 820, but the present embodiment is limited to this. Instead, after performing these processes in the processing device 40, the processing data may be transmitted to the processing device 50 and displayed on the display 508 of the processing device 50. By doing so, the processing load in the processing apparatus 50 can be reduced.

Further, this embodiment can also be implemented by attaching a depth sensor 22 or the like to a self-propelled robot or the like that can move in the factory instead of the user 20.

<< Summary >>
As described above, in each embodiment of the present invention, the difference image 820, which is the difference between the actual object 10 and the object 10 in the ideal state, is superimposed on the real image 800 or the captured image of the object 10. Therefore, the user 20 or another user can display the difference between the current state of the object 10 and the object 10 in the ideal state, that is, the object 10 such as a defect of the object 10. You can intuitively grasp the state of.

Further, in each embodiment of the present invention, the depth information of the object 10 acquired by the depth sensor 22 is used as the information for grasping the state of the object 10. Therefore, according to each embodiment of the present invention, the depth information does not include confidential information such as character information displayed on the surface of the object 10 due to the characteristics of the depth sensor 22, and thus such confidentiality. It is possible to prevent information from leaking to the outside.

Further, in each embodiment of the present invention, not only the difference image 820 is superimposed and displayed, but also the sensor data obtained from the object side sensor 12 and the shape information of the object 10 stored in the DB 410 are used. The image generated by the above may also be superimposed and displayed. By doing so, the user 20 or the user of the shop can more easily grasp the state of the object 10. At this time, it is preferable to draw the difference image 820 with the wire frame 830, and by doing so, the difference image 820 can be displayed without obscuring the real image 800 or the captured image of the object 10.

As described above, each embodiment of the present invention supports the installation and maintenance of equipment and facilities in factories, business establishments, etc. where factories, financial institutions, data sensors, and various infrastructure facilities are installed. Can be used for

Furthermore, each embodiment of the present invention is not limited to the above-mentioned use, and can be used for other uses. For example, according to the present embodiment, when the object 10 is a person instead of a device or the like, the passage status of the person at a predetermined place and the direction (posture) of the person are detected, and the contour of the detected person is detected. Can be superimposed and displayed on a captured image at a predetermined location as a difference image 820. More specifically, by using this embodiment, it is possible to display the passage status of the corridor in the building, the flow of passengers at the ticket gate of public transportation, and the congestion status. At this time, in the present embodiment, since the depth information acquired by the depth sensor 22 is used as the information for grasping the state of the person (object 10 in the above-described embodiment), the existence and orientation of the person (the existence and orientation of the person) ( Although the posture) can be grasped, it is difficult to obtain detailed facial information for identifying the person. As a result, according to the present embodiment, it is possible to easily grasp the crime prevention in the building and the congestion situation of the ticket gate while protecting the privacy of the passing person.

<< Hardware configuration >>
The embodiments of the present invention have been described above. Subsequently, the hardware configuration according to each embodiment of the present invention will be described. FIG. 13 is a block diagram showing a hardware configuration example of the information processing device 900, which is the processing device 40 (processing device 50) according to the embodiment of the present invention. The information processing apparatus 900 has a CPU 902, a ROM 904, and a RAM 906. Further, the information processing device 900 has an input unit 908, an output unit 910, a storage device 912, and a network interface 914. Hereinafter, each hardware block of the information processing apparatus 900 will be described.

(CPU902, ROM904, RAM906)
The CPU 902 functions as an arithmetic processing device, and controls the overall operation in the information processing device 900, which is the processing device 40, according to various programs. Further, the CPU 902 may be a microprocessor. The ROM 904 stores programs, calculation parameters, and the like used by the CPU 902. The RAM 906 temporarily stores a program used in the execution of the CPU 902, parameters that appropriately change in the execution, and the like. These are connected to each other by a host bus 916 composed of a CPU bus or the like. The CPU 902, ROM 904, and RAM 906 can realize the functions of the position / orientation calculation unit 400 of the processing device 40 and the like described with reference to FIG. 4 in collaboration with software.

(Input unit 908)
The input unit 908 is an input means for the user to input information such as a mouse, a keyboard, a touch panel, a button, a microphone, a sensor, and a switch, and an input control circuit that generates an input signal based on the input by the user and outputs the input signal to the CPU 902. It is composed of such things. By operating the input unit 908, the user can input various data to the processing device 40 and instruct the processing operation.

(Output unit 910)
The output unit 910 includes, for example, a display device such as a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, a projector device, an OLED (Organic Light Emitting Diode) device, and a lamp. Further, the output unit 910 may include the above-mentioned smart eyeglass 30 or the like.

(Storage device 912)
The storage device 912 is a device for storing data. The storage device 912 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, a deleting device for deleting data recorded on the storage medium, and the like. The storage device 912 is composed of, for example, a ROM, an HDD, an SSD (Solid Storage Drive), or the like. The storage device 912 drives the storage and stores programs and various data executed by the CPU 902.

(Network interface 914)
The network interface 914 is, for example, a communication interface configured by a communication device or the like for connecting to a network. The communication interface may be, for example, a short-range wireless communication interface such as Bluetooth (registered trademark) or ZigBee (registered trademark), a wireless LAN (Local Area Network), Wi-Fi (registered trademark), or a mobile communication network (LTE, It can be a communication interface such as 3G). Further, the network interface 914 may include a wired communication device that performs wired communication.

The hardware configuration example of the information processing apparatus 900 which is the processing apparatus 40 according to the embodiment of the present invention has been shown above. Each of the above-mentioned components may be configured by using a general-purpose member, or may be configured by hardware specialized for the function of each component. Such a configuration may be changed as appropriate depending on the technical level at the time of implementation.

<< Supplement >>
Although the preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, the present invention is not limited to these examples. It is clear that a person having ordinary knowledge in the field of technology to which the present invention belongs can come up with various modifications or modifications within the scope of the technical ideas described in the claims. , These are also naturally understood to belong to the technical scope of the present invention.

Further, the information processing method in the above-described embodiment of the present invention does not necessarily have to be operated or processed in the order described. For example, the steps of each operation may be processed by changing the order as appropriate. Further, each step may be partially processed in parallel or individually instead of being processed in chronological order. Further, the operation method or processing method of each step does not necessarily have to be operated or processed according to the described method, and may be operated or processed by another functional unit, for example, by another method.

Further, at least a part of the information processing method according to the embodiment of the present invention described above can be configured by software as an information processing program for operating a computer, and when configured by software, the above operation or the above operation or A program that realizes at least a part of processing is stored in a recording medium such as a flexible disk or a CD-ROM (Compact Disk Read Only Memory), and is stored in the processing device 40 or the like or another device connected to the processing device 40 or the like. It may be read and executed. The recording medium is not limited to a removable one such as a magnetic disk or an optical disk, and may be a fixed recording medium such as a hard disk device or a memory. Further, a program that realizes at least a part of the above information processing method may be distributed via a communication line (including wireless communication) such as the Internet. Further, the program may be encrypted, modulated, compressed, and distributed via a wired line or a wireless line such as the Internet, or stored in a recording medium.

1 Information processing system 10 Object 10a Drawer 12 Object side sensor 20 User 22 Depth sensor 24 Camera 26 Accelerometer 28 Geomagnetic sensor 30 Smart eyeglass 32 HMD
34 Outward camera 40, 50 Processing device 60 Object 70 Communication network 400 Position / orientation calculation unit 402, 502 Plane projection unit 404, 504 Difference calculation unit 406, 506 Drawing calculation unit 408 Storage unit 410 DB
508 Display 800 Real image 802 Real depth image 810 3D information 812 Ideal depth image 820 Difference image 822 Superimposed image 830 Wireframe 840 Display image 900 Information processing device 902 CPU
904 ROM
906 RAM
908 Input section 910 Output section 912 Storage device 914 Network interface 916 Host bus

Claims (15)

A sensor unit that acquires depth information of an object in real space,
A storage unit that stores shape information related to the three-dimensional shape of the object, and
A positional relationship information calculation unit that calculates positional relationship information indicating the positional relationship of the sensor unit with respect to the object in the real space and the positional relationship of the display unit with respect to the object object .
An image acquisition unit that acquires an actual depth image based on the depth information and further acquires a shape depth image based on the shape information converted based on the positional relationship of the sensor unit with respect to the object.
A difference calculation unit that calculates difference information that is a difference between the actual depth image and the shape depth image .
The calculated difference information is converted based on the positional relationship of the display unit with respect to the object, superimposed on the real image of the object or the captured image of the object, and displayed to the user. With the display unit
An information processing device equipped with. Further provided with an effect processing unit that performs effect processing on the difference information in order to emphasize and display the difference information on the real image of the object or the captured image of the object.
The information processing apparatus according to claim 1. The effect processing unit processes the difference information so that the difference information is displayed with a color, brightness, or transparency different from the real image of the object or the captured image of the object. The information processing apparatus according to 2. The information processing apparatus according to claim 2, wherein the effect processing unit processes the difference information so that the difference information is displayed in a wire frame. The sensor unit irradiates the object with irradiation light, receives the reflected light reflected by the object, and detects the phase difference or time difference between the irradiation light and the reflected light. The information processing apparatus according to any one of claims 1 to 4, which is a depth sensor for acquiring the depth information. The information processing device according to any one of claims 1 to 4, wherein the sensor unit is a stereo camera that captures the object with a plurality of cameras and acquires the depth information. The display unit is a smart eyeglass that is attached to the head of the user and displays the difference information superimposed on the real image of the object, according to any one of claims 1 to 6. Information processing device. The display unit is a head-mounted display that is attached to the head of the user and displays the difference information superimposed on the captured image of the object, according to any one of claims 1 to 6. Information processing equipment. The information processing apparatus according to claim 8, further comprising an imaging unit for imaging the object. It comprises at least one of the accelerometer and the geomagnetic sensor worn on the object or the user.
The positional relationship information calculation unit calculates the positional relationship information based on sensor data obtained from at least one of the acceleration sensor and the geomagnetic sensor.
The information processing apparatus according to any one of claims 1 to 9. The image acquisition unit is a plane projection unit that projects the converted shape information and depth information onto a plane.
The difference calculation unit calculates the difference information, which is the difference between the shape depth image, which is the shape information projected on the plane, and the actual depth image, which is the depth information projected on the plane.
The information processing apparatus according to any one of claims 1 to 10. The information processing apparatus according to any one of claims 1 to 11, wherein the difference calculation unit performs processing on a region where the depth is infinity in the depth information, assuming that no difference has occurred. A sensor unit that acquires depth information of an object in real space,
A storage unit that stores shape information related to the three-dimensional shape of the object, and
A positional relationship information calculation unit that calculates positional relationship information indicating the positional relationship of the sensor unit with respect to the object in the real space and the positional relationship of the display unit with respect to the object object .
An image acquisition unit that acquires an actual depth image based on the depth information and further acquires a shape depth image based on the shape information converted based on the positional relationship of the sensor unit with respect to the object.
A difference calculation unit that calculates difference information that is a difference between the actual depth image and the shape depth image .
The display unit that converts the calculated difference information based on the positional relationship of the display unit with respect to the object object and superimposes it on the real image of the object object or the captured image of the object object.
Information processing system equipped with. Obtain the depth information of the object in the real space from the sensor part,
The positional relationship information indicating the positional relationship of the sensor unit with respect to the object in the real space and the positional relationship of the display unit with respect to the object is calculated.
The actual depth image is acquired based on the depth information, and the shape information regarding the three-dimensional shape of the object stored in advance is converted and converted based on the positional relationship of the sensor unit with respect to the object. A shape depth image is acquired based on the shape information obtained, and the shape depth image is acquired.
Difference information, which is the difference between the actual depth image and the shape depth image , is calculated.
The calculated difference information is converted based on the positional relationship of the display unit with respect to the object object, superimposed on the real image of the object object or the captured image of the object object, and displayed on the display unit .
Information processing methods including that. The function to acquire the depth information of the object in the real space from the sensor part to the computer,
A function for calculating the positional relationship of the sensor unit with respect to the object in the real space and the positional relationship information indicating the positional relationship of the display unit with respect to the object .
An actual depth image is acquired based on the depth information, and further, the shape information regarding the three-dimensional shape of the object stored in advance is converted and converted based on the positional relationship of the sensor unit with respect to the object. A function to acquire a shape depth image based on the shape information provided, and
A function to calculate difference information which is a difference between the actual depth image and the shape depth image , and
A function of converting the calculated difference information based on the positional relationship of the display unit with respect to the object, superimposing it on the real image of the object or the captured image of the object, and displaying it on the display unit. When,
A program that realizes.

Images