JP2013250479A - Information display device, program, and information display recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information display device such that an instruction can be confirmed without interrupting an operation.SOLUTION: There is provided an information display device which analyzes an image photographed by a camera 7 to detect a predetermined target object 16, and projects some operation instruction and auxiliary information for the target object 16 by a projector 8, the information display device including object extracting means 4 of extracting a target object image from the camera image, object image saving means 5 of saving the extracted target object 16, composite image generating means 6 of putting a projection image of the operation instruction and auxiliary information together with the saved target object image, and composite image display means of displaying the composite image 19 at a different place from the position on a projection screen of the projector 8 where the operation instruction and auxiliary information are displayed.

Description

  The present invention relates to an information display device, a program, and an information recording medium for displaying a projected image projected on a display screen and a synthesized image obtained by synthesizing the projected image.

  It is considered to combine a camera and a projector, capture the situation with the camera, and issue some instruction with the projector.

  For example, when dropping three fish in a dish, a fish object is extracted from an image acquired by a camera, a position to be cut is searched from a knowledge base on cooking, and an instruction line to be cut on the fish is projected. . Since the operator cuts according to the instruction line, the operation can be performed without knowledge of the user.

  In addition, while viewing the camera image from a remote location, the instructor inputs information such as cut lines and parts to watch out for, and projects them to the operator with a projector. There is also a system that gives instructions in

  Further, Patent Document 1 discloses a projection display device that is improved in operability by arranging an image projection surface and a writing surface written by a user on the same screen, and is inexpensive.

  Since the operator is assumed to be away from the operation target (hereinafter referred to as a target object) before starting work, the operator can recognize the projection image such as the instruction line well.

  However, with the prior art and the technique disclosed in Patent Document 1, once the actual work is started, the projected image may be lost due to being disturbed by the shadow of one's hand or the like. For confirmation, it is necessary to stop the work one by one, but this is not only cumbersome, but also because there is a problem that the speed of operation is important in cooking and medical situations, so the work cannot be interrupted one by one.

  The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide an information display device capable of confirming an instruction without interrupting work.

  In order to achieve this object, the present invention has the following features.

  An information display device according to the present invention is an information display device that analyzes an image captured by a camera, detects a predetermined target object, and projects some operation instruction or auxiliary information to the target object by a projector. An object extracting unit for extracting the target object image from the camera image, an object image storing unit for storing the extracted target object, the stored target object image, the operation instruction and auxiliary information. A composite image generation unit configured to combine the projection image; and a composite image display unit configured to display the composite image at a location different from a position where the operation instruction or auxiliary information is projected within the projection plane of the projector. It is characterized by that.

  According to the present invention, the operation instruction and the auxiliary information are projected as an image obtained by synthesizing the target object image and the projection image at a place different from the place where the instruction line or the like is projected. Even if the original operation instruction or auxiliary information becomes invisible due to a shadow such as, it is possible to proceed with the operation while confirming the projected composite image with the side eyes.

It is a figure which shows the structure of the information display apparatus which concerns on 1st Embodiment. It is a processing flow figure in a 1st embodiment. It is a figure which shows the example of instruction information. It is the schematic of an operation assistance system. It is a figure which shows the visual field of the first operator. It is a figure which shows the image of the projected instruction | indication line. It is a figure which shows an operator's visual field when an instruction line is projected. It is a figure which shows the projection image of this embodiment. It is a figure which shows an operator's visual field at the time of projecting a synthesized image. It is a figure which shows the structure of the information display apparatus which concerns on 2nd Embodiment. It is a processing flow figure in a 2nd embodiment. It is a figure which shows the image of the projection image with which the instrument object was synthesize | combined. It is a figure which shows the structure of the information display apparatus which concerns on 3rd Embodiment. It is a flowchart by the side of the operator in 3rd Embodiment. It is a flowchart of the instruction | indication side in 3rd Embodiment.

  Hereinafter, the present embodiment will be described in detail with reference to the drawings.

  FIG. 1 is a diagram illustrating the configuration of the information display device according to the first embodiment. The information display device includes a UI 1, a CPU 2, a memory 3, a target object extraction unit 4, an object storage unit 5, a composite image generation unit 6, a camera 7, a projector 8, an operation instruction holding unit 9, a projection position determination unit 10, and a microphone 11. The speaker 12 and the HDD 13 are included.

  The UI 1 is a part for inputting an operator's instruction in the user interface unit. Since the operator may be out of hand, an input using voice recognition by inputting the voice of the operator with the microphone 11 is also conceivable. The user may consider an input for selecting an “operation menu” for which instruction / assistance is desired. In the case of a dedicated system for a predetermined operation, such a selection menu is unnecessary.

  The CPU 2 is a part that controls the entire system. The memory 3 is used as a program, a temporary work buffer, and the like. An image signal or the like input from the camera 7 once enters here.

  The target object extraction unit 4 analyzes the image signal on the memory 3 input from the camera 7 and extracts an operation target object. The object storage unit 5 stores the target object image extracted by the target object extraction unit 4. An image input from the camera image once enters the memory 3, where an object is extracted. After the extraction, the image of the object part is stored in another storage memory or HDD 13 from the memory 3.

  The composite image generation unit 6 stores the stored target object image and information on operation instructions (such as a cutting auxiliary line) in the memory 3 as a composite image.

  The camera 7 is an image input device. The projector 8 is a projection device. The operation instruction is projected on the target object, and at the same time, the composite image created by the composite image generation unit 6 is projected at the position determined by the projection position determination unit 10.

  The operation instruction holding unit 9 holds what operation instructions and auxiliary information are projected to the user. As the storage device, the memory 3 and the HDD 13 are explicitly shown. The projection position determination unit 10 determines a position where the composite image is projected. For example, deciding the upper right corner of the screen.

  The microphone 11 inputs an operator's voice and is attached with a voice recognition device. The speaker 12 issues a voice instruction to the operator. The HDD 13 is a storage device.

  FIG. 2 is a processing flow diagram according to the first embodiment. An image is input from the camera 7 (step S1), and the target object is extracted from the image by the target object extraction unit 4 (step S2).

  When the target object can be extracted (step S3, Yes), the target object image is stored in the object storage unit 5 (step S4), and the operation instruction information and the target object image are combined by the composite image generation unit 6 (step S4). S5) A composite image is projected (step S6). If the target object could not be extracted (step S3, No), an end check is performed (step S7), and the process returns to image input from the camera.

  Many techniques have already been proposed for extracting objects from images, and there are also provisions using OSS (open source software) and the like. The instruction information holds what figure or character is to be written at which position of the target object, and the target object and the operation instruction information are synthesized based on the information.

  FIG. 3 is a diagram illustrating an example of instruction information. The procedure indicates at which operation stage it is presented. An instruction attribute 1 represents an instruction form. “Line” represents an instruction by line image display for the target object, “Audio” represents an instruction by voice, and “Text” represents an instruction by character display. The attribute 2 is each detailed instruction information.

  First, the line in step 1 is projected onto the target object. Also, the target object image and the line are combined and projected onto another position on the projection screen.

  FIG. 4 is a schematic diagram of the operation assistance system. The camera 7 and the projector 8 are positioned above the work table 15, and the camera 7 inputs an image of the work table 15 to the operation assisting system main body 14. The operation assisting system main body 14 extracts the target object 16 and projects an instruction line or the like on the target object 16 by the projector 8. The operator 17 can visually recognize an instruction line or the like on the target object 16. Further, a microphone 11 and a speaker 12 are connected to the operation assistance system main body 14.

  FIG. 5 is a diagram showing the field of view of the first operator 17. This image is taken by the camera 7, the target object 16 on the work table 15 is extracted, and an instruction line for cutting is projected by the projector 8.

  FIG. 6 is a diagram showing an image of the projected instruction line 18. It can be confirmed that the instruction line is projected on the work table 15.

  FIG. 7 is a diagram illustrating the field of view of the operator 17 when the instruction line 18 is projected. In the field of view of the operator 17, it can be confirmed that the instruction line 18 is placed on the target object 16.

  FIG. 8 is a diagram showing a projected image 19 of the present invention. In addition to the instruction line 18 shown in FIG. 6, a composite image 19 of the target object, the instruction line, and the character image “Please indicate“ next ”when cut” is projected on the upper right of the screen.

  FIG. 9 is a diagram illustrating the field of view of the operator 17 when the composite image 19 is projected. A composite image 19 is projected on the upper right of the target object 16 on which the instruction line 18 is placed.

  Returning to the description of FIG. 3, Audio of attribute 1 gives a voice instruction through the speaker 12. In this example, “Please cut along the auxiliary line” at the time of step 1, and “Turn it over” at the time of step 2. Similarly to Line, Text of attribute 1 projects a character image “Please indicate“ next ”when cut” to the vicinity of the object image. When the operator 17 utters “next”, the system knows that the procedure 1 has been completed by voice recognition via the microphone 11, and moves to the procedure 2 instruction.

  In this example, the end of the procedure is detected by the operator 17 uttering, but it is also conceivable to determine automatically based on the state of the target object 16.

  According to the present embodiment, the operation instruction, auxiliary information, and the like are projected on another place as a composite image of the target object image and the projection image, so that the operator can perform the original operation with the shadow of his / her hand or the like. Even if the instruction or the auxiliary information becomes invisible, the operation can be performed while confirming the projected composite image with the side eyes.

  FIG. 10 is a diagram illustrating a configuration of an information display device according to the second embodiment. The same constituent elements as those in the first embodiment are denoted by the same reference numerals, and description thereof is omitted.

  The information display device includes a UI 1, a CPU 2, a memory 3, an object extraction unit 21, an object dictionary 22, an object storage unit 5, an object positional relationship extraction unit 23, a composite image generation unit 6, a camera 7, a projector 8, and an operation instruction dictionary 24. The projection position determination unit 10, the color correction unit 25, the HDD 13, and the external IF 26 are configured.

  The object extraction unit 21 analyzes the image signal on the memory 3 input from the camera 7 and performs object extraction. What is extracted is determined according to the selected “operation menu”, and the feature amount is read from the object dictionary.

  The object dictionary 22 stores feature quantities of target objects and appliance objects. As the storage device, the memory 3 and the HDD 13 are explicitly shown.

  The object storage unit 5 stores images of target objects and appliance objects extracted by the object extraction unit. An image input from the camera image once enters the memory 3, where an object is extracted. After the extraction, the image of the object portion is stored in another storage memory 3 or HDD 13 from the memory 3.

  The object positional relationship extraction unit 23 extracts the positional relationship between the extracted target object and the appliance object. A relative positional relationship is extracted from the detection position of the target object and the detection position of the instrument object.

  The synthesized image generating unit 6 synthesizes the object image so as to maintain the extracted object positional relationship with the stored object image, and further synthesizes information related to the operation instruction (such as an auxiliary cutting line) to store the memory 3. Save to.

  The operation instruction dictionary 24 is determined according to the selected “operation menu”. It stores what operation instructions and auxiliary information are projected to the user. The storage devices are the memory 3 and the HDD 13 but are explicitly shown.

  The projection position determination unit 10 determines a position where the composite image is projected. The position of the extracted target object is avoided. Further, for example, the operator's position is also extracted, and a white surface as much as possible is selected other than the line connecting the position of the target object and the operator's position.

  The color correction unit 25 acquires an image at a position where the composite image is projected from the camera 7 and performs correction so as to approach the original color when the composite image is projected. The external IF 26 is connected to the network 27.

  FIG. 11 is a processing flow diagram according to the second embodiment. This flow is for one procedure in FIG. First, instruction information related to the procedure is acquired (step S11). And an image is acquired from the camera 7 (step S12), and a target object is extracted by the object extraction part 21 (step S13). If the target object cannot be extracted (No in step S14), the end check is performed (step S25), and the process returns to the camera input.

  If it can be extracted (step S14, Yes), the target object image is stored in the object storage unit 5 (step S15). Next, an appliance object is extracted by the object extraction unit 21 (step S16). If the appliance object can be extracted (step S17, Yes), it is checked whether the appliance object image has been saved, and if not saved (step S18, No), the appliance object image is saved in the object storage unit 5 (step S18, No). Step S19).

  Further, the positional relationship between the target object and the appliance object is extracted by the object positional relationship extraction unit 23 (step S20). And according to the positional relationship extracted above, a target object and an instrument object synthesize | combine both by the synthetic | combination image generation part 6, and also synthesize | combine instruction information (step S21).

  If the appliance object has not been extracted (No at Step S17), only the target object and the instruction information are synthesized by the synthesized image generation unit 6 (Step S21). Next, the projection position of the composite image is determined by the projection position determination unit 10 (step S22), the color correction of the projection image is performed from the projection position by the color correction unit 25 (step S23), and the composite image is projected (step S24). ).

  According to the present embodiment, since the instrument object is also projected as a composite image, the operator can confirm the positional relationship between the target object and the instrument object, so that confirmation with higher accuracy is possible.

  FIG. 12 is a diagram showing an image of the projection image 29 in which the instrument object 28 is synthesized. Color correction is to bring a color component that can be acquired from a camera input image at a projection position from a color component of an image to be projected and then project it so that it is close to the primary color.

  Since the composite image is projected after subtracting the color component at the projection position, the color of the composite image becomes close to the primary color, and the operator can check the target / appliance object, the operation instruction, auxiliary information, and the like more accurately.

  Saving an object image does not necessarily have to be linked with object extraction. Even if it can be extracted as an object, the object image may be partially hidden by the operator's hand. If it is stored as the target object image as it is, the synthesized image itself can be an image in which a part is hidden. Therefore, it is preferable to store the image only when the object can be extracted and the entire image of the object can be seen or hidden can be restored from the immediately preceding object storage image.

  Furthermore, it is also conceivable to replace only the part that has changed on the surface of the target object. By doing so, even if the object is partially hidden by the operator, it is possible to update the part where the object has changed to the latest state. Since the target object image is updated as appropriate, the target object can be confirmed with the latest processing.

  Since the instrument object 28 is likely to be held during operation, there is a possibility that a complete image of the instrument object 28 cannot be acquired. Therefore, it is conceivable that the system prompts the operator to start the operation after having the camera 7 take a picture of the device alone before the operator enters the operation.

  FIG. 13 is a diagram illustrating a configuration of an information display device according to the third embodiment. Since the amount of projection image generation processing increases from object extraction, it is conceivable that the operator side is configured only by the camera 7 and the projector 8 and the other processing is left to an external server. The operator side and the instruction side are connected via the network 27, and the operator side transmits a camera image to the instruction side, and the instruction side performs a series of object extraction, generation of a composite image, determination of a projection position, and color correction. Then, the projection image is sent back to the operator side and projected by the projector 8 on the operator side. The instructing side includes a display 31 and a UI 1, and in some cases, it may be possible for a person to intervene to give an instruction.

  The operator side of the information display device includes a UI 1, a CPU 2, a memory 3, a camera 7, a projector 8, an image encoder 32, an image decoder 33, and an external IF 26.

  The instruction side of the information display device includes UI1, CPU2, memory 3, object extraction unit 21, object dictionary 22, composite image generation unit 6, HDD 13, external IF 26, display 31, operation instruction dictionary 24, projection position determination unit 10, and color. The correction unit 25 includes an image encoder 34 and an image decoder 35.

  New elements of this embodiment are image encoders 32 and 34 and image decoders 33 and 35. The same constituent elements as those in the first and second embodiments are denoted by the same reference numerals, and description thereof is omitted.

  The image encoders 32 and 34 are encoders for the operator side to compress the camera image and send it to the instruction side, and the instruction side is an encoder for sending the composite image to the operator side. The image decoders 33 and 35 are decoders for the operator side to decode and project the projection image sent from the instruction side, and the instruction side from the operator side to extract an object. It is a decoder that decodes a camera image that is sent.

  FIG. 14 is a flowchart on the operator side in the third embodiment. An image is acquired from the camera 7 (step S31). The captured image is encoded by the image encoder 32 (step S32). The captured image is transmitted to the instruction side (step S33), and the projection image is received from the instruction side (step S34). The captured image is decoded by the image decoder 33 (step S35) and projected (step S36). An end check is performed (step S37), and the process returns to camera input.

  FIG. 15 is a flowchart of the instruction side in the third embodiment. First, instruction information related to the procedure is acquired (step S41).

  The photographed image is received / decoded (step S42), and the object extraction unit 21 extracts the target object (step S43). If the target object cannot be extracted (No at Step S44), an end check is performed (Step S55), and the process returns to the camera input.

  If it can be extracted (step S44, Yes), the target object image is stored in the object storage unit 5 (step S45). Next, an appliance object is extracted by the object extraction unit 21 (step S46). If the appliance object can be extracted (step S47, Yes), it is checked whether the appliance object image has been saved, and if not saved (step S48, No), the appliance object image is saved in the object storage unit 5 (step S48, No). Step S49).

  Furthermore, the positional relationship between the target object and the appliance object is extracted by the object positional relationship extraction unit 23 (step S50). Then, in accordance with the positional relationship extracted above, the target object and the appliance object are combined by the combined image generation unit 6 and the instruction information is also combined (step S51).

  If the appliance object has not been extracted (No at Step S47), only the target object and the instruction information are synthesized by the synthesized image generation unit 6 (Step S51). Next, the projection position of the composite image is determined by the projection position determination unit 10 (step S52), the color correction of the projection image is performed by the color correction unit 25 from the projection position (step S53), and the composite image is projected (step S54). ).

  According to the present embodiment, even a terminal with low calculation performance (operator side) is limited to terminal performance constraints by performing calculations with a large processing amount on the external server side. More accurate instructions can be presented.

DESCRIPTION OF SYMBOLS 4 Target object extraction part 5 Object preservation | save part 6 Composite image generation part 7 Camera 8 Projector 10 Projection position determination part 16 Target object 19 Composite image 23 Object positional relationship extraction part 25 Color correction part 28 Instrument object

Japanese Patent No. 4037128

Claims (8)

An information display device that analyzes an image captured by a camera, detects a predetermined target object, and projects some operation instruction or auxiliary information on the target object by a projector,
Object extraction means for extracting the target object image from the camera image;
Object image storage means for storing the extracted target object;
A synthesized image generating means for synthesizing the stored target object image and the projection image of the operation instruction and auxiliary information;
Composite image display means for displaying the composite image at a location different from the position where the operation instructions and auxiliary information are projected on the projection plane of the projector;
An information display device comprising: The object image storage means for extracting and storing an appliance object handled by an operator from the camera image;
Object positional relationship acquisition means for acquiring a positional relationship between the instrument object and the target object from the camera image;
The synthesized image generating means for synthesizing the target object, the appliance object, and a projection image of the operation instruction and auxiliary information;
The information display device according to claim 1, further comprising:   The information display apparatus according to claim 1, further comprising a projection position determining unit that projects the composite image at a position avoiding the target object.   The information display apparatus according to claim 1, wherein the object image storage unit updates the stored image every time the target object can be extracted from the camera image.   5. The information according to claim 3, further comprising color correction means for subtracting a color component obtainable from a camera input image at the projection position determined by the projection position determination means from a color component of a composite image to be projected. Display device. A client machine having the camera, the projector, and the composite image display means;
A server machine having the object extraction means, the object image storage means, the composite image generation means, the object positional relationship acquisition means, the projection position determination means, and the color correction means;
The information display device according to claim 1, wherein the two are connected via a network. Analyzing a camera-captured image, detecting a predetermined target object, and projecting any operation instruction or auxiliary information to the target object with a projector;
Extracting the target object image from the camera image;
Storing the extracted target object;
Combining the stored target object image with the projection image of the operation instruction and auxiliary information;
Displaying the composite image at a location different from the position where the operation instructions and auxiliary information are projected within the projection plane of the projector;
A program that causes a computer to execute.   An information recording medium on which the program according to claim 7 is recorded.

Images