JP2012038025A - Display device, control method for display device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display device that is operated in front of a display screen, and is capable of displaying without hindrance for an operation or browsing even if a dead angle area occurs on a display by operation.SOLUTION: A display device directs to an item on a display screen with directing means in front of the display screen; detects a dead angle area hiding the display screen with the directing means (STEP 160); and corrects and displays the display screen in accordance with positional relations of the detected dead angle area and the item (STEP 170).

Description

  The present invention relates to a display device that operates on the front surface of a display screen, a control method for the display device, and a program.

  2. Description of the Related Art In an information processing apparatus represented by a personal computer, a keyboard and a mouse are provided as conventional devices for inputting an operation by an operator, and a display such as a liquid crystal monitor is provided as a device from which an operator receives information. On the other hand, many forms in which no keyboard or mouse is used to operate the apparatus have been proposed.

  One specific example is a touch panel system. A touch panel is a device in which a transparent tactile sensor (also called a touch sensor) is superimposed on the surface of a conventional display. At first glance, it is similar to a conventional display, but is characterized by being an information display device and an input device. When the operator touches the display with his / her finger or a stick-shaped instrument, the position on the display touched by the finger or instrument is detected by the touch sensor. That is, instead of designating a position on the display with a mouse which is a conventional pointing device, an operator's finger or the like becomes a direct pointing device.

  As a result, it is possible to operate by directly touching the contents displayed on the display, so that it can be expected that intuitive operation can be performed in combination with a graphical user interface. In addition, since the display device and the input device are integrated in the touch panel, user interfaces such as buttons and options suitable for the contents to be operated can be freely controlled and presented by software. As an application example, a touch panel system is adopted in many bank ATMs. On-screen operations and character input keyboards are dynamically displayed, enabling intuitive operation without getting lost. As a touch sensor system that realizes an operation touch panel, a form using a mechanical switch that is deformed and touched by a touch pressure (Patent Document 1), or a human body has electrical conductivity on the surface of the touch sensor. There is a method of observing changes in the target state. In recent years, the latter method occupies almost all.

  Another specific example of a form that does not use a special device for operation is gesture recognition. Gesture recognition refers to accepting input of information corresponding to a state by acquiring the movement and shape of the operator's hand or body and recognizing the state. In many cases, the operator does not need to have a special device and does not need to touch the display like a touch panel. In addition to pointing like a mouse, it is also possible to input a plurality of types of operations according to a plurality of shapes and states that can be taken by the hand. For example, if an operation is performed to reach and grasp an object displayed on the display, it is possible to realize that the object moves when the hand is moved as if the object is grasped.

  As an application example, in addition to operating a personal computer, use of gesture recognition instead of a controller having a large number of buttons in the field of video games has been studied. There are several ways to acquire the hand state to recognize a gesture. There is a method (Non-Patent Document 1) for performing image processing on a two-dimensional image captured by a widely available camera in the visible light region. Also, a method of acquiring and processing a three-dimensional shape by actively emitting infrared rays toward the subject, or a method of supplementing the information of the three-dimensional shape by actively emitting ultrasonic waves toward the subject together with image processing. and so on.

  Gesture recognition enables three-dimensional operation and input using depth, which cannot be achieved with a mouse or touch panel. An example of the operation described in the previous example is to reach out and grasp an item. On the other hand, a three-dimensional display capable of three-dimensional display having a depth is proposed for a display device instead of displaying on a conventional plane. A three-dimensional display is a display that presents an image capable of perceiving depth to a viewer, and several methods have been proposed as a method for realizing it.

  One method is to actually arrange the light emitting points on a three-dimensional space. There are a method using a volume type display that actually distributes light emitting points in a three-dimensional space by applying light to a rotating object, and a method for distributing light generated by burning air with a laser in a three-dimensional space. Can be mentioned. The other is a method in which a right-eye and a left-eye are different, that is, an image of a plane with parallax is shown, and a human being viewed perceives a three-dimensional image by complementing the parallax in the brain. An example of this method is a method in which depth can be perceived by looking through special glasses. For example, there is a method in which a head-mounted display having two independent small displays is attached to each of the left and right eyes, and the depth can be perceived by watching what displays different images on the left and right.

  In addition, there is a method (Patent Document 2) for viewing an image displayed on a flat display through a special lens. On the other hand, there are some that do not use special glasses. There is a method in which minute barriers with a pixel size are arranged on the surface of a flat display at regular intervals so that different images can enter the right eye and the left eye. In recent years, application examples of 3D displays have been actively proposed in the fields of television and movies, and many of them are of a type that is viewed through special glasses.

Japanese Patent No. 026889900 Japanese Patent No. 02666055

Hiroshi Yamada, Nobutaka Shimada, Yoshiaki Shirai: Hand shape recognition for sign language recognition by image processing: 2009 IEICE General Conference, A_19 — 001, pp. 331, 2009

  In an example where an operator inputs information using a touch panel or gesture recognition, when operating on a specific item displayed on the graphical user interface, a method of extending the hand toward the display is considered. It is done. The touch panel needs to touch the area on the display where the item is displayed. In gesture recognition, it is necessary to reach out to the direction in which the item is displayed as seen from the operator and to perform some kind of operation.

  However, at that time, the user's hand is applied to the display, and some of the displayed items may be hidden. As a specific example, when a right-handed operator extends his hand to the upper left of the display, his hand covers the right side of the display and a blind spot is created. At this time, if there is an important item in the blind spot area, the operator may lose sight of it and the operation may be hindered. In the case of using a mouse which is a conventional pointing device, the body of the mouse or hand is located away from the display. Only the cursor indicating the pointing position is displayed on the display, and such a phenomenon that the display is hidden does not occur.

  In other words, the problem is that the operation is hindered in the blind spot region that is created when the hand operated by the operator extends on the display. Further, when there is a person browsing the display separately from the operator, a plurality of different viewpoints are viewing the display. Even if the operator's operation does not generate a blind spot area that causes a problem for the operator himself, there is a possibility that a blind spot area that hides an important item for the viewer may be generated. That is, it is also a problem for a person other than the operator to create a blind spot area on the display depending on the viewpoint, which may be an obstacle.

  The present invention has been made in view of the above problems, and its object is to prevent operation and browsing even if a blind spot area is created on the display by the operation in a display device operated on the front surface of the display screen. It is to enable display that should not be.

In order to achieve the above object, a display device according to one embodiment of the present invention includes the following arrangement. That is,
A display device for instructing an item on a display screen by an instruction unit in front of the display screen, the display unit displaying the item on the display screen, and a blind spot that hides the display screen by the instruction unit Detecting means for detecting an area; blind spot area detected by the detecting means; and changing means for changing the display of the item displayed by the display means from the positional relationship of the items displayed by the display means; Is provided.

A display device control method according to another aspect of the present invention for achieving the above object has the following configuration. That is,
A display device control method for instructing an item on the display screen in front of a display screen by an instruction unit, wherein the display unit displays the item on the display screen; A detecting step in which the blind spot area detecting means detects a blind spot area that is hidden by the display screen, a blind spot area detected by the detecting means, and a positional relationship between the items displayed in the display step, the changing means for changing the display screen. A change process for changing and displaying.

  According to the present invention, in a display device operated on the front surface of a display screen, it is possible to perform a display that does not hinder operation or browsing even if a blind spot region is created on the display by the operation.

It is a figure which shows the structure of the display apparatus 1000 and the means to instruct | instruct to it. It is a flowchart which shows the flow of a process. It is a figure which shows the blind spot area | region when it sees from an operator's viewpoint. FIG. 11 is a state transition diagram of items on the display unit 1100. It is a figure which shows the example of operation on a graphical user interface. It is a figure which shows the example of operation on a graphical user interface. It is a figure which shows control of the graphical user interface based on a blind spot area | region. It is a figure which shows the example of operation on a graphical user interface. It is a figure which shows control of the graphical user interface based on the blind spot area | region of a viewer viewpoint. It is a figure which shows the display apparatus 3000. FIG. It is a flowchart which shows the flow of a process. It is a figure which shows the blind spot area | region approximately detected about the operator's viewpoint.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Example 1
In the present embodiment, an example will be described in which the present technique is applied to a desktop personal computer when an operator operates a graphical user interface by recognizing a hand gesture.

<Overall configuration>
FIG. 1 is a diagram showing a configuration of a display device 1000 to which the present technique is applied and means for instructing the display device 1000. As shown in FIG. 1, the display device 1000 includes a display unit 1100 (also referred to as a display screen) that displays information input / output with the operator, and an imaging unit 1200 that acquires an instruction from the operator. Further, a control unit 1300 that controls the graphical user interface and the operating system is provided separately from the display unit 1100 and is connected to the display unit 1100. Although not constituting the display device 1000, an instruction means 1400 which is an operator's hand giving an instruction to the display device is illustrated.

<Display device 1000>
The display apparatus 1000 does not include a mouse as a pointing device included in a typical personal computer and a keyboard as a character input device. The operator substitutes the function of the mouse by moving the instruction means 1400 on the front of the display screen and inputting a gesture operation. There are several examples of how the operator can enter characters as an alternative to the keyboard. Typically, the same image as a conventional keyboard is displayed on the display unit 1100, and the operator can input characters by performing a gesture operation of hitting a key on the image. As another method, there is a method in which a character input by an operator is uttered and the display device 1000 recognizes it and converts it into a character.

  A typical personal computer displays on the display screen an image called a cursor indicating the current pointing position on the display. In this embodiment as well, it is desirable to display a cursor on the display unit 1100 in the same manner. The display apparatus 1000 according to the present embodiment does not include a mouse as a pointing device, but recognizes and tracks the position of the instruction unit 1400 that is an operator's hand from an image obtained by the imaging unit 1200, and moves the cursor according to the movement. Move. However, simply tracking the position of the instruction means 1400 and reflecting the corresponding movement on the cursor causes the cursor to move even when the operator does not move the cursor.

  Specifically, it is when the operator finishes some operation and tries to return his hand. Therefore, it is desirable to recognize that the instruction unit 1400 has a certain shape and reflect the movement of the instruction unit 1400 on the cursor only in this state. For example, it is preferable that the instruction unit 1400 has a shape that is pointed at. As a technique for detecting the position of the instruction unit 1400 from the image obtained from the imaging unit 1200 and recognizing the shape, a known technique is used. As a method for detecting a human hand from within an image, there is “Viola & Jones (2001)“ Rapid Object Detection using a Boosted Cascade of Simple Features ”, Computer Vision and Pattern Recognition. Details are described in the literature, but the outline is as follows.

  This method combines multiple weak classifiers that can determine whether a given image is a hand or not in two classes with slightly better accuracy than random, and accurately determines whether a given image is a hand or not. This is a method of constructing a strong classifier that can be classified into two classes. The method of combining weak classifiers is not performed manually but automatically. This is a method for searching a combination of weak discriminators that can discriminate well about teacher data by using a group of images that have been discriminated manually or not in advance as teacher data. As a detection result, if a hand is shown in the image obtained from the imaging unit 1200, the partial region is detected as a hand.

  Subsequently, a technique such as Non-Patent Document 1 may be used as a technique for recognizing the shape of the hand based on a partial region of the image in which the detected hand is captured. Details are described in the literature, but the outline is as follows. This is a method for preparing as many strong discriminators as described in the previous document for the number of shapes that have meaning as gestures and can be taken by the instruction means 1400. One of these strong classifier groups is to discriminate into two classes, for example, whether the image of a given hand is a pointing shape or a hand shape other than that. For a given image, each strong discriminator discriminates the shape of the hand and integrates the results to recognize the hand shape. That is the summary.

<Display unit 1100>
The display unit 1100 is placed at a fixed position and presents an image to the operator. Typically, the display unit 1100 is a liquid crystal display with a diagonal of about 20 inches and the like that can secure a certain work area with a resolution sufficient for human recognition. Is good. In this embodiment, an example will be described in which the display unit 1100 is a display that allows an operator to perceive a three-dimensional image with depth by wearing the naked eye or special glasses and viewing the image. However, as will be described later, the display unit 1100 does not have to be a display for projecting a three-dimensional image, and may be a display for projecting a typical two-dimensional image. In this embodiment, as long as the operator can perceive a deep three-dimensional image by wearing the naked eye or special glasses and watching the image, the size and method are not limited.

  As a method for presenting a three-dimensional image, a known method as shown in Patent Document 2 is used. Details are described in the literature, but the outline is as follows. Similar to a typical 2D video display, this display is configured by arranging minute pixels at equal intervals in the vertical and horizontal directions. The difference is that on the surface of the display pixel, a barrier with a width equal to that of the pixel and a slit arranged vertically are arranged, and the vertical line of the video to enter the left eye and the vertical line of the video to enter the right eye are displayed on the display. It is a method of displaying alternately. With the right eye, the image that should enter the right eye can be seen through the slit, and the image that should enter the left eye is blocked by a barrier and cannot be seen. Therefore, only the video that should enter the right eye can be seen with the right eye. Since the left eye also looks the same, the operator sees an image with parallax that is different between the right eye and the left eye. Thus, the operator can interpret the parallax with the brain and perceive that the visible image has a three-dimensional depth.

  However, if a slit is simply provided on the display surface, there is a problem that when the operator's viewpoint moves to the left or right, the positional relationship between the image and the slit shifts and the appearance changes. Therefore, it is known that this problem can be alleviated by installing a long and narrow kamaboko-shaped convex lens called a lenticular lens in front of a pixel.

<Imaging unit 1200>
The imaging unit 1200 is typically a device that obtains an optical image, such as a CCD camera, and is preferably installed at a position where the operator's head, the instruction unit 1400, and the trunk are in the field of view of the imaging device. . In this embodiment, as shown in FIG. 1, the display unit 1100 is installed facing the front. The viewing angle of the imaging unit may be appropriately selected according to the distance that the operator gives an instruction with a gesture. If the position where the operator performs the gesture operation is extremely close to the display device 1000, it may be out of the field of view. In that case, it is necessary to secure a viewing angle by using a fish-eye lens. In this embodiment, two imaging units are provided to acquire the operator's viewpoint and the position of the instruction unit 1400 in the three-dimensional space.

  However, since at least two positions are required to acquire the position in the three-dimensional space, the position can be acquired with high accuracy if more image capturing units are provided. It is not desirable to install the plurality of imaging units 1200 together at close positions. As will be described later, parallax is not generated in the images obtained from the respective imaging units, and it is difficult to detect the three-dimensional position. However, as will be described later in the embodiment, when the display device 1000 is used in a situation where the position of the operator's hand or the viewpoint is restricted and it is easy to detect the three-dimensional position, conversely, only one imaging device is used. It is also possible to detect the three-dimensional position approximately by using it. Details of the method for detecting the operator's viewpoint and the position of the instruction means 1400 will be described later in this embodiment.

<Control unit 1300>
The control unit 1300 performs all computations of information input and output, and controls various processes to be described later with reference to flowcharts. These include execution of an operating system, implementation of a graphical user interface, recognition of gesture instructions from an operator, and control of items on the graphical user interface for a blind spot area created by the gesture operation. The control unit 1300 may typically exist as a program executed in a single dedicated circuit and general-purpose circuit (PC CPU), but the dedicated circuit (ASIC), processor (reconfigurable processor, DSP, CPU etc.).

  When present as a program, a memory including a program memory and a data memory is mounted. The program memory stores a program for control by the CPU for PC including various processing procedures to be described later with reference to flowcharts. This memory may be a ROM (Read-Only Memory) (not shown) or a RAM (Random Access Memory) (not shown) in which a program is loaded from an external storage device or the like. Or you may implement | achieve with these combination.

  Although FIG. 1 is described in the embodiment in which the display unit 1100 is stored in an exterior different from the display unit 1100 and connected to the display unit 1100, the place and method of installing the control unit 1300 are not limited. However, a place that hinders the display such as being installed between the operator and the display unit 1100 is not desirable.

<Instruction means 1400>
The instruction means 1400 is typically an operator's hand that operates the display device 1000. Typically, the operator extends his / her hand toward the display device 1000 at a position of several tens of centimeters or about 1 meter from the front of the display device 1000. A specific example of gesture operation and recognition will be described later. The instruction unit 1400 may be a bare hand, but may be operated with a marker serving as a recognition mark attached to the hand. For example, you can wear a glove with a colored marker that is easy to distinguish from the surrounding environment, or apply paint.

<Overall flow>
FIG. 2 is a flowchart showing an overall flow in which the display device 1000 detects a blind spot area created by an operation of the operator and controls the display content of the display unit 1100. All the processes shown in FIG. 2 are performed by processing an image obtained through the imaging unit 1200, and all the processes are performed by the control unit 1300. In addition to the series of processes shown in FIG. 2, the control unit 1300 also executes the original process of the operating system in parallel. When the process of FIG. 2 is operating, the original process of the operating system is stopped. is not. In addition, when the processing of FIG. 2 is stopped, the original processing of the operating system is working.

  First, the control unit 1300 obtains an image from the imaging unit 1200 and determines whether there is a person who operates the display device 1000 or a person who views the display device 1000 (STEP 110). This process will be described in detail later. If at least one person is detected (YES in STEP 120), it is determined whether a gesture instruction is currently issued from the operator, and the three-dimensional position of the gesture instruction is detected (STEP 130). This process will also be described in detail later.

  If a gesture instruction from the operator is detected in this process (YES in STEP 140), the process proceeds to a process for detecting a blind spot area. If NO in (STEP 120) or (STEP 140), the process proceeds to a process for controlling the display method of items on the graphical user interface without detecting the blind spot area (STEP 170). The processing of (STEP 170) will be described in detail later.

  Returning to the description (YES in STEP 140), the flow will be continued. First, for each person detected in the previous step (STEP 110), the viewpoint position, that is, the three-dimensional position of the human eye is detected (STEP 150). A detailed description of this process will be given separately. From the position of the instruction means obtained in (STEP 130) and the viewpoint position obtained in (STEP 150), an area where the instruction means covers the display unit 1100 when viewed from the viewpoint is detected (STEP 160). . Finally, control is performed to change the display method for items on the graphical user interface in the detected blind spot area. Here, a specific method for detecting the blind spot area and a description of how to specifically change the display method will be described in detail separately.

  The series of processing shown in FIG. 2 is not performed only once at a certain point of time. The situation of the operator is constantly changing, and the display apparatus 1000 needs to perform the process of FIG. 2 regularly and continuously. In order to respond to a quick gesture operation by the operator, the processing (STEP 110) to (STEP 170) shown in FIG. 2 is typically completed in a very short time, and repeatedly executed several tens of times in one second. It is desirable that

<Person detection processing>
The person detection process indicated by (STEP 110) in the flowchart of FIG. 2 will be described here. This process is performed by the control unit 1300, and the image obtained from the imaging unit 1200 is processed to detect the presence of a person and specify its three-dimensional position. The general flow is as follows. In this embodiment, since two imaging units 1200 are provided, two images can be obtained simultaneously. First, for each image, it is detected in which region in the image a person exists. Then, the detection results for the two images are matched, and the three-dimensional position where the person actually exists is specified. However, the purpose of this process is to detect the presence of a person who operates or views the display device 1000. Even if a person is detected, if the position is too far from the display device 1000, the person is not detected. Typically, the distance is set to several meters.

  In order to detect the presence of a person in one image obtained from the imaging unit 1200, a method of detecting the face of a person facing to some extent is preferable. This is because a method of detecting the entire human body from the image is conceivable, but a person who turns his back to the display device 1000 does not look at the display device 1000, and detection does not make sense. As a method for detecting a human face from an image, a known method is used. As an example, the same method as the method for detecting a hand described in the description of the display device 1000 can be used. As teacher data for determining the combination of weak classifiers, an image group that has been manually determined whether it is a face or not in advance may be used.

  A three-dimensional position with the detected face display unit 1100 as the origin is detected by matching the images respectively obtained from the two imaging units 1200 with the face detection results corresponding thereto. As the method, a stereo matching method of face detection results may be used. In the two images obtained by the two imaging units 1200, the positions where the face existing in a certain three-dimensional position appears in the images are different. This is because the positions where the two imaging units 1200 are installed are different and parallax is born. The three-dimensional position is detected using this parallax. If the positional relationship between the two imaging units 1200 is known in the control unit 1300, the point exists in the three-dimensional space from a combination of a point in the image in one image and a point in the image in the other image. The position can be determined. By utilizing this fact, it is possible to determine the position of the face where the point exists in the three-dimensional space for the face detection results corresponding to the two images.

  By the way, as a phenomenon that rarely occurs in the face detection method described above, there are undetected faces and false detections of non-faces. When an area detected in the image obtained from one imaging unit 1200 is not detected from an image obtained by the other imaging unit 1200, the three-dimensional position of the area cannot be specified. If the imaging unit 1200 is not fixed to the display device 1000 and is portable to some extent, the positional relationship between the two imaging units 1200 may not be known in the control unit 1300. In that case, a calibration process for storing the positional relationship between the two imaging units 1200 in advance in the control unit 1300 is necessary.

<Indication means position detection process>
The position detection processing of the instruction unit 1400 indicated by (STEP 130) in the flowchart of FIG. 2 will be described here. This process is performed by the control unit 1300, processes the image obtained from the imaging unit 1200, detects the presence of the instruction unit 1400, and specifies its three-dimensional position. The general flow is the same as that described in the <viewpoint position detection process>, and the three-dimensional position is specified from the position of the instruction means 1400 in the images obtained from the two imaging units 1200, respectively.

  As described in the description of <display device 1000>, if the operating system detects the position of the hand in advance in order to recognize the gesture, the position of instruction unit 1400 is detected again in the process of (STEP 130). do not have to. It is desirable to use the position of the instruction means 1400 that has been obtained in advance. However, here, in order to accurately estimate the blind spot area in the <dead spot area detection process> described later, it is necessary to detect the area where the hand is reflected in the image and the fingertip as accurately as possible. desirable.

  A known method is used as a method for extracting the hand region in the image. For example, as a simple example, all the pixels having a flesh color near the area detected as a hand are extracted as the hand area. A known method may be used for detecting the fingertip. As a simple example of this method, there is a method using a thinning algorithm generally used in image processing. A thinning process is performed on the hand region for a binary image in which the hand region and the other region in the image are separately painted in binary. Since the hand region is converted into a thin line like a bone, the tip position of the thin line may be used as a fingertip.

<Viewpoint position detection processing>
The viewpoint position detection process indicated by (STEP 150) in the flow of FIG. 2 will be described. This process is performed by the control unit 1300, and the image obtained from the imaging unit 1200 is processed to specify the three-dimensional position of the eye of the person who views the display device 1000. As described in the above description of <person detection processing>, an area in which a person's face is present is specified from the image obtained by the imaging unit 1200. As a method for accurately detecting the position of the eyes from the face region, a method using a convolutional neural network described in Japanese Patent No. 3078166 can be used.

  Alternatively, a method of approximating the face detection result without detecting the exact eye position is an option. Based on the assumption that the face is upright on the screen and the position of the eyes in the face is not different between individuals, a specific point in the area in the image detected as a face Position. Subsequently, the method of detecting the three-dimensional position of the eyes from the positions of the eyes obtained by the two imaging units 1200 is performed by the same method as the above-described <person detection process>.

<Detection process of blind spot area>
The blind spot area detection processing indicated by (STEP 160) in the flow of FIG. 2 will be described. This process is performed by the control unit 1300, and the three-dimensional position of the instruction unit 1400 obtained by the above-described <position detection process of the instruction unit> and the viewpoint position of the person obtained by the above-described <view point position detection process>. Based on one, the instruction means estimates a region that covers the display unit 1100. The position / size of the displayable area of the display unit 1100 and the three-dimensional position of the imaging unit 1200 are known. Further, the viewpoint position of the person and the position of the instruction means 1400 are detected from the <instruction means position detection process> and the <viewpoint position detection process> described above. From these positional relationships, the positional relationship between the display unit 1100 and the instruction unit 1400 when viewed from the viewpoint of a person is virtually estimated. Then, the indication unit 1400 can obtain a blind spot area that overlaps the displayable area of the display unit 1100. The blind spot area obtained here is specifically a part of the displayable area on the display unit 1100.

  FIG. 3 illustrates an example of a blind spot region when viewed from the operator's viewpoint. As a method for estimating the appearance of an object from a certain viewpoint used for the processing here, a perspective transformation method known in the fields of image processing and affine transformation may be used. When the blind spot area is estimated, it is desirable not to include the area where the vicinity of the fingertip of the instruction unit 1400 covers the display unit 1100 as the blind spot area. The reason is as follows. For one thing, the fingertips are thin and there are gaps between the fingers, which cannot be said to be a blind spot area that hinders operation and browsing. Details will be described later in <Display Content Control Processing>. However, if the display method of items on the display unit 1100 near the blind spot area of the fingertip is changed, the operation is difficult. For example, when an item that overlaps the blind spot area moves so as to escape from the blind spot area, even if it is attempted to select the item with the fingertip, it may not be touched indefinitely.

  When there is only one person's viewpoint detected in the <viewpoint position detection process>, the blind spot area when viewed from the viewpoint position may be estimated. However, in the <viewpoint position detection process> described above, when a plurality of persons are detected, it is necessary to take the following measures. As a typical example, it can be assumed that the plurality of persons are an operator who operates the display device 1000 and a viewer who browses the display device 1000 from the surroundings. When there are a plurality of persons around the display device 1000, it is difficult for the display device 1000 to determine which viewpoint should be prioritized to estimate the blind spot area. Therefore, it can be said that the operator who operates the display device 1000 should specify which person's viewpoint should be given priority.

  Specifically, it is preferable to provide a graphical user interface on the display unit 1100 to show a person present in the surrounding area and to allow the operator to select the person. It is desirable that this graphical user interface is not always displayed on the display unit 1100 but appears when the operator makes a predetermined gesture. If the graphical user interface is not called by the operator, the blind spot area may be automatically estimated by giving priority to the viewpoint of the person closest to the display device 1000. This is because the person is typically considered to be an operator.

<Control processing of display contents>
The display content control process indicated by (STEP 170) in the flow of FIG. 2 will be described. This process is performed by the control unit 1300. The display method of each item displayed on the display unit 1100 is changed based on the blind spot region on the display region of the display unit 1100 obtained in the above-described <dead zone detection process>. In the present embodiment, the specific items to be changed in the display method and the specific change in the appearance are as follows.

  The item whose display method is to be changed is an area on the display unit 1100 where the operator can perform operations such as generation, movement, selection, transformation, and deletion. In an operating system having a typical graphical user interface having a window system, this corresponds to a window, a file, a folder, an icon, etc. on the desktop. In addition, how to change the view specifically shows how to change the item so as to avoid the blind spot area, and how to change toward the original position when the blind spot area is farther away. I will explain later.

  First, a method in which the control unit 1300 changes the item display method will be described. The control unit 1300 is independent of each of the items displayed on the graphical user interface displayed on the display unit 1100 (hereinafter abbreviated as items on the display unit 1100 in this section). Manage which one of them.

  FIG. 4 shows the state of the initial display (STATE 110) in which items are displayed in the display method that should be. Furthermore, a state in which the display method is being changed by the blind spot area (STATE 120) and a state in which the display method is being restored to the original display method (STATE 130) are shown. Hereinafter, a method for changing the display method for one item will be described with reference to FIG.

  Before starting the explanation, add a supplement. First, in FIG. 2 and <Overall flow> described above, it has been described that the flow (STEP 110) to (STEP 170) is repeatedly executed at constant time intervals. Therefore, the display content control process (STEP 170) is repeatedly executed. Furthermore, it is described in advance that the control unit 1300 manages the state of each item existing on the display unit 1100 in parallel in (STEP 170). The supplement has been described above, and the description returns to FIG.

  The starting state of an item in FIG. 4 is that a new file or folder is created, typically a window is opened. When an item is generated on the display unit 1100 and the state is started, first, automatic transition is made to (STATE 110). Thereafter, when there is no operation by the operator (STEP 170) is executed, (STATE 110) self-transitions to (STATE 110). During this time, there is no change in the item display method. However, when the blind spot area is generated on the display unit 1100 by the operation of the operator and the item is immersed in the blind spot area, the state transitions to (STATE 120). In (STATE 120), as long as the item is in the blind spot area, each time (STEP 170) is executed, self-transition is made to (STATE 120). However, the difference from (STATE110) is that the item display method is not constant and is changed when in (STATE120).

  Specifically, when the state of the item transitions to (STATE 120), the item moves by a minute distance so as to exit the blind spot area. When (STEP 170) is repeatedly executed at a small time interval, the operator perceives the action of moving out of the blind spot area little by little like an animation. Only the minute change of the display method in one transition to (STATE 120) is because the operator moves the item greatly outside the blind spot area in one transition to (STATE 120), so that the operator moves the item instantaneously. This is for perception and confusion. The purpose is to notify the operator that the item is in the blind spot area and the display method is changing by making the operator perceive a motion that changes little by little like an animation instead of an instantaneous movement.

  Subsequently, when the movement is repeated until the item comes out of the blind spot area, or when the indication unit 1400 returns to the operator and the blind spot area on the display unit 1100 disappears, the state of the item transitions to (STATE 130). To do. In (STATE 130), each time (STEP 170) is executed, a self-transition is made to (STATE 130) unless the item is completely restored to the initial display method that should have existed. When in this state, the item display method is not constant and is changed as in (STATE 120). Specifically, when the state of the item transitions to (STATE 130), the item is moved by a minute distance so that the item is restored to the initial display position that should have existed. The reason why only the minute display method is changed in one transition to (STATE 130) is the same as in the case of (STATE 120).

  Finally, when the display method is restored in a state where the item is (STATE 130), the state automatically transitions to the initial state (STATE 110). In (STATE 120), the distance that the item moves in one movement does not always have to be constant. The distance by which the item moves may be increased according to the degree of immersion in the blind spot area, and the item may be removed from the blind spot area more quickly. Further, as a condition for transition from (STATE 110) to (STATE 120), it is also possible to make the condition that the blind spot area is approached to some extent, not the moment of being immersed in the blind spot area. By doing so, the display method can be changed before the item is immersed in the blind spot area.

  So far, it has been described how the control unit 1300 changes and controls the graphical user interface of the display device 1000 according to the blind spot area. Here, from the viewpoint of the operator, how the graphical user interface changes according to the blind spot area will be described with reference to the drawings. As a specific example, a case will be described in which a file is stored by dragging and dropping (grabbing, moving, and releasing) a file in a folder at a remote location on a graphical user interface provided in a typical operating system.

  FIG. 5A shows the display device 1000 in such a manner that the instruction unit 1400 tries to drag and drop the file 5 a located at the upper left of the display unit 1100 to the folder 5 b located at the lower right of the display unit 1100. It is shown from the viewpoint of the operator who operates. In addition, although the instruction | indication means 1400 in Fig.5 (a) is shown supposing the right hand of a right-handed person, it cannot be overemphasized that the effect of this case is not limited with respect to a right-handed person.

  FIG. 5B shows what can be seen from the operator's viewpoint when this operation is performed with a gesture by the instruction means 1400. As shown in FIG. 5B, if nothing is changed in the display method for the folder 5b, it can be understood that when the file 5a is selected by the instruction means 1400 by the gesture, the blind spot area covers the folder 5b.

  Therefore, in this embodiment, as shown in FIG. 5C, it is preferable to move items such as files and folders so as to avoid the blind spot area. In FIG. 5C, the instruction unit 1400 extends toward the file 5a, and the blind spot area also extends from the lower right of the display unit 1100. In response, items such as files and folders approaching the blind spot area move so as to avoid the blind spot area. When the instruction unit 1400 finishes dragging and dropping the desired file 5a to the folder 5b and then returns, the blind spot area on the display unit 1100 disappears. Accordingly, items such as files and folders that have been moved will return to their original positions.

  However, in this series of operations, the operation target file 5a and folder 5b must not move away from the blind spot area. If so, it cannot be selected and cannot be operated. As described in <Detection process of blind spot area>, it has been described that it is desirable not to detect the tip of instruction means 1400, that is, the fingertip as a blind spot area. This can solve the problem of escaping from the fingertip when trying to select the file 5a and the folder 5b. The vicinity of the fingertip that is ignored without considering it as a blind spot area depends on what items are changed in the graphical user interface and how the display method is changed.

  As another specific example, an example in which an image is dragged and dropped into a document being displayed / edited on a graphical user interface and inserted between appropriate lines will be described with reference to the drawings. FIG. 6A shows the display device 1000 in such a manner that the instruction unit 1400 tries to drag and drop the photograph 8a positioned on the upper left of the display unit 1100 between appropriate lines of the document 8b positioned on the right of the display unit 1100. It is shown from the viewpoint of the operator who operates.

  FIG. 6B shows what can be seen from the operator's viewpoint when this operation is performed with a gesture by the instruction means 1400. As shown in this figure, it is understood that if no display method is changed for the document 8b, the blind spot area covers the document 8b when the instruction unit 1400 selects the photograph 8a by the gesture.

  Therefore, when the instruction unit 1400 drags (moves) the photograph 8a and approaches the document 8b, when the state shown in FIG. 7A is reached, the document 8b is wrapped around to the left of the photograph 8a. Move to position. This movement is shown in FIG. With the arrangement as shown in FIG. 7B, the operator can insert the photograph 8a between appropriate rows without being blocked by the blind spot area. When the operator's drag-and-drop ends and the instruction means 1400 leaves the display unit 1100, it is desirable that the blind spot area disappears and the position of the document 8b returns.

  In these two specific examples, an example in which an item is moved is shown. However, the method used by the control unit 1300 to change the display method of the item can be applied only to a change in appearance as in this embodiment. I add that it is not. Even if you change the color and transparency of the item that overlaps the blind spot area, or change the color and transparency of the item that overlaps the blind spot area, you can notify the viewer or operator from another viewpoint, The object of the present invention can be achieved.

<When priority is given to the viewpoint from the viewer>
In the above two specific examples, the case where the display method of the items on the display unit 1100 is changed with respect to the blind spot region that is visible from the viewpoint of the operator who operates the display device 1000 is shown. As another case, a case where priority is given to the viewpoint of the viewer who browses the display device 1000 around the operator will be described. FIG. 8 shows the display unit 1100 viewed from the viewpoint of the operator in front of the display device 1000. In the figure, a document 12a, a photograph 12b, and a photograph 12c are shown. There is one viewer next to the operator, and the viewpoint of the viewer is indicated by a broken line arrow. Further, in the figure, when the instruction unit 1400 instructs a part of the display unit 1100, a region that cannot be seen from the viewer's viewpoint is shown as a black translucent region. If priority is given to the viewpoint of the viewer rather than the operator, it can be seen that if the display method of the photos 12b and 12c is not changed, the photo is blocked by the instruction means 1400 and cannot be seen.

  Therefore, as shown in FIG. 9, it is desirable to move the positions of the photos 12 b and 12 c so as to avoid the blind spot area for the viewer, which is shown in black translucent. In other words, the photograph 12b and the photograph 12c are not in the blind spot area for the operator who operates the display device 1000, but are in the blind spot area for the viewer who gives priority to the viewpoint. This is an example of moving a photograph 12c. In this example, an example is shown for a single viewer who is not an operator, but it is of course possible to give priority to the viewpoints of a plurality of viewers simultaneously. However, the desired result can be obtained by controlling the display contents when a plurality of viewers are close to each other and the distance from the display unit 1100 is sufficiently long. If so, the angle at which the viewer's viewpoint deviates from the front is substantially the same for each viewer, and the display device 1000 viewed from each viewer is substantially the same.

As a method for detecting the viewer's viewpoint, among the viewpoints detected by the <viewpoint position detection process>, it is automatically set that the distance from the display unit 1100 is the closest to the non-operator. desirable. Moreover, it is desirable that the user should be able to select a viewer that should be given priority as necessary.
Example 1 is as described above.

  As described above in the first embodiment, the present invention enables display that does not hinder operation or browsing even if a blind spot area is created on the display by the operation in the display device operated on the front surface of the display screen.

(Example 2)
In the first embodiment, parallax of images obtained from at least two or more imaging devices is used in order to detect a position in a three-dimensional space of an instruction unit that issues a viewpoint and a gesture instruction. On the other hand, in the present embodiment, an example in which the position of the instruction means or the viewpoint of a person is detected from one imaging apparatus will be described. The method will be described more specifically. The overall configuration of the display device 1000 in this embodiment is different from that of the first embodiment in that the imaging unit does not include two imaging devices. Instead, only one imaging device is provided. In addition, in order to avoid duplication, the same part as previous embodiment is abbreviate | omitted in the following description. Since the function of each part not described in the present embodiment is the same as that of the first embodiment, refer to the first embodiment for the description of each part.

<When the imaging unit 1200 includes a single imaging device>
There are two methods for detecting the position of the pointing means and the position of the viewpoint by the imaging unit 1200 configured by one imaging device. One is a method using a special camera that can acquire depth information up to an object instead of a typical CCD camera. The other is a method of approximately estimating using the assumption of the positional relationship with the detection target under limited conditions.

  The details will be described below in order. Since the imaging unit 1200 according to the present exemplary embodiment has only one, unlike the first exemplary embodiment, it is desirable to arrange the imaging unit 1200 at a position that is directly in front of the instruction unit to be detected and the face. In FIG. 1 of the first embodiment, the imaging units 1200 are installed on both sides of the display unit 1100, in order to increase the parallax of images obtained from both imaging units 1200.

  A camera that can acquire depth information up to an object as a first method for detecting the position of the instruction means or the position of the viewpoint may use a known technique when used for the imaging unit 1200. For example, there is a method called “Time of Flight”. This method uses a transmission unit that emits a light beam having a specific wavelength toward an object, and a reception unit that receives the light beam reflected back from the object. The receiving unit estimates the distance to the object from the time from when the light beam is emitted until it is reflected by the object and returned, and the phase of the light beam. The receiving unit is typically an image receiving element such as a CCD, and an infrared ray having a property closer to a radio wave is typically used as a light ray. In a typical image pickup apparatus such as a CCD camera, color information is obtained as the value of each pixel. The pixel value obtained by this method can be obtained in addition to the color, that is, the distance from the image pickup apparatus, that is, the depth.

  In this method, the resolution of the depth decreases as the distance of the object to be measured increases from the imaging device. Typically several meters are considered practical. The image obtained from the imaging unit 1200 in this embodiment is information obtained by adding a depth to a typical two-dimensional image obtained from the imaging unit 1200 of the first embodiment. Therefore, it is supplemented that techniques such as face detection from an image as described in the first embodiment can be used as they are.

  As a second method of detecting the position of the instruction means and the position of the viewpoint, a method using one typical CCD camera will be described. In the first embodiment, the imaging unit 1200 includes a plurality of imaging devices. It has been described that the viewpoint position and the position of the pointing means in the three-dimensional space are detected using the parallax of the video obtained from each imaging device. If the imaging unit 1200 is configured with a single imaging device as in the present embodiment, parallax of video cannot be obtained. Therefore, the three-dimensional position of the viewpoint and the instruction means cannot be obtained, and only the two-dimensional position from the imaging unit 1200 is obtained.

  However, in this embodiment, which is a desktop personal computer, in some cases, it is considered that the functions described in this embodiment can be maintained if limited to the operator. Specifically, this is a case where the operator is directly in front of the display device 1000, which is a natural position for the operator. If it is assumed that the size of the human face is constant regardless of the individual, the distance between the face and the display device 1000 is determined from the size of the face detected in the <person detection process> in the image obtained from the imaging device. Can estimate the operator's viewpoint position.

  As described above in the second embodiment, in the display device operated on the front surface of the display screen, even if a blind spot area is created on the display by the operation, a display that does not hinder the operation or browsing is enabled. Furthermore, the present invention can be realized with a simple configuration as compared with the first embodiment. The second embodiment is as described above.

(Example 3)
In the first and second embodiments, an embodiment has been described in which an operator recognizes a gesture operation by a hand and operates a personal computer typically used at a fixed position on a desk or the like. On the other hand, in the present embodiment, the display device is a portable device that can be carried, and the operator operates by touching a touch panel provided on the display device. The application example of this proposal in that case will be described more specifically. In addition, in order to avoid duplication, the same part as previous embodiment is abbreviate | omitted in the following description. Since the function of each part not described in the present embodiment is the same as that of the first embodiment, refer to the first embodiment for the description of each part.

<Overall configuration>
FIG. 10 shows a configuration of a display device 3000 to which the present technique is applied and means for instructing the display device 3000. As shown in the figure, the display device 3000 includes a display unit 3100 that displays information and a contact sensing unit 3200 that acquires an instruction from the operator. Note that the display unit 3100 is a touch panel and is integrated with the contact sensing unit 3200. Further, although not shown in the figure, a control unit 3300 for controlling the graphical user interface and the operating system is provided inside. Although not constituting the display device 3000, an instruction means 3400 which is an operator's hand giving an instruction to the display device is illustrated.

<Display device 3000>
The display device 3000 has a size that allows the operator to carry the display device 3000 in his / her hand. Typically, the display device 3000 is preferably about the size of a palm, that is, within a few tens of centimeters. The operator picks up the display device 3000 and operates with the other hand. The distance between the operator's viewpoint and the display device 3000 is typically several tens of centimeters, and basically, it is assumed that a person who views the display device 3000 is rare in addition to the person having the display device 3000. Good. A mouse as a pointing device and a keyboard as a character input device provided in a typical personal computer are not provided. The operator moves the instruction means 3400 and touches the display unit 3100 which is a touch panel to input a pointing operation, thereby substituting the mouse function. As a method for the operator to input characters as an alternative to the keyboard, the same video as the conventional keyboard is displayed on the display unit 1100 as in the first embodiment, and the operator performs a gesture operation by hitting a key on the video. This makes it possible to input characters. As another method, there is a method in which a character input by an operator is uttered and the display device 1000 recognizes it and converts it into a character.

<Display unit 3100 and contact sensing unit 3200>
The display unit 3100 is typically a planar liquid crystal display capable of displaying a two-dimensional image. Any device capable of presenting an image is not limited to a liquid crystal display, but it is difficult to realize that it can be carried using, for example, a CRT monitor. The display unit 3100 is a device that presents visual information to the operator, but is also a device that accepts the operation of the operator by being integrated with the contact sensing unit 3200. Specifically, the two devices can be integrated by attaching a contact sensing unit 3200 formed of a transparent thin film to the display unit 3100. Note that a known touch panel method may be used, and it is desirable to use one of the introduced techniques. The contact sensing unit 3200 senses that the instruction unit 3400 touches the display unit 3100 and the contact sensing unit 3200, and obtains position information indicating where the position is on the display unit 3100.

<Instruction means 3400>
The instruction unit 3400 is typically an operator's finger that operates the display device 3000. However, the present invention is not limited to a human finger and is not limited as long as the position can be specified by touching the touch panel. For example, an embodiment in which a pen-shaped pointing device is held in the hand and touches the touch panel is also conceivable.

<Overall flow>
FIG. 11 is a flowchart illustrating an overall flow in which the display device 3000 detects a blind spot region generated by an operation of the operator and controls the display content of the display unit 3100. Since there is a common part with FIG. 2 which shows the whole flow of Example 1, the part is omitted. In this embodiment, all the processing shown in FIG. 11 is performed by processing information obtained through the contact sensing unit 3200, and all the processing is performed by the control unit 3300.

  First, the control unit 3300 detects a position where the touch sensing unit 3200 is touching the display unit 3100 on the display unit 3100 (STEP 310). When contact with the instruction unit 3400 is detected (YES in STEP 320), the instruction unit 3400 detects a blind spot area that covers the display unit 3100 (STEP 330). If no in (STEP 320), the process moves to a process for controlling the display method of items on the graphical user interface without detecting the blind spot area (STEP 340). This (STEP 330) will be specifically described. The processing in (STEP 340) is omitted because the same method as in the first embodiment can be used. As in the first embodiment, the series of flows (STEP 310) to (STEP 340) shown in FIG. 11 are typically completed in a very short time, and are preferably repeatedly executed several tens of times in one second. .

<Detection process of blind spot area>
The blind spot area detection process indicated by (STEP 330) in the flow of FIG. 11 will be described. In the first embodiment, the display unit 1100, the instruction unit 1400, and the viewpoint of the operator or the viewer exist with a certain distance and degree of freedom in the three-dimensional space. Therefore, in order to detect the blind spot area, it is necessary to precisely detect the positional relationship between the display unit 1100, the instruction unit 1400, and the viewpoint of the operator or the viewer. In this embodiment, the blind spot area is obtained without detecting the position of the instruction means 3400 in the three-dimensional space and the position of the viewpoint in the three-dimensional space. This is because the relative positional relationship between the display unit 3100, the instruction unit 3400, and the operator's viewpoint has three restrictions.

  First, the positional relationship between the display unit 3100 and the instruction unit 3400 is shown. The two are very close to a few centimeters. As shown in FIG. 10, the instruction means 3400 touches the display unit 3100 when operating the touch panel. Further, the instruction unit 3400 extends from the lower right of the display unit 3100 when the operator's dominant hand is right, and from the lower left of the display unit 3100 toward the upper side of the display unit 3100 when the dominant hand is left.

  More typically, the operator can hold the display device 3000 in his / her hand and put it in front of his / her face for an easy posture with no burden. In addition, it is possible to limit which side of the display device 3000 the hand is held by the operator from the direction of characters and images displayed on the display unit 3100. Another constraint is the positional relationship between the operator's viewpoint and the display unit 3100. This is typically a distance of a few tens of centimeters. As long as the operation method of the display device 3000 is a touch panel, the two cannot be separated beyond the length of the pointing means 3400, that is, the length of the hand and arm or the pointing device. As a third restriction, since the area of the display unit 3100 is small and the operator holds it in his hand, it can be said that a plurality of people rarely see the display device 3000. That is, it can be limited that there is a viewpoint only for the operator.

  Because of the above three restrictions, in this embodiment, it is assumed that the positional relationship between the display unit 3100, the instruction unit 3400, and the viewpoint is substantially constant, and the operator can use the information obtained only by the touch sensing unit 3200. The blind spot area is detected approximately.

  FIG. 12 shows a display unit 3100, an instruction unit 3400, and a blind spot area estimated on the display unit 3100 as viewed from the operator's viewpoint. Specifically, in the processing described here, the blind spot area is uniquely obtained from the dominant hand of the operator and the position information obtained from the contact sensing unit 3200. The obtained information is only the position of the tip of the instruction means 3400, and the specific shape of the instruction means 3400 cannot be obtained. Therefore, instead of the exact shape of the blind spot area, for example, an elliptical area may be set based on the position of the tip of the instruction means 3400 and set as the blind spot area. The size and shape of the ellipse are not constant, and may be dynamically changed depending on the position on the display portion 3100. Moreover, since it cannot be estimated whether the operator's dominant hand is either, it is good to collect from an operator beforehand.

  As described above in the third embodiment, in a display device that is operated by touching the touch panel provided on the front surface of the display screen, a display that does not interfere with operation or browsing even if a blind spot area is created on the display by the operation. Is possible.

(Other examples)
Needless to say, the object of the present invention can be achieved as follows. That is, a recording medium (or storage medium) that records a program code (computer program) of software that implements the functions of the above-described embodiments is supplied to the system or apparatus. Needless to say, such a storage medium is a computer-readable storage medium. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, an operating system (OS) or the like running on the computer performs part or all of the actual processing based on the instruction of the program code. Needless to say, the process includes the case where the functions of the above-described embodiments are realized.

  Furthermore, it is assumed that the program code read from the recording medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. After that, based on the instruction of the program code, the CPU included in the function expansion card or function expansion unit performs part or all of the actual processing, and the function of the above-described embodiment is realized by the processing. Needless to say.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

DESCRIPTION OF SYMBOLS 1000 Display apparatus 1100 Display part 1200 Imaging part 1300 Control part 1400 Instruction means

Claims (13)

A display device for instructing an item on a display screen by an instruction means on the front surface of the display screen,
Display means for displaying the items on the display screen;
Detecting means for detecting a blind spot area where the display screen is hidden by the instruction means;
A display comprising: a blind spot area detected by the detecting means; and a changing means for changing the display of the item displayed by the display means from the positional relationship of the items displayed by the display means. apparatus.   The display device according to claim 1, wherein the changing unit moves the item when at least a partial region of the item enters the blind spot region.   The display device according to claim 2, wherein the movement is performed little by little.   The display device according to claim 1, wherein the changing unit performs at least one of movement, deformation, change of color and transparency, and blinking of the item. Further comprising a viewpoint position detecting means,
5. The display device according to claim 1, wherein the detection unit detects a region where the instruction unit becomes a blind spot from a viewpoint position detected by the viewpoint position detection unit. The viewpoint position detecting means detects positions of a plurality of viewpoints;
The display device according to claim 1, further comprising a selection unit that selects a priority viewpoint from the plurality of viewpoints when a plurality of viewpoints are detected by the viewpoint position detection unit.   The display device according to claim 1, wherein the detection unit does not include a tip of the instruction unit in a blind spot area. It further has a restoration means,
The restoration means controls the display to restore the display screen before being displayed by the changing means when the item comes out from the blind spot area after the changing means has changed the display screen. The display device according to any one of claims 1 to 4.   The viewpoint position detection means includes two imaging means, and each of the two imaging means detects the position of a person from the parallax obtained by imaging the user who uses the instruction means, and detects the position of the face from the detected person. The display device according to claim 5, wherein a position is detected, and a viewpoint position is detected from the position of the face.   The display device according to claim 5, wherein the viewpoint position detection unit includes an imaging unit, and detects the position of the viewpoint from the size of the face imaged by the imaging unit.   The said instruction | indication is at least a gesture instruction | indication or a touch instruction | indication with respect to a touchscreen, The said instruction | indication means is an operator's arm or finger | toe which instruct | indicates this instruction | indication means, The any one of Claims 1-10 characterized by the above-mentioned. The display device described. A control method of a display device that gives an instruction by an instruction means for an item on the display screen in front of the display screen,
A display step of displaying the item on the display screen;
A detection step in which the detection means detects a blind spot area where the display screen is hidden by the instruction means;
A display device control method comprising: a blind spot area detected by the detecting means; and a changing step in which the changing means changes and displays the display screen based on a positional relationship between items displayed in the display step. .   A program for causing a computer to execute each step of the control method for a display device according to claim 12.

Images