JPH10124681A - Portable information processor and method therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a portable information processor and an information processing method in which movement orbits from an image and plural inputs or a surrounding three-dimensional environment can be automatically restore and recorded, and the action pattern of a user can be estimated for real. SOLUTION: A moving area detecting device 3 reads the image information of each frame photographed by an image inputting device 2, and detects a moving area on an image screen between the two frame images. A moving component detecting device 4 calculates the moving components of the user of a portable information processor between the two frame images in the moving area on the image screen between the two transmitted frame images, successively adds the calculated moving components, and calculates the movement orbit of the user. A three-dimension restoring device 5 calculates what kind of three-dimensional structure is obtained in an image projected on the image screen according to the calculated moving components and the condition of the change of a way of viewing on the image screen in the surrounding environment.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a portable information processing apparatus and an information processing method, and more particularly to facilitation of real information management.

[0002]

2. Description of the Related Art With the remarkable progress in electronic technology, portable information devices for managing own information have become widespread. For example, Japanese Patent Application Laid-Open Nos. 3-55655 and 5-158887, and Japanese Utility Model Application Laid-Open No. H5-158887.
Various devices have been developed as disclosed in Japanese Patent Publication No. -15316. The portable information processing apparatus disclosed in Japanese Patent Application Laid-Open No. 3-55655 is provided with an image input device in addition to a key input device as an input device, and performs various information processing and communicates using a telephone line. , Making it possible to conduct business more reliably and efficiently.
The apparatus disclosed in Japanese Patent Application Laid-Open No. 5-158887 is
The GPS receiver and the electronic still camera are electrically connected to and incorporated in a computer of a multi-top type to improve the accuracy of fixed point photographing of the electronic still camera. In the device disclosed in Japanese Utility Model Publication No. 6-15316, a light receiving surface provided with a condensing device is provided on a side wall of a housing, and a monitor display of an imaging device is provided on another side wall of the housing, and characters, figures, and the like are provided. Can be entered directly into an electronic memo to simplify the operation. This type of device is used mainly for inputting customer information and data such as schedules when outdoors or when traveling, and then transferring the data to a personal computer for advanced information management. ing.

[0003] In addition, digital cameras that take images and the like, digitize the images, and record the digitized images in a memory have also appeared. With the digital cameras, images taken by the digital cameras are transferred to a personal computer, and the images are transferred to the personal computer. 2. Description of the Related Art Processing and editing image information and utilizing it for various purposes is becoming popular.

[0004]

However, these devices lack the ability to automatically process the information generated on the spot in real time, and thus, symbol information such as schedules and pattern information such as images. Also lacks the ability to fuse

The present invention has been made to solve the above problem, and automatically restores and records a moving trajectory from an image and a plurality of inputs and a surrounding three-dimensional environment to realistically estimate a user's action pattern. It is an object of the present invention to obtain a portable information processing apparatus and an information processing method capable of performing the above.

[0006]

A portable information processing apparatus according to the present invention has an image input device, a moving area detecting device, a moving component detecting device, and a three-dimensional restoring device, and the image input device captures an image. The moving region detecting device detects the moving region between the input frame images, the moving component detecting device calculates the moving component of the user itself from the detected moving region, and the three-dimensional reconstruction device calculates the moving component. And a three-dimensional structure of the surrounding environment is calculated from the image information and the image information.

A second portable information processing apparatus according to the present invention includes an image input device, a moving area detecting device, a moving component detecting device, a three-dimensional restoring device, a behavior storage device, and a behavior recognizing device. Captures and inputs an image, the moving region detecting device detects a moving region between the input frame images, the moving component detecting device calculates a moving component of the device itself from the detected moving region, and performs three-dimensional reconstruction. The device calculates the three-dimensional structure of the surrounding environment from the calculated moving component and the image information. The action storage device stores in advance symbol information representing various actions for the moving trajectory and the three-dimensional structure of the surrounding environment, and recognizes the action. The device calculates the user's behavior pattern from the moving component calculated by the moving component detection device, the three-dimensional structure of the surrounding environment calculated by the three-dimensional reconstruction device, and the symbol information indicating various actions stored in the behavior storage device. And wherein the Rukoto.

A third portable information processing apparatus according to the present invention has an image input device, a moving area detecting device, a distance measuring device, a moving component detecting device, and a three-dimensional restoring device. The moving area detecting device detects the moving region between the input frame images, and the distance measuring device calculates the distance to one point of the surrounding environment projected on the input image plane. The moving component detection device calculates the user's own moving component from the detected moving region and the calculated distance to one point in the surrounding environment, and the three-dimensional reconstruction device calculates the three-dimensional structure of the surrounding environment from the calculated moving component and image information. Is calculated.

A fourth portable information processing apparatus according to the present invention has an image input device, a moving area detecting device, a distance measuring device, a moving component detecting device, and a three-dimensional restoring device. The moving area detecting device detects the moving region between the input frame images, and the distance measuring device calculates the distance to a plurality of points of the surrounding environment projected on the input image plane, The moving component detecting device calculates the moving component of the user itself from the detected moving region and the calculated distance to a plurality of points in the surrounding environment. The three-dimensional restoration device calculates the three-dimensional structure of the surrounding environment from the calculated moving component and the image information. Is calculated.

A fifth portable information processing apparatus according to the present invention includes an image input device, a moving area detecting device, a position detecting device, a moving component detecting device, and a three-dimensional restoring device. The moving region detecting device detects a moving region between the input frame images, the position detecting device detects the position coordinates of the user based on latitude and longitude, and the moving component detecting device detects the moving position. The moving component of the user itself is calculated from the area and the position coordinates of the user, and the three-dimensional reconstruction device calculates the three-dimensional structure of the surrounding environment from the calculated moving component and the image information.

A sixth portable information processing apparatus according to the present invention includes an image input device, a moving area detecting device, a motion detecting device, a moving component detecting device, and a three-dimensional restoring device. The moving area detecting device detects the moving region between the input frame images, the motion detecting device detects the motion state of the user, and the moving component detecting device detects the moving region and the user. The three-dimensional reconstruction device calculates a three-dimensional structure of the surrounding environment from the calculated moving component and image information based on the motion state of the user.

The information processing method according to the present invention captures and inputs an image around the user, detects a moving area between the input frame images, and detects a moving component of the user from the detected moving area. Is calculated, and the three-dimensional structure of the surrounding environment is calculated from the calculated moving component and the image information.

In a second information processing method according to the present invention, an image around a user is captured and input, a moving area between the input frame images is detected, and the apparatus itself is detected from the detected moving area. A moving component is calculated, a three-dimensional structure of the surrounding environment is calculated from the calculated moving component and the image information, and the calculated moving component and the three-dimensional structure of the surrounding environment calculated by the three-dimensional reconstruction device and a predetermined moving trajectory are calculated. It is characterized in that a user's action pattern is calculated from symbol information indicating various actions on the three-dimensional structure of the surrounding environment.

In a third information processing method according to the present invention, an image around a user is photographed and input, a moving area between the input frame images is detected, and a surrounding area projected on an image plane is detected. The distance to one point in the environment is calculated, the moving component of the user itself is calculated from the detected moving region and the calculated distance to one point in the surrounding environment, and the three-dimensional surrounding environment is calculated from the calculated moving component and the image information. It is characterized in that the structure is calculated.

In a fourth information processing method according to the present invention, an image around a user is captured and input, and the moving area detecting device detects a moving area between frame images input by the image input device. Calculate the distance to a plurality of points in the surrounding environment projected on the image plane, calculate the moving component of the user itself from the detected moving area and the calculated distance to the plurality of points in the surrounding environment, and calculate the calculated movement. The three-dimensional structure of the surrounding environment is calculated from the components and the image information.

In a fifth information processing method according to the present invention, an image around a user is captured and input, a moving area between the input frame images is detected, and the position of the user is detected by latitude and longitude. Coordinates are detected, a moving component of the user itself is calculated from the detected moving area and the detected position coordinates of the user, and a three-dimensional structure of the surrounding environment is calculated from the calculated moving component and image information. .

In a sixth information processing method according to the present invention, an image around a user is captured and input, a moving area between the input frame images is detected, and a motion state of the user is detected. A moving component of the user itself is calculated from the detected moving region and the detected motion state of the user, and a three-dimensional structure of the surrounding environment is calculated from the calculated moving component and image information.

[0018]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A portable information processing apparatus according to the present invention manages the entire apparatus and controls processing procedures and data transfer, an image input apparatus, a moving area detecting apparatus, and a moving component detecting apparatus. It has a three-dimensional restoration device and a storage device. The image input device has a CCD camera, for example, and captures and stores an image for each frame. The moving area detecting device reads out image information for each frame taken by the image input device, detects a moving area on the image plane between two frame images, and sends the detected moving area to the moving component detecting device. The moving component detection device calculates a component that the user of the portable information processing device has moved between the two frame images based on a moving area on the image plane between the two transmitted frame images, The calculated moving components are sequentially added to obtain a moving locus of the user. The three-dimensional reconstruction device has a three-dimensional structure of the image projected on the image plane based on the moving component calculated by the moving component detection apparatus and the condition of the change in the appearance of the surrounding environment on the image plane. Is calculated. This processing is performed at regular intervals until an instruction to stop image capturing is input, and the input image information, the movement trajectory of the user, and the three-dimensional information of the surrounding environment are stored in the storage device. By outputting and confirming the image information, the user's movement trajectory, and the three-dimensional information of the surrounding environment stored in the storage device to a display device or the like, visual and realistic information management can be performed.

Also, an action storage device in which the portable information processing device previously stores symbol information representing various actions such as walking, a meeting, and eating with respect to the three-dimensional structure of the movement trajectory and the surrounding environment, and a user. Trajectory obtained from the moving component of
An action recognition device for calculating a user's action pattern by referring to the three-dimensional information obtained by the three-dimensional reconstruction device and the symbol information representing various actions stored in the action storage device is provided. By clarifying, more accurate information management can be performed.

When obtaining three-dimensional information of a user's movement trajectory and surrounding environment, a distance measuring device measures a distance to a point or a plurality of points of the surrounding environment projected on an image plane. Accordingly, the three-dimensional structure of the surrounding environment can be calculated including the length information.

Furthermore, the user's movement trajectory and surrounding environment 3
When obtaining the dimensional information, the user, that is, the current position and the actual movement trajectory of the portable information processing device are detected by the position detection device, and the motion status of the imaging system of the portable information processing device is provided with an acceleration sensor or a gyro. By detecting with the motion detection device, the three-dimensional structure of the surrounding environment including the movement trajectory and the length information of the user can be calculated more accurately.

[0022]

FIG. 1 is a block diagram showing the configuration of an embodiment of the present invention. As shown in the figure, the portable information processing apparatus manages the entire apparatus, and performs a processing procedure and data transfer control, an image input apparatus 2, a moving area detecting apparatus 3, and a moving component detecting apparatus 4. It has a three-dimensional restoration device 5, a display device 6, and a storage device 7. The image input device 2 has a photographing device including, for example, a CCD camera 21 for inputting an image and an image processing unit 22 for processing the input image. Image processing unit 2
Reference numeral 2 denotes an A / D converter 23 for converting an image signal obtained by the CCD camera 21 into a digital signal, a frame storage 24 for taking in image information converted into a digital signal, and storage in the frame storage 24. And a D / A converter 25 for D / A converting the image information thus obtained and sending it to the control device 1. Control device 1
Outputs the image information sent from the image input device 2 to the display device 6 and stores it in the storage device 7. The moving area detection device 3 inputs the frame image information stored in the frame storage unit 24 of the image input device 2, processes at least two pieces of frame image information, and moves between the frame images. Detect the area. The detection of the moving area is performed by a method of obtaining a difference between frame images, a method of performing correlation using local image features such as a correlation method, a feature matching method, a sparse / dense method, or a spatio-temporal differentiation method. This is performed by a method of calculating a moving area using the method. The moving component detecting device 4 calculates the moving component of the user of the portable information processing device, that is, the portable information processing device itself from the moving region detected by the moving region detecting device 3 and sequentially adds the components. The three-dimensional restoring device 5 calculates a three-dimensional structure of the surrounding environment of the user from the moving component of the user calculated by the moving component detecting device 4 and the image information input by the image input device 2. For example, when local features are correlated between two-frame images, if the user's own motion is only translation without rotation, the correspondence between the two points indicates the scale factor. If the user's own motion consists of both translation and rotation, the correspondence between the five points is known, and the user's own movement components and the three-dimensional surrounding environment are known. It is known that the shape is uniquely determined except for the scale factor. By repeating this process and sequentially adding the moving components, the moving locus of the user can be obtained. The storage device 7 records the thus obtained movement trajectory and the three-dimensional information of the surrounding environment.

The operation of the portable information processing apparatus configured as described above will be described with reference to the flowchart of FIG.

When a signal for instructing image capturing is input from the user (step S1), the image input device 2 starts image capturing for each frame and stores it in the frame storage unit 24 (step S2). The moving area detecting device 3 reads out image information for each frame stored in the frame storage section 24, detects a moving area on an image plane between two frame images, and detects a moving component. 4 (step S3).
The moving component detection device 4 calculates the component moved by the user between the two frame images based on the moving area on the image plane between the two transmitted frame images, and sequentially adds the components (step). S4). The moving trajectory of the user can be obtained by sequentially adding the calculated moving components of the user from the start of image capturing. The three-dimensional restoring device 5 has a three-dimensional structure of the image projected on the image plane based on the moving component calculated by the moving component detection device 4 and the condition of the change of the appearance of the surrounding environment on the image plane. Is calculated (step S5). This process is performed at regular intervals until an instruction to stop image capturing is input (step S6). When the image capturing stop signal is input, the image capturing is stopped, and the input image information, the moving trajectory of the user, and the three-dimensional information of the surrounding environment are stored in the storage device 7 as necessary (steps S6 to S8). ).

By displaying and confirming the image information stored in the storage device 7 and the three-dimensional information of the user's movement trajectory and surrounding environment on the display device 6, visual and realistic information management can be performed. .

In the above-described embodiment, the case where the information management is performed by clarifying the user's movement trajectory and the three-dimensional information of the surrounding environment has been described. It can be performed.

FIG. 3 is a block diagram showing the configuration of the embodiment which clarifies the behavior of the user. As shown in FIG. 3, the portable information processing device displays the control device 1, the image input device 2, the moving region detecting device 3, the moving component detecting device 4, and the three-dimensional restoring device 5 shown in the first embodiment. An action storage device 8 and an action recognition device 9 are provided in addition to the device 6 and the storage device 7. The behavior storage device 8 stores in advance symbol information indicating various behaviors such as walking, meetings, and meals with respect to the movement trajectory and the three-dimensional structure of the surrounding environment. The action recognition device 9 refers to the movement trajectory obtained from the movement component of the user, the three-dimensional information obtained by the three-dimensional restoration device 5, and the symbol information representing various actions stored in the action storage device 8, and refers to the user. Is calculated and output.

The operation of the portable information processing apparatus having the above-described configuration for clarifying the user's action will be described with reference to the flowchart of FIG.

The processing from inputting the image by instructing the user to take an image to calculating the three-dimensional structure of the image projected on the image plane is performed in the first embodiment. This is exactly the same as the example (steps S11 to S15). The action recognition device 9 refers to the movement trajectory of the user calculated by the movement component detection device 4, the three-dimensional information obtained by the three-dimensional restoration device 5, and the symbol information representing various actions stored in the action storage device 8. To calculate the user's behavior pattern (step S1).
6). This process is performed at regular intervals until an instruction to stop image capturing is issued (step S17). Then, when the image shooting stop signal is input, the image shooting is stopped, and the input image information, the moving trajectory of the user, and the three-dimensional information of the surrounding environment are stored in the storage device 7 as necessary (steps S17 to S1).
9).

In this way, the user's action can be recognized together with the user's movement trajectory and the three-dimensional information of the surrounding environment.

According to the second embodiment, continuous images are actually input to the image input device 2 at three types of places: a conference room, a corridor, and a resting place.
Each of the five sets was photographed at 640 × 480 pixels using the CCD camera 21, and the movement locus and the three-dimensional position were photographed. As a camera coordinate system of the CCD camera 21, as shown in FIG. 5, the focal point of the CCD camera 21 is defined as the origin O, and the image plane is defined as a plane parallel to the XY plane with a focal length f from the origin O. And a left-handed perspective transformation coordinate system 26
The dimension restoration device 5 calculates a coordinate value in the X, Y, Z coordinate system, that is, a three-dimensional position at a location corresponding to the selected local image feature, and sends the calculated three-dimensional position to the action recognition device 9 to perform action recognition. The device 9 outputs symbol information indicating the behavior of the user.

Then, about 20 local image features such as corners of the object are selected in three types of places, such as a conference room, a corridor, and a resting place, and the correspondence between the two frames is determined by a correlation method. Then, the moving area is detected, and the plane is reconstructed by the three-dimensional reconstruction device 5 from the three-dimensional coordinates of the local image feature and the edge information, color information, and connection relations in the vicinity thereof, and the form shown in FIG. And the three-dimensional structure 51 is restored. Then, four sets of three-dimensional positions and movement trajectories of the five sets of data were stored in the action storage device 8 as models. The behavior storage device 8 has a three-dimensional structure 52 of a conference room as shown in FIG. 7A, a three-dimensional structure 53 of a corridor as shown in FIG. 7B, and a break as shown in FIG. The three-dimensional structure 54 is stored as a sketch viewed from above.
The action recognition device 9 converts the three-dimensional structure and the user's movement trajectory restored from the input image into a sketch form, and matches the model stored in the action storage device 8 during a meeting or a corridor walk. And symbol information indicating the user's behavior during a break. Here, when a continuous image from entering the conference room to sitting on the seat is selected as an input, a moving locus 55 and a three-dimensional structure 56 as shown in FIG. 8 can be obtained. As a result of inputting the dimensional structure 56 to the action recognition device 9, an action information symbol “during a meeting” was output, and it could be confirmed that the action of the user was correctly recognized.

In the above-described embodiment, the case where the model-based recognition of recognizing the user's action is performed by referring to the symbol information representing various actions stored in the action storage device 8 has been described. Then, the user's action may be recognized by the learning / memory function.

Next, a description will be given of a third embodiment in which the distance to a certain point of the surrounding environment projected on the image plane is detected to obtain the moving trajectory of the user and the three-dimensional information of the surrounding environment.

As shown in the block diagram of FIG. 9, the portable information processing apparatus according to the third embodiment includes a control device 1, an image input device 2, a moving area detection device 3 and a moving device shown in the first embodiment. In addition to the component detecting device 4, the three-dimensional restoring device 5, the display device 6, and the storage device 7, a distance measuring device 10 for detecting a distance between a certain point of the surrounding environment projected on the image plane is provided. The distance measuring apparatus 10 has a surrounding environment by using a method of projecting a wave such as an infrared stereo method or an ultrasonic wave using the principle of triangulation and measuring a distance from a propagation time of reflection from an object to be measured. Detect the distance to a point. The three-dimensional reconstruction device 5 outputs a three-dimensional structure of the surrounding environment from the movement trajectory calculated by the movement component detection device 4 and the distance measurement result by the distance measurement device 10. For example, when associating local features between two frames,
If the correspondences of the five locations are known, the user's own moving component and the three-dimensional shape of the surrounding environment are uniquely determined. However, since the scale factor is unknown, the user's own moving amount and the object Size and distance to the target cannot be calculated.
However, if the distance to one projected point on the image is known, the scale factor is determined, and the three-dimensional structure of the surrounding environment can be calculated including the length information.

The operation when the three-dimensional structure of the surrounding environment including the length information is calculated by the portable information processing apparatus constructed as described above will be described with reference to the flowchart of FIG.

When a signal for instructing image capturing is input from the user (step S21), the image input device 2 displays one frame.
The image shooting for each program is started and stored in the frame storage unit 24 (step S22). The moving area detecting device 3 reads out image information for each frame stored in the frame storage section 24, detects a moving area on an image plane between two frame images, and detects a moving component. 4 (Step S2
3). On the other hand, the distance measuring device 10 measures the distance from the user, that is, the portable information processing device itself to one point in the surrounding environment corresponding to a certain point on the image, and sends it to the moving component detection device 4 (step S24). The moving component detecting device 4 calculates the moving component of the user between the two frame images, that is, the moving vector, using the transmitted moving detection result and the distance measurement result, and captures the calculated moving component of the user as an image. The moving locus of the user is obtained by successively adding from the start (step S2).
5). The three-dimensional restoring unit 5 determines what three-dimensional structure the image projected on the image plane has based on the moving component calculated by the moving component detection device 4 and the condition of the change of the appearance of the surrounding environment on the image plane. Is calculated with size information (step S26). This process is performed at regular intervals until an instruction to stop image capturing is input (step S27). When the image shooting stop signal is input, the image shooting is stopped,
3 of input image information, user's movement trajectory and surrounding environment
The dimensional information is stored in the storage device 7 as needed (steps S27 to S29). In this way, the three-dimensional structure of the surrounding environment can be calculated including the length and size information.

As described above, in the third embodiment, the distance to a certain point of the surrounding environment projected on the image plane is detected, and the three-dimensional structure of the surrounding environment is calculated including the length and size information. Although the case has been described, a more accurate three-dimensional structure can be output by detecting the distance to a plurality of points in the surrounding environment projected on the image plane.

A portable information processing apparatus according to a fourth embodiment, which detects distances from a plurality of points in the surrounding environment projected on an image plane and outputs a three-dimensional structure, as shown in the block diagram of FIG. First, the distance measuring device 11 includes, for example, a light irradiation device 12 and a light receiving device 13, and detects distances to a plurality of points in the surrounding environment projected on an image plane by a light projection method. Figure 1 shows the light projection method
As shown in FIG. 2, the light irradiation device 12 irradiates a light wave such as a spot light or a slit light to the object 14 to be measured.
This is a three-dimensional measurement method that measures the distance of a target object based on the principle of triangulation by observing reflections on four surfaces from different directions with a light receiving device 13. The dimensional shape can be measured. As the distance measuring device 11 for detecting the distance to a plurality of points in the surrounding environment, a method in which an infrared array is provided in an infrared stereo method or a plurality of ultrasonic transceivers is provided in a distance measuring method using ultrasonic waves. A technique may be adopted.

The three-dimensional compounding device 5 outputs a three-dimensional structure of the surrounding environment from the movement trajectory of the user calculated by the moving component detecting device 4 and the distance measurement result by the distance measuring device 11. That is,
As shown in the third embodiment, the three-dimensional structure restoration in consideration of the scale factor can be obtained if the distance to a certain point is known. However, in general, when a three-dimensional structure is restored by calculation only from an image, a large error may occur in the calculation result due to noise in the image or the geometric relationship between the surrounding environment and the user's own moving direction. Also, when tracking a moving area on an image, the size of the movement between the two frames is unknown, so the search range must be large. On the other hand, the distance measuring device 11 detects the distances to a plurality of points, and outputs the accurate three-dimensional structure by comparing the result of the three-dimensional reconstruction from the image with the distance detection result of the distance measuring device 11. Can be. In addition, by utilizing the distance detection result for detecting the moving area, the search range on the image is reduced, and the calculation cost can be reduced.

When the three-dimensional structure of the surrounding environment is to be calculated more accurately by the portable information processing apparatus constructed as described above, as shown in the flowchart of FIG. Is input (step S31),
The image input device 2 starts photographing an image for each frame and stores it in the frame storage unit 24 (step S32). The moving area detection device 3 stores one frame stored in the frame storage unit 24.
The image information for each frame is read, a moving area on the image plane between the two frame images is detected, and sent to the moving component detecting device 4 (step S33). On the other hand, the distance measuring device 11 measures the distance from the user, that is, the portable information processing device itself to a plurality of points in the surrounding environment, and sends it to the moving component detection device 4 (step S34). The moving component detecting device 4 calculates the moving component of the user between the two frame images, that is, the moving vector, using the transmitted moving detection result and the distance measurement result, and captures the calculated moving component of the user as an image. The moving locus of the user is obtained by successively adding from the start (step S3).
5). The three-dimensional restoring unit 5 determines what three-dimensional structure the image projected on the image plane has based on the moving component calculated by the moving component detection device 4 and the condition of the change of the appearance of the surrounding environment on the image plane. Is calculated with size information (step S36). This processing is performed at regular intervals until an instruction to stop image shooting is input. When an image shooting stop signal is input, image shooting is stopped, and the input image information, the user's movement trajectory, and the surrounding environment The dimension information is stored in the storage device 7 as needed (steps S37 to S39). In this way, the three-dimensional structure of the surrounding environment can be calculated more accurately, including information on the length and size.

Next, the current position of the user, that is, the portable information processing device is detected, and the movement trajectory of the user and the surrounding environment are detected.
A fifth embodiment for obtaining dimensional information will be described.

A portable information processing apparatus according to a fifth embodiment for detecting a current position of a user and outputting a three-dimensional structure is shown in FIG.
As shown in the block diagram of FIG. 1, in addition to the control device 1, the image input device 2, the moving region detecting device 3, the moving component detecting device 4, the three-dimensional restoring device 5, the display device 6, the storage device 7, and the GSP
(Global Positioning System) or the like, and has a position detecting device 15 for obtaining a user's current position and an actual trajectory using a navigation system. As described above, the scale factor is unknown only from the detection result of the moving area on the image plane, so that the actual moving amount of the user and the distance to the target of the size of the object cannot be calculated. However, if the actual moving direction and the moving amount of the user are known, the scale factor is determined, and the three-dimensional structure of the surrounding environment can be calculated including the length information. Therefore, in the fifth embodiment, the current position of the user and the actual trajectory of the user are detected by the position detecting device 15.

In the fifth embodiment, as shown in the flowchart of FIG. 15, a signal for instructing image capturing is input from the user, and the image input device 2 is used for each frame. An image is taken, the moving area detecting device 3 detects the moving area on the image plane between the two frame images, and the position detecting apparatus 15 measures the current position of the user to determine the position detected last time. (Steps S41 to S41)
44). The moving component detecting device 4 detects a moving vector of the user between the two frame images based on the detection result of the moving region and the change in the position of the user detected by the position detecting device 15, and detects the moving vector from the start of image capturing to the present. The movement vector of the user is calculated by adding the movement vectors (step S45). 3
From the detected movement vector and the condition of the change in the appearance of the surrounding environment on the image plane, the dimension restoration apparatus 5 determines the size of the three-dimensional structure of the image projected on the image plane. It is calculated with information (step S46). This processing is performed at regular intervals until an instruction to stop image shooting is input. When an image shooting stop signal is input, image shooting is stopped, and the input image information, the user's movement trajectory, and the surrounding environment The dimensional information is stored in the storage device 7 as needed (steps S47 to S49). In this way, the three-dimensional structure of the surrounding environment including the movement trajectory of the user and information on the length and size can be calculated more accurately.

Next, a description will be given of a sixth embodiment in which the imaging system, that is, the movement state of the user is detected to obtain the three-dimensional information of the movement trajectory of the user and the surrounding environment. As shown in the block diagram of FIG. 14, the portable information processing apparatus according to the sixth embodiment includes a control device 1, an image input device 2, a moving region detecting device 3, a moving component detecting device 4, and a three-dimensional restoring device 5. And a motion detection device 16 in addition to the display device 6 and the storage device 7. The motion detection device 16 includes, for example, an X-axis acceleration sensor 17a, a Y-axis acceleration sensor 17b, a Z-axis acceleration sensor 17c, and the X-axis gyro 1.
8a, a gyro 18b for the Y axis and a gyro 18c for the Z axis. The acceleration components of the X, Y, and Z axes are measured, and the user's movement vector is obtained by integrating the measured accelerations. it can. In addition, each of the gyros 18a to 18c can determine a rotation component around each of the X axis, the Y axis, and the Z axis. Since the actual movement direction and movement amount of the user can be known from the movement vector, the scale factor is determined, and the three-dimensional structure of the surrounding environment can be calculated including the length information. Further, since the rotation component of the imaging system can be known, the calculation cost required for detecting the movement trajectory and the three-dimensional structure can be reduced.

That is, in the sixth embodiment,
As shown in the flowchart of FIG. 17, a signal for instructing image capturing is input from the user, the image input device 2 captures an image for each frame, and the moving area detecting device 3 performs image capturing. The movement area on the image plane is detected between the frame images, and the motion detection device 16 measures the imaging system, that is, the motion status of the user, and changes the current position of the user and the position from the previous measurement position. And the rotation component of the imaging system are detected (steps S51 to S54). From this detection result and the detection result of the movement area, the movement vector and the movement trajectory of the user between the two frame images are calculated, and the three-dimensional structure of the image projected on the image plane is obtained. Is calculated with size information (steps S55 and S56). This processing is performed at regular intervals until an instruction to stop image shooting is input. When an image shooting stop signal is input, image shooting is stopped, and the input image information, the user's movement trajectory, and the surrounding environment The dimensional information is stored in the storage device 7 as needed (steps S57 to S59). In this way, the three-dimensional structure of the surrounding environment including the movement trajectory of the user and information on the length and size can be calculated more accurately.

In each of the above-described embodiments, a case has been described in which a series of processing of detecting a moving region and restoring a three-dimensional structure is performed each time an image is captured, but the moving region is detected using images of a plurality of frames. , A series of processing of three-dimensional structure restoration may be performed.

Further, in each of the above embodiments, a case has been described in which a series of processing of detecting a moving area and restoring a three-dimensional structure is performed in real time. However, image information is collectively stored in the storage device 7 and processed offline. You may do it.

Further, the image information, the movement trajectory and the three-dimensional information may be stored in the storage means 7 and transferred by a network or the like, or may be transferred and confirmed and managed by a network or the like directly.

[0050]

As described above, the present invention reads out image information for each frame taken by an image input device and detects a moving area on an image plane between two frame images. Calculating a component that the user of the portable information processing apparatus has moved between the two frame images based on the moving area on the image plane between the two detected frame images, and calculating the calculated moving component. The moving trajectory of the user is obtained by successive addition, and based on the calculated moving components and the condition of the change of the appearance of the surrounding environment on the image plane, what kind of three-dimensional structure is obtained for the image projected on the image plane. Is calculated, it is possible to check the three-dimensional information of the user's movement trajectory and the surrounding environment in a realistic manner, and to more accurately manage the information.

Further, the portable information processing apparatus stores in advance symbol information indicating various actions such as walking, meeting, and eating with respect to the three-dimensional structure of the movement trajectory and the surrounding environment, and stores the user's information. By referring to the movement trajectory obtained from the movement component, the three-dimensional information, and the symbol information representing various actions stored in advance, the user's action pattern is calculated to clarify the user's action. More accurate information management can be performed.

When obtaining the three-dimensional information of the user's movement trajectory and the surrounding environment, the distance to a certain point or a plurality of points of the surrounding environment projected on the image plane is measured. The three-dimensional structure of the environment can be calculated including the length information, and brilliant information can be obtained.

Further, the user's movement trajectory and surrounding environment 3
When obtaining dimensional information, the user, ie, the current position and the actual movement trajectory of the portable information processing device, or the movement state of the imaging system of the portable information processing device is detected to detect the movement trajectory of the user. Of the surrounding environment including the length and length information
The dimensional structure can be calculated more accurately.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of an embodiment of the present invention.

FIG. 2 is a flowchart showing the operation of the embodiment.

FIG. 3 is a block diagram showing a configuration of a second embodiment.

FIG. 4 is a flowchart showing the operation of the second embodiment.

FIG. 5 is an explanatory diagram illustrating a camera coordinate system according to a second embodiment.

FIG. 6 is an explanatory diagram showing a specific example of a three-dimensional structure according to a second embodiment.

FIG. 7 is an explanatory diagram illustrating a model stored in a behavior storage device according to a second embodiment.

FIG. 8 is an explanatory diagram showing a movement locus and a three-dimensional structure obtained in a second embodiment.

FIG. 9 is a block diagram showing a configuration of a third embodiment.

FIG. 10 is a flowchart showing the operation of the third embodiment.

FIG. 11 is a block diagram showing a configuration of a fourth embodiment.

FIG. 12 is an explanatory diagram illustrating a measuring operation of the distance measuring apparatus according to the fourth embodiment.

FIG. 13 is a flowchart showing the operation of the fourth embodiment.

FIG. 14 is a block diagram showing a configuration of a fifth embodiment.

FIG. 15 is a flowchart showing the operation of the fifth embodiment.

FIG. 16 is a block diagram showing a configuration of a sixth embodiment.

FIG. 17 is a flowchart showing the operation of the sixth embodiment.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 control device 2 image input device 3 moving region detecting device 4 moving component detecting device 5 three-dimensional reconstruction device 6 display device 7 storage device 8 behavior storage device 9 behavior recognition device 10 ranging device 11 ranging device 15 position detection device 16 motion Detector

Claims (12)

[Claims] 1. An image input device, a moving region detecting device, a moving component detecting device, and a three-dimensional restoring device. The image input device captures and inputs an image, and the moving region detecting device inputs a frame image. The moving component detecting device calculates the moving component of the user itself from the detected moving region, and the three-dimensional reconstruction device calculates the three-dimensional structure of the surrounding environment from the moving component and the image information. Characteristic portable information processing device. 2. An image input device, a moving region detecting device, a moving component detecting device, a three-dimensional restoring device, a behavior storage device, and a behavior recognizing device. The device detects a moving region between the input frame images, the moving component detecting device calculates the moving component of the device itself from the detected moving region, and the three-dimensional reconstruction device calculates the surrounding environment from the calculated moving component and the image information. The action storage device stores in advance the movement trajectory and the symbol information representing various actions with respect to the three-dimensional structure of the surrounding environment, and the action recognition device stores the movement component calculated by the movement component detection device. A portable information processing apparatus, wherein a behavior pattern of a user is calculated from a three-dimensional structure of a surrounding environment calculated by a three-dimensional restoration apparatus and symbol information representing various actions stored in a behavior storage device. 3. An image input device, a moving region detecting device, a distance measuring device, a moving component detecting device, and a three-dimensional restoring device. The image input device captures and inputs an image, and the moving region detecting device inputs. The distance measuring device calculates the distance to one point of the surrounding environment projected on the input image plane, and the moving component detecting device calculates the detected moving region. Portable information processing, wherein a moving component of the user itself is calculated from a distance to one point in the surrounding environment, and the three-dimensional reconstruction device calculates a three-dimensional structure of the surrounding environment from the calculated moving component and image information. apparatus. 4. An image input device, a moving region detecting device, a distance measuring device, a moving component detecting device, and a three-dimensional restoring device, wherein the image input device captures and inputs an image, and the moving region detecting device inputs. The distance measuring device calculates the distance to a plurality of points in the surrounding environment projected on the input image plane, and the moving component detecting device calculates the moving region as the detected moving region. Portable information processing, wherein a user's own moving component is calculated from distances to a plurality of points in the surrounding environment, and the three-dimensional reconstruction device calculates a three-dimensional structure of the surrounding environment from the calculated moving component and image information. apparatus. 5. An image input device, a moving region detecting device, a position detecting device, a moving component detecting device, and a three-dimensional restoring device, wherein the image input device captures and inputs an image, and the moving region detecting device inputs the image. The moving region between the detected frame images is detected, the position detecting device detects the position coordinates of the user, and the moving component detecting device calculates the moving component of the user itself from the detected moving region and the position coordinates of the user. And a three-dimensional restoration device that calculates a three-dimensional structure of a surrounding environment from the calculated moving component and image information. 6. An image input device, a moving region detecting device, a motion detecting device, a moving component detecting device, and a three-dimensional restoring device. The image input device captures and inputs an image, and the moving region detecting device inputs. The movement detecting device detects the movement state of the user, and the movement component detection device calculates the movement component of the user itself from the detected movement region and the movement state of the user. And a three-dimensional restoration device that calculates a three-dimensional structure of a surrounding environment from the calculated moving component and image information. 7. An image around the user is captured and input,
A moving area between input frame images is detected, a moving component of the user is calculated from the detected moving area, and a three-dimensional structure of a surrounding environment is calculated from the calculated moving component and image information. Portable information processing method. 8. Photographing and inputting an image around the user,
Detecting a moving region between the input frame images, calculating a moving component of the apparatus itself from the detected moving region, calculating a three-dimensional structure of a surrounding environment from the calculated moving component and image information;
The user's action pattern is calculated from the calculated moving component, the three-dimensional structure of the surrounding environment calculated by the three-dimensional reconstruction device, and a predetermined moving trajectory and symbol information indicating various actions with respect to the three-dimensional structure of the surrounding environment. A portable information processing method, comprising: 9. An image around the user is taken and input,
The moving area between the input frame images is detected, the distance to one point in the surrounding environment projected on the image plane is calculated, and the distance is used from the detected moving area and the calculated distance to one point in the surrounding environment. A portable information processing method comprising: calculating a moving component of a user himself; and calculating a three-dimensional structure of a surrounding environment from the calculated moving component and image information. 10. An image of a user's surroundings is taken and input, and the moving area detecting device is a frame input by the image input device.
Of the surrounding environment projected on the image plane, and calculates the distance to a plurality of points in the surrounding environment. A portable information processing method comprising: calculating a moving component; and calculating a three-dimensional structure of a surrounding environment from the calculated moving component and image information. 11. An image around the user is captured and input, a moving area between the input frame images is detected, the position coordinates of the user are detected based on latitude and longitude, and the detected moving area is detected. A portable information processing method comprising: calculating a moving component of the user itself from the detected position coordinates of the user; and calculating a three-dimensional structure of a surrounding environment from the calculated moving component and image information. 12. A surrounding image of a user is captured and input, a moving area between the input frame images is detected, a motion state of the user is detected, and the detected moving area and the detected user are detected. A mobile information processing method comprising: calculating a moving component of a user itself from a motion state of the user; and calculating a three-dimensional structure of a surrounding environment from the calculated moving component and image information.