CN105917385A - Information processing device and information processing method - Google Patents

Abstract

In the present invention, an information processing device communicates with a block set (260) assembled by a user and thereby obtains structural information for the block set (260). In addition, a corresponding location setting screen (300), which includes an image (302) of an object corresponding to the block set (260), is displayed on a display device. The user moves a joint in the block set (260) for which the user desires to create an association, by bending and stretching the joint (for example, see arrow A). Next, the user uses a cursor (308) to specify a joint in a 3D object in the 3D object image (302) to be associated with the joint in the block set, and then confirms this specification. The joint in the block set (260) and the joint in the 3D object are associated with each other. By setting corresponding locations in the block set (260) and the object, the actions of the block set and the object are linked to each other.

Description

Information processing device and information processing method

technical field

The present invention relates to an information processing technique using objects in real space.

Background technique

The following techniques are used in various fields: measuring parameters related to objects (e.g., people, things, etc.) in real space by some means, capturing parameters as input values into a computer, and analyzing the input values and displaying the input values as images . In the field of computer games, intuitive and easy operation is achieved by acquiring the movement of the user himself/herself or markers held by the user and moving characters in the virtual world within the display screen accordingly (see, for example, PTL 1). Therefore, it is expected to apply a technique of reflecting movement or change of a shape of an object in real space in a screen display not only to games but also to toys, learning materials, etc. (for example, see NPL 1).

[citation list]

[Patent Document]

[PTL 1] WO 2007/050885A2

[Non-patent literature]

[NPL 1] Posey: Instrumenting a Poseable Hub and Strut Construction Toy, Michael Philetus Weller, Ellen Yi-Luen Do, Mark D Gross, Proceedings of the Second International Conference on Tangible and Embedded Interaction, 2008, pp.39-46

Contents of the invention

[technical problem]

In order to create a sense of reality and enable intuitive operation in a mode in which information processing progresses by using another object other than the information processing layer performing the main operation, it is important to provide perceptual provision as described above. A device that has a shape close to the real thing (such as, for example, the shape of a steering wheel of a car, the shape of a pistol, etc.) and can perform similar operations can be provided, but the device is limited in use. When the shape of the device is variable, the range of use is widened. However, inventions for measuring changes in shape or movement accordingly become necessary.

In the technology disclosed in NPL 1, for example, an infrared LED and a photosensor receiving light from the infrared LED are included in a coupling section of a component to measure a rotation angle of the component and identify the shape of the component. In this case, the measurable rotation angle is limited, and thus the variable range of shape is also limited. In addition, all parts need to include elements, so that the manufacturing cost increases. Therefore, the more flexible the form of the device, the more complex the mechanism for measuring the form of the device. Therefore, manufacturing costs and handling costs tend to increase.

The present invention has been made in view of these problems. The purpose of the present invention is to achieve different and advanced expressions by using a device that can be freely formed.

[Solution to problem]

A mode of the invention relates to an information processing device. The information processing apparatus includes: a structure information acquisition section that acquires information related to the position of the movable part in the overall structure of the external object. The information processing device further includes: an information processing section for displaying a setting screen on the display device for receiving a setting input from a user regarding an interlocking rule for the movable part in order to associate the object and the object with the object Model interlocking, generating information related to interlocking rules based on the received setting input, and storing the information related to interlocking rules in a storage device.

Another mode of the present invention relates to an information processing device. The information processing apparatus includes: a structure information acquisition section that acquires information related to the position of the movable part in the overall structure of the external object. The information processing device further includes: an information processing section for displaying a corresponding position setting screen on the display device for receiving from the user a link for associating the movable part of the object with the movable part of the object model associated with the object. to interlock the object and the object model with each other by reflecting changes in the movable parts of one of the object and the object model in the associated movable parts of the other of the object and the object model, and Corresponding information is generated based on the received setting input, and stored in a storage device.

Still another mode of the present invention relates to an information processing method. The information processing method is executed by an information processing device. The information processing method includes the step of acquiring information on the position of the movable part in the overall structure of the external object. The information processing method further includes the step of: displaying a corresponding position setting screen on the display device for receiving a setting input for associating the movable part of the object with the movable part of the object model associated with the object from the user, so as to interlock the object and the object model with each other by reflecting changes in the movable parts of one of the object and the object model in the associated movable parts of the other of the object and the object model. The information processing method further includes the steps of generating corresponding information based on the received setting input, and storing the corresponding information in a storage device.

Yet another mode of the present invention relates to an information processing device. The information processing apparatus includes: a structure information acquisition section that acquires information related to the position of the movable part in the overall structure of the external object. The information processing device further includes: an information processing section displaying a movement correspondence setting screen on the display device for receiving from a user a movable part for setting a movable part for the object and an object model associated with the object The setting input of the movement reflection rule, the movable parts are associated with each other so that by reflecting the change of the movable parts of one of the object and the object model in the associated movable parts of the other of the object and the object model The method interlocks the object and the object model with each other, generates corresponding information based on the received setting input, and stores the corresponding information in the storage device.

Still another mode of the present invention relates to an information processing method. The information processing method is executed by an information processing device. The information processing method includes the step of acquiring information on the position of the movable part in the overall structure of the external object. The information processing method further includes the step of displaying a movement correspondence setting screen on the display device for receiving from a user a movement reflection rule for setting a movable part for an object and a movable part of an object model associated with the object The setting input of that the movable parts are associated with each other so that the object can be transformed by reflecting changes in the movable parts of one of the object and the object model in the associated movable parts of the other of the object and the object model and object models are interlocked with each other. The information processing method further includes generating corresponding information based on the received setting input, and storing the corresponding information in a storage device.

Yet another mode of the present invention relates to an information processing device. The information processing apparatus includes: a structure information acquisition section that acquires information related to the position of the movable part in the overall structure of the external object. The information processing device further includes: an information processing section for displaying a corresponding position setting screen on the display device for receiving from the user a link for associating the movable part of the object with the movable part of the object model associated with the object. The setting input of to interlock the object and the object model with each other by reflecting changes in the movable parts of one of the object and the object model in the associated movable parts of the other of the object and the object model, receiving A setting input for associating different numbers of movable parts between the object and the object model with each other, generating correspondence information based on the received setting input, and storing the correspondence information in the storage device.

Still another mode of the present invention relates to an information processing method. The information processing method is executed by an information processing device. The information processing method includes the step of acquiring information on the position of the movable part in the overall structure of the external object. The information processing method further includes the step of: displaying a corresponding position setting screen on the display device for receiving a setting input for associating the movable part of the object with the movable part of the object model associated with the object from the user, so as to interlock the object and the object model with each other by reflecting changes in the movable parts of one of the object and the object model in the associated movable parts of the other of the object and the object model. The information processing method further includes the step of receiving a setting input for associating different numbers of movable parts between the object and the object model with each other. The information processing method further includes the steps of generating corresponding information based on the received setting input, and storing the corresponding information in a storage device.

Note that any combination of the above constituent elements and modes acquired by converting the expression of the present invention between methods, devices, systems, computer programs, recording media recording the computer programs, etc. is also effective as a mode of the present invention.

[Advantageous Effects of the Invention]

According to the present invention, various and advanced expressions can be realized using a device that can be freely formed.

Description of drawings

FIG. 1 is a diagram depicting an example of a configuration of an information processing system to which the present embodiment can be applied.

FIG. 2 is a diagram depicting an example of the external appearance of a block set in this embodiment.

FIG. 3 is a diagram depicting only the structure of communication blocks in the set of blocks depicted in FIG. 1 .

Fig. 4 is a diagram schematically depicting the central axis (the central axis is derived) of the core in the present embodiment.

FIG. 5 is a diagram schematically depicting an example of an internal configuration of a communication block included in a block set in the present embodiment.

FIG. 6 is a diagram depicting in detail the block set and the configuration of the information processing device in the present embodiment.

FIG. 7 is a diagram schematically depicting an information transmission path in a block set in the present embodiment and an example of transmitted information.

FIG. 8 is a diagram depicting an example of the structure of data of basic information on communication blocks in the present embodiment.

FIG. 9 is a diagram of assistance in explaining basic processing of identifying a state of a block set including non-communication blocks in the present embodiment.

FIG. 10 is a diagram depicting an example of the structure of data of basic information on non-communication blocks in the present embodiment.

FIG. 11 is a diagram of assistance in explaining the process of identifying the shape of a block set in a time-evolving manner in the present embodiment.

FIG. 12 is a diagram of assistance in explaining the process of identifying the shape of a block set whose structure is changed in the present embodiment.

FIG. 13 is a diagram of assistance in explaining the process of identifying the shape of a block set modified due to a change in the joint angle of the core in the present embodiment.

FIG. 14 is a flowchart depicting a processing procedure for identifying a state of a block set including non-communication blocks in the present embodiment.

FIG. 15 is a diagram showing the relationship between block sets and displays that can be realized in the present embodiment.

FIG. 16 is a diagram showing a relationship between a block set and a display in the case where an appearance is set for the block set in the present embodiment.

FIG. 17 is a diagram showing a relationship between a block set and a display in the case where one 3D object is associated with the assembled block set in the present embodiment.

FIG. 18 is a diagram showing information necessary for associating a block set and movement of a 3D object with each other in the present embodiment.

FIG. 19 is a flowchart depicting a processing procedure in which the information processing device in this embodiment associates movement of a block set and movement of a 3D object with each other.

FIG. 20 is a diagram depicting an example of a screen displayed on a display device to receive model selection input by a user in S46 in FIG. 19 .

FIG. 21 is a diagram depicting an example of a screen displayed on the display device to set the coordinate system common to the block set and the selected model in S48 in FIG. 19 .

FIG. 22 is a diagram depicting an example of a set of blocks when an object is created by two blocks and a transition of a display screen at the time of registration in the present embodiment.

FIG. 23 is a diagram showing a case where joints associated with each other move at the same angle on a one-to-one basis in the present embodiment.

FIG. 24 is a diagram depicting an example of correspondence between angles of respective joints in the case where joints of two groups are associated with one joint in the present embodiment.

FIG. 25 is a diagram depicting another example of correspondence between angles of respective joints in the case where two joints are grouped and associated with one joint in the present embodiment.

FIG. 26 is a diagram showing a case where changes in angles of joints associated with each other on a one-to-one basis are made different in the present embodiment.

FIG. 27 is a diagram depicting an example in which one joint of a block set is associated with a plurality of joints of a 3D object in the present embodiment.

FIG. 28 is a diagram depicting an example of a screen displayed on a display device to set a correspondence between a block set and movement of a 3D object in S52 in FIG. 19 .

FIG. 29 is a diagram depicting an example of a data structure of information on correspondence between block sets and corresponding positions of 3D objects and movements of the respective corresponding positions in the present embodiment.

FIG. 30 is a flowchart depicting a processing procedure for setting correspondence between movements of 3D objects and block sets in the present embodiment.

FIG. 31 is a diagram of assistance in explaining the case where the settings related to the round of block sets in this embodiment are extended to composite links.

detailed description

In the present embodiment, a plurality of blocks are assembled or modified, and the shape, posture, and position of the blocks are used as input values for information processing. That is, these blocks can be regarded as input devices for information processing devices. Also, there are cases where the shape, posture, and position of assembled blocks are changed so as to reflect the results of processing performed by the information processing device. In this case, the block is regarded as an output device for the information processing device. Although the processing performed by the information processing device is not particularly limited in this case, preferred modes thereof will be shown later. A union of these blocks or an assembly of blocks will hereinafter be referred to collectively as a "block set". Also, as will be described later, the block set may include objects other than blocks in the general sense (such as things imitating costume accessories, clay artifacts, etc.), and the shapes and materials of the objects are not limited. A block including these objects will be referred to as a "block" hereinafter.

FIG. 1 depicts an example of the configuration of an information processing system to which the present embodiment can be applied. The information processing system 2 includes: a block set 120; a camera 122, which images the block set 120; an information processing device 10, which performs predetermined information processing with the block set 120 as an input device or an output device; a user operation of the processing device 10; and a display device 16 that displays data output from the information processing device as an image.

The information processing device 10 may be, for example, a game device or a personal computer, and may realize an information processing function by loading necessary application programs. The display device 16 may be a general display such as a liquid crystal display, a plasma display, an organic EL display, or the like. The display device 16 may also be a television including one of these displays as well as speakers. The input device 14 may be one of common input devices such as a game controller, keyboard, mouse, joystick, touch pad disposed on the screen of the display device 12, or any combination thereof.

The connection of the information processing device 10 to the camera 122, the input device 14, and the display device 16 may be made by wire or radio, and may be made via various networks. Alternatively, two or more or all of the camera 122 , the information processing device 10 , the input device 14 , and the display device 16 may be integrally combined and provided. Furthermore, the camera 122 does not necessarily need to be mounted on the display device 16 . There may be a plurality of chunk sets 120 depending on the content processed by the information processing device 10 . The block set 120 and the information processing device 10 establish a wireless connection using the Bluetooth (registered trademark) protocol, the IEEE 802.11 protocol, or the like. Alternatively, one block in block set 120 and information processing device 10 may be connected to each other via a cable.

As described above, the block set 120 according to the present embodiment can be used as an input device for the information processing device 10 , or can be used as an output device for the information processing device 10 . Specifically, in the former case, the information processing device 10 executes information processing using the result of the user changing the position, attitude, or shape of the block set 120 as an input value, and displays the result of the processing as an image on the display device 16 . In the latter case, the information processing device 10 performs information processing according to the operation of the input device 14 by the user, and moves the block set 120 itself as a result of the information processing. The present embodiment may be configured to be able to realize both modes, or may be configured to be able to realize only one of the modes.

FIG. 2 depicts an example of the external appearance of individual blocks constituting a block set. In this embodiment, blocks are roughly classified into two categories. One is a block configured to be able to communicate with another block or the information processing device 10 . The other is a block without communication means. Hereinafter, the former will be referred to as "communication block", and the latter will be referred to as "non-communication block". In addition to mechanisms for communicating with another piece, a communicating piece may also include various kinds of sensors that measure physical quantities such as the piece's orientation, angle, position, etc.

Whether a block is a communicating block or a non-communicating block, as depicted in the figures, the blocks can have various shapes (such as quadrangular prismatic blocks 102a, 102b, and 102c, cubical blocks 102d, circular cylindrical blocks 102f and 102k, spherical block 102e, plate-shaped block 102i, rectangular parallelepiped block 102j, etc.). Each block is provided with a protruding section 104 and a concave section 106 of predetermined size and shape. By inserting the protruding section 104 into the concave section 106, the pieces can be coupled to each other at the desired location. Alternatively, as for the rectangular parallelepiped block 102j, the circular cylindrical block 102k, etc., the block may be equipped with a concave section 107 having a shape in which another block can fit itself and thus be able to contain said other block.

The set of blocks may also include joint blocks 102g and 102h having two ends that may be inserted into the recessed sections 106 of different blocks to adjust the spacing between the blocks coupled to each other. In addition, the joint block may be capable of changing the position and posture relationship between the blocks connected to each other by rotation or the like.

The protruding sections 104 and recessed sections 106 of the communication blocks also function as terminals enabling signal transmission between the blocks. To this end, the ends of the respective segments are equipped with connectors having a structure conforming to the standard or bus etc. provided within the block. Both signal transmission and physical coupling between blocks can be achieved by employing various kinds of connectors that are generally used or providing dedicated special connectors. In addition, when the signal transmission path can be separately prepared and the coupling position can be separately identified, the means for connecting the pieces to each other is not limited to the coupling between the protruding section 104 and the concave section 106, but can be made by hooks. Ring fasteners, magnets, adhesive tapes, adhesives and more are realized. The path of the signal transmission of the separately prepared path may be a mechanism of radio communication.

Additionally, some of the communication blocks (in the case of FIG. 2 , quadrilateral prism block 102b ) may include two blocks: a bend and stretch rod 110 that enables the two blocks to bend and stretch; The angle formed between the blocks. By the way, in addition to the form in which both ends of the bending and stretching rod 110 penetrating one block are engaged with the protrusions of the other block as described in the drawings, the bending and stretching mechanism may be two blocks that can be bent through a hinge. And the form in which stretched metals and the like are coupled to each other, or may be a mechanism with multiple degrees of freedom which is similar to a joint in a ball joint doll or the like, is not particularly limited thereto. The angle between the blocks may be made continuously changeable, or it may be made changeable in multiple steps. Furthermore, the orientation of the rods is not limited to the orientation depicted in the figures.

Even after the user removes the hand from the blocks, the angles between the blocks as constituent elements are preferably maintained. By the way, the angle between the blocks can be measured by an angle sensor other than a potentiometer. For example, when a piece includes within it a sensor that measures a relative angle to another piece, the pieces do not necessarily need to be connected to each other. Also, as described later, one block can be configured to be able to bend and stretch, or to be able to rotate, and the bending and stretching angle or the rotation angle of the block can be measured.

Hereinafter, a mechanism for making the angle variable in this way may be referred to as a "joint", and two blocks whose relative angles are changed according to movement of the joint may be referred to as a "link". Furthermore, the joint angle of such a communication block having a joint can be made controllable according to a request from the information processing device 10 . In this case, the communication block is equipped with an actuator (such as a servo motor, etc.) for controlling the joint angle.

In addition, some of the communication blocks (in the case of FIG. 2 , plate-shaped blocks 102i) may have rotatable rods 109 protruding from the side surfaces. Wheels mounted on bars 109 enable the mass to move as in the case of a vehicle. The movement may be realized by pushing the blocks by the user, or may be realized according to a request from the information processing device 10 . In the latter case, the communication block is equipped with an actuator (such as a motor, etc.) for rotating the lever. Where the lever 109 is a wheel shaft, a mechanism for changing the orientation of the wheel, such as a rack or pinion, may be provided and also made controllable by an actuator according to a request from the information processing device 10 .

In addition, some of the communication blocks include therein one or a combination of acceleration sensors, gyro sensors, geomagnetic sensors, etc., and motion sensing functions such as using cameras and A method of tracking gestures such as markers attached to an object, or the object's shape itself, etc.). The blocks to include sensors and the kind and combination of sensors to be included are determined according to information processing to be realized by using a set of blocks. Alternatively, the user selects the block to include the sensor and the kind and combination of the sensor to be included from among various variations at the time of assembly.

Furthermore, some of the communication blocks (in the case of FIG. 2 , quadrangular prism blocks 102 a ) may be equipped with markers 108 . The marker 108 is used to identify a position in a three-dimensional space according to the position and size of the marker 108 in an image captured by a camera to be described later. Therefore, the marker 108 is formed to have a size, shape, and color that can be detected from a captured image by matching processing or the like. For example, marker 108 may be a spherical body comprising a general light emitting body such as a light emitting diode, light bulb, etc. within an optically transparent spherical resin, or marker 108 may be a barcode, a QR code, or the like. When multiple blocks are equipped with markers 108, the markers 108 may vary in color from block to block.

The housings of the communication block and the non-communication block are typically formed by synthetic resin. However, the material of the housing of the communication block and the non-communication block is not limited, and may be metal, glass, or the like. Since the non-communication block specifically does not include a communication mechanism or the like, the material, shape, and size of the non-communication block can be freely determined. For example, non-communicating pieces may be various kinds of parts (such as clothes created using cloth, doll's heads created using rubber, etc.), or may be additional things (such as weapons, charms, etc.). The user himself/herself can make non-communication blocks. A non-communicating piece may be a solid body formed, for example, by cutting an eraser, a clay workpiece, a workpiece made of paper, a workpiece made of folded paper, or the like. In addition, the non-communication block may be, for example, a block having LEDs that emit light of a predetermined color by being excited from a communication block or a display device that displays an image.

The information processing apparatus 10 according to the present embodiment uses information on the skeleton shape, posture, etc. of the block set (which can be acquired through communication with the communication block) and information on the external shape captured by the camera 122 to obtain High accuracy identifies the state of the block. Therefore, the external appearance of a collection of blocks can be freely expressed by using non-communicating blocks. For example, a block having a shape including a communication block, such as a rectangular parallelepiped block 102j or a circular cylindrical block 102k in FIG. 2 , may be implemented as a non-communication block.

FIG. 3 only depicts the structure of the communication blocks of the block set 120 described in FIG. 1 . Specifically, by excluding the rectangular parallelepiped block 102j and the circular cylinder block 102k as non-communication blocks from the block set 120 of FIG. to form block set 120a in the figure. Among the blocks, the lower block of the quadrangular prism block 102b and the cubic block 102d are respectively contained by the rectangular parallelepiped block 102j and the circular cylindrical block 102k which are non-communication blocks in the block set 120 in FIG. 1 . The lower ones of the quadrangular prism blocks 102b and the cubic blocks 102d are thus not visible from the outside. The structure formed by these communication blocks can also be regarded as the skeleton forming the entire set of blocks 120 . The component constituted by the communication blocks in the assembled block set 120 will be referred to as "core" hereinafter.

The present embodiment efficiently calculates the pose and shape of the block set 120 by detecting necessary parameters by motion sensors and potentiometers included in the communication blocks constituting the core. For example, in the case of block set 102a in FIG. 3 , the orientation of each block and thus the shape and pose of the central axis of block set 120 can be derived based on: (1) the coupling position of each block and the position of the block Kind; (2) the inclination vector m1 of the quadrilateral prism block 102a or 102b; (3) the angle θ between the two blocks constituting the quadrilateral prism block 102b; and (4) the lengths L1, L2, L3, L4 and L5.

Given that (1) and (4) above are identified by signal transmission between the blocks and (3) above can be measured by a potentiometer, it is necessary and full. Alternatively, it is sufficient to select a block including a motion sensor as the quadrangular prism block 102a or 102b.

In addition, the position coordinates of the block set in the three-dimensional space of the real world are identified by using the image captured by the camera 122 . Here, when the camera 122 is a stereo camera, the absolute position of the block set in the three-dimensional space formed with respect to the depth direction of the camera 122 and the field plane of the camera may be acquired. A technique of acquiring the position of an object in a three-dimensional space according to the principle of triangulation by using parallax in images captured by a stereo camera from mutually different left and right viewpoints is well known. Instead of a stereo camera, a depth or three-dimensional information acquisition device different from a binocular stereo image may be used. For example, a point-of-view camera can be used, or the location of the set of blocks can be identified by a time-of-flight (TOF) method using an infrared radiation mechanism and an infrared sensor that detects reflected light. A touch panel may be provided on the top surface of the pedestal on which the block set 120 is placed, and a position where the block set 120 is placed may be detected by the touch panel.

Alternatively, when using the quadrangular prism block 102 a equipped with the marker 108 as described in the drawings, the position may be identified based on a frame image of a still image or a moving image captured by the monocular camera 122 . When the marker 108 is a light-emitting body of known color, brightness, and size, as described above, the image of the marker can be easily detected from the captured image. Then, the position coordinates (x1, y1, z1) of the marker in the three-dimensional space can be identified from the position and size of the image of the marker in the captured image. In the case of using other markers, common image recognition techniques (such as pattern matching, feature point extraction, etc.) can be applied. In a case where the set of blocks 120 moves and the moved set of blocks 120 is photographed as a moving image, efficient detection may be performed by applying an existing tracking technique.

Incidentally, the marker 108 may be a device that emits invisible light (such as infrared rays, etc.). In this case, a device for detecting invisible light is introduced separately to detect the position of the marker 108 . Similarly, depth sensors, ultrasonic sensors, sound sensors, etc. may be used. The final position coordinates can be calculated by combining two or more of the above-mentioned absolute position detection methods. Figure 4 schematically depicts the central axis of the core thus derived. As depicted in the drawings, the position, attitude and shape of the central axis 124 in the three-dimensional space are identified. The three-dimensional space may be the camera coordinate system of the camera 122 , or may be a desired coordinate system transformed from the camera coordinate system of the camera 122 . The position, pose and shape of a core or collection of blocks may be collectively referred to as a "state" hereinafter.

Fig. 5 schematically depicts an example of an internal configuration of a communication block. The block can be used properly according to the usage by providing various variations as the internal configuration of the block, as described above. Furthermore, when sensors assumed to be necessary for identifying the state of a core are provided so as to be distributed in a plurality of blocks, excessive inclusion of sensors can be avoided, and thus manufacturing costs can be reduced.

In the example of FIG. 5, block 126a includes battery 128a, communication mechanism 130a, memory 132a, position sensor 134, and motion sensor 136a. Assume in this case that the communication mechanism 130 a includes not only a wired communication mechanism that receives a signal from another piece via a connection terminal but also a mechanism that performs radio communication with the information processing device 10 . Memory 132a holds the identification number of block 126a. The identification number is associated with information in the information processing apparatus 10 such as the size of the block 126a, the position of the concave section or the protruding section of the block 126a, and the like. Blocks of the same kind can be given the same identification number. Alternatively, the identification number may be uniquely set for each block so as to be useful for routing of signal transmissions etc. within the assembled set of blocks.

The position sensor 134 is a sensor for acquiring the absolute position of the block 126a. The position sensor 134 also includes markers for image recognition. However, in the case of a marker, the absolute position is detected by a combination of the marker and the externally mounted camera 122, as described above. The motion sensor 136a is one or a combination of two or more of an acceleration sensor, a gyro sensor, and a geomagnetic sensor, and a method using a camera, etc., as described above.

Block 126b includes battery 128b, communication mechanism 130b, memory 132b, and motion sensor 136b. The mechanism may be similar to that described above for block 126a. However, the communication mechanism 130b may be formed only by a wired communication mechanism that receives a signal from another block. Such a block is used in combination with the block 126 a capable of communicating with the information processing device 10 . The same holds true for the communication mechanisms of the other blocks.

Block 126c includes battery 128c, communication mechanism 130c, memory 132c, angle sensor 138, and actuator 139a. Block 126c is a communication block (such as quadrilateral prism block 102b in FIG. 2 ) with joints. The angle sensor 138 is a sensor (such as a potentiometer or the like) that detects the joint angle. The actuator 139 a changes the joint angle according to a control signal from the information processing device 10 . A general technique corresponding to the kind of actuator can be applied to drive the actuator according to the control signal.

Block 126d includes battery 128d, communication mechanism 13Od, memory 132d, rotary encoder 141, and actuator 139b. Block 126d is a communication block (such as plate-shaped block 102i in FIG. 2 ) having a rotatable lever protruding outward. When the block 126d is assembled with the wheels, the block 126d itself can be advanced manually or automatically. Alternatively, rods and wheels may be integrally provided in advance.

The rotary encoder 141 is a sensor that detects the amount of rotation of the wheel. The actuator 139 b is a motor or the like that rotates a wheel according to a control signal from the information processing apparatus 10 . Block 126e includes communication mechanism 130e and memory 132e. That is, block 126e includes neither a battery nor a sensor. Therefore, the block 126e is used in combination with another block 126a or 126b including a battery.

Incidentally, the communication block of FIG. 5 is only an example, and various kinds of sensors or other mechanisms may be combined in any way. For example, not only joints and axles but also mechanisms that change the steering direction and displace part of the mass may be provided as moving parts of the mass assembly. These mechanisms can be moved by actuators driven according to control signals from the information processing device 10 . Additionally, LED or display devices may be provided. Mechanisms may be provided to energize connected non-communicating blocks. In addition, any sensor in actual use may be included in addition to the sensors described in the drawings.

Communication blocks with these various internal configurations were prepared in various shapes as depicted in FIG. 2 . Multiple blocks of the same kind can be prepared. Alternatively, all blocks may have a uniform shape and a uniform size, or may have a uniform internal configuration. When the shape and internal configuration vary, each kind of block can be purchased individually, and a desired block set can be flexibly assembled at minimum cost according to each individual user's use. A collection of basic blocks that can be assembled may be provided initially, and additional blocks may be purchased thereafter.

FIG. 6 describes the configuration of the block set 120 and the information processing device 10 in detail. Elements to be described as functional blocks that execute various processes in FIG. 6 may be configured in terms of hardware by a central processing unit (CPU), a memory, or another LSI, or realized in terms of software by a program loaded into a memory or the like . Furthermore, as described above, the blocks in the block set 120 include communication mechanisms, memory, various kinds of sensors, and actuators. Therefore, those skilled in the art should understand that these functional blocks can be realized in various forms by only hardware, only software or a combination thereof, and are not limited to any one of the forms.

As described above, the collection of blocks 120 is formed by the user by selecting and assembling each block. Figure 6 depicts the functional blocks of the core components in the assembled set of blocks. The communication blocks constituting the core are set as a first block 142a, a second block 142b, a third block 142c, . . . and so on. In order to prevent complication of information, basically only one block establishes communication with the information processing device 10 among the communication blocks forming the block set 120 . Accordingly, the role of a hub is assigned to the first block 142a. Then, information is transferred starting with the communication block in the farthest connection relationship to the first block 142a, and information on the entire core is aggregated in the first block 142a.

Hereinafter, blocks relatively closer to the first block 142a in the connection of blocks will be referred to as "higher-level" blocks, and blocks relatively farther from the first block 142a in the connection of blocks will be referred to as "lower-level" blocks . One block to be set as the first block 142a may be determined in advance. Alternatively, a block having a mechanism for communicating with the information processing apparatus 10 may be equipped with a switch or the like not described in the drawings, and a block whose switch is turned on by the user may be set as the first block 142a. Alternatively, a block that first establishes communication with the information processing device 10 in the assembly level may be set as the first block 142a.

When the user couples another communication block to the thus determined first block 142a, the block becomes the second block 142b. When a further communication block is coupled to the second block 142b, that block becomes the third block 142c. Incidentally, although the drawing depicts only three communication blocks, the number of communication blocks constituting the core is not limited. The configuration and operation of one communication block or four or more communication blocks can be considered in a similar manner.

The first block 142a, the second block 142b, and the third block 142c include first communication sections 143a, 143b, and 143c, component information acquisition sections 144a, 144b, and 144c, and second communication sections 146a, 146b, and 146c, respectively. The second block 142b also includes a drive section 148 . However, the drive section 148 may be provided in any other communication block. The first communication sections 143a, 143b, and 143c receive information transmitted from lower-level blocks directly connected to the first communication sections 143a, 143b, and 143c. The information received in this case includes the identification numbers of the blocks connected at a lower level than the corresponding blocks, the identification numbers of the coupling positions and the results of the measurements made by the built-in sensors. When multiple blocks are coupled to each other, information is superimposed along each path through the block from the lowest level block.

The component information acquisition sections 144a, 144b, and 144c include sensors built in corresponding blocks and terminals provided at positions for connecting other blocks. The element information acquisition sections 144a, 144b, and 144c acquire information on the results of measurements made by the sensors and the locations to which lower-level blocks are connected. The second communication sections 146a, 146b, and 146c add the information acquired by the component information acquisition sections 144a, 144b, and 144c of the corresponding blocks to the information received by the first communication sections 143a, 143b, and 143c. The information received by sections 143a, 143b and 143c includes identification numbers of lower level blocks, identification numbers of coupling locations and results of measurements made by built-in sensors. The second communication sections 146a, 146b and 146c signal the resulting information to directly connected higher level blocks. However, the second communication section 146 a of the first block 142 a transmits information to the information processing device 10 . In addition, the second communication section 146a is used, for example, by receiving processing start and end request signals from the information processing apparatus 10, various kinds of signals necessary to establish communication, control signals for driving actuators of block sets, and the like. interface of the information processing device 10 .

When a control signal for driving an actuator is sent from the information processing apparatus 10, the signal is sequentially transmitted from the first block 142a to lower-level blocks. Specifically, the first communication sections 143a, 143b, and 143c of the corresponding blocks transmit signals to directly connected lower-level blocks. The second communication sections 146b and 146c of the respective blocks receive signals from directly connected higher-level blocks. The drive section 148 of the second block 142b includes actuators that change the joint angle or rotate the axle. When the second block 142b is designated as an object to be driven in the control signal transmitted from the higher-level block, the driving section 148 moves the actuator by an amount corresponding to the control signal.

The information processing device 10 includes: a core information receiving section 20 that receives information related to the state of the core from the first block 142a in the block set 120; The state-related information of the core to identify the shape, posture and position of the block set 120; the information processing section 30, which performs predetermined information processing according to the shape, posture and position of the block set 120 or the user operation on the input device 14; display A processing section 32 that generates an image to be displayed as a result of information processing and outputs the image to the display device 16 ; and a drive control section 34 that sends a signal for controlling the operation of the block set 120 . The information processing device 10 also includes: a block information storage section 24 that stores information related to individual blocks; a model data storage section 26 that stores model data of 3D objects to be displayed on the display device 16; and corresponding information storage Section 28, which stores the correspondence information between the set of blocks and the parts and movements of the 3D object.

The core information receiving section 20 receives a signal including information on the identification numbers of the communication blocks constituting the core, the coupling positions of the communication blocks, and the results of measurements made by built-in sensors, the information being determined by the first block in the block set 120. 142a polymerized. The structure analysis section 22 acquires data of a moving image or a still image acquired by photographing the block set 120 from the camera 122 . Then, the information received by the core information receiving section 20 and the information obtained from the captured image are integrated to identify the position, posture and shape of the entire block set 120 . Signals from block set 120 and image data from camera 122 are input instantaneously. Therefore, it is assumed that a temporal correspondence is established between the signal from the set of blocks 120 and the data from the camera 122 . However, depending on necessary temporal resolution, synchronization processing and the like may be performed.

The structure analysis section 22 identifies the shape and pose of the core of the block set 120 based on the information from the core information receiving section 20 . For example, the information on L1 to L5 in FIG. 3 is derived based on the identification numbers of the communication blocks constituting the core. In addition, the coupling position and angle formed between the blocks are identified according to the identification number of the real coupling position and the information of the angle sensor. Furthermore, the vector m1 in Fig. 3 is derived from the information of the motion sensor. Information about the position of the block set 120 in three-dimensional space and the surface shape of the block set 120 including non-communication blocks is identified based on the captured image transmitted from the camera 122, a depth image generated from the captured image, and the like.

At this time, an image of a communication block included in the core (such as, for example, the marker 108 in FIG. 3 , etc.) is detected from the image. The marker 108 is set as a reference member, and the position of the marker 108 is derived from a depth image or the like. Then, the positional relationship between the core and the non-communicating block can be identified based on the structure of the core connected from the reference component (such as the central axis 124 in FIG. 4 ) and the positional relationship of the image of the non-communicating block to the image of the reference component and This in turn identifies the position, pose and shape of the entire collection of blocks. When this processing is performed at a predetermined frequency, even the structure of a block set that is in the process of being assembled by the user can be recognized in real time.

The block information storage section 24 stores basic information on each block used as a block set. In the case of a communication block, the basic information is information that associates an identification number given to the block in advance with information on the shape, size, and position where other blocks can be connected. In the case of non-communicating blocks, the basic information is information that associates an identification number given to the block in advance with external features such as color, pattern, material, texture, and the like. In the case of non-communicating blocks, the more detailed these external features are, the more accurately the block can be identified. However, when information processing performed by the information processing device 10 does not require identification of each non-communication block, there is no need to store information on the non-communication block.

The information processing section 30 performs processing to be performed according to the state of the block set 120 identified by the structure analysis section 22 or via a user operation of the input device 14 . For example, after the block set 120 is assembled, a 3D object representing the shape of the block set or a 3D object of a model associated with the block set is displayed. Then, the displayed 3D object is caused to move according to the movement of the block set 120 . Alternatively, the computer game is started and made to progress according to user manipulation via the input device 14, and the set of pieces 120 is moved according to the progress of the computer game.

To this end, the model data storage section 26 stores data required by the information processing section 30 to render the object model displayed on the display device 16 . The object model can be designed in advance (such as a character appearing in a game, etc.), or can be created by the user from a collection of assembled blocks. The information processing section 30 further performs processing for associating the block set and parts of the object (such as joints, wheels, etc.) with each other and further associating the movements of the parts of both the block set and the object with each other. At this time, the information processing section 30 may set all correspondences, or may display a setting screen that allows the user to establish association and receive setting input. Alternatively, they may be combined with each other as appropriate. The correspondence information storage section 28 stores information on such set correspondences of parts and movements.

Therefore, even when a user freely creates a block set, not only the position but also the shape and posture of the block set can be interlocked with the object on the screen. For example, the world of a game may be reflected in a block set in the real world, or the movement of a block set may be reflected in a character in a virtual world. At this time, the movement of both the block set and the object does not necessarily need to be exactly the same as each other, but various changes are allowed to be set by associated movement. Furthermore, movement does not need to be reflected in real time. For example, when a change in the state of a block set moved by a user with the lapse of time is stored, a mode in which a corresponding object reproduces movement at an arbitrary time can be realized. In turn, the movement of characters in computer games or animations can be created with simple operations.

In a mode in which the information processing apparatus 10 moves the block set 120 , the drive control section 34 sends a control signal to the block set 120 according to a request from the information processing section 30 . Specifically, the transmitted signal varies depending on the control system, and techniques commonly used in the field of robotics and the like may be adopted as appropriate. The transmitted control signal is received by the second communication section 146a of the first block 142a in the block set 120, and is reflected on the target block (in the case of FIG. 6, the second block 142b) by signaling within the block set 120 The drive section 148 is in operation. Alternatively, the control signal may be directly transmitted to the target block by radio communication or the like.

The display processing section 32 creates image data as a result of the processing performed by the information processing section 30 and displays the image data on the display device 16 . In the example of displaying an object that moves according to the movement of the set of blocks 120 , the object is rendered at the output frame rate of the display device 16 to correspond to the movement of the set of blocks 120 and is output as a video signal on the display device 16 . Common computer graphics techniques can be applied to the rendering process itself. The display processing section 32 further displays, on the display device 16 , a screen for setting association of parts and movements of the block set 120 and objects with each other. In the case where the information processing section 30 makes all the associations, a corresponding screen for the user to check or correct the settings may be displayed. Furthermore, the display processing section 32 displays an image (such as a game screen or the like) corresponding to the information processing being performed by the information processing section 30 as appropriate.

Fig. 7 schematically depicts information transmission paths in block set 120 and an example of the transmitted information. Each circle in which a numeral is written in the information transmission path 150 represents a block. The straight lines between the respective circles represent the state in which the blocks are coupled to each other. Also, set the number inside the circle as the identification number of the corresponding block. A block with identification number "1" corresponds to the first block 142a in FIG. 6, and establishes communication with the information processing device 10. Furthermore, the blocks with identification numbers "2" and "3" in FIG. 7 are connected in series with the block with identification number 1, and can thus be viewed as corresponding to the second block 142b and third block 142c in FIG. 6, respectively. correspond.

On the other hand, multiple blocks can be coupled to one block. In the example of FIG. 7 , a block with identification number "2" and a block with identification number "5" are connected to a block with identification number "1". As described above, the block with the identification number "3" and the block with the identification number "4" are connected in series with the identification number "2" in this order. The block with identification number "6" and the block with identification number "7" are connected to each other in parallel with the block with identification number "5". In this example, the block without identification number is further connected to the block with identification number "6", and the block with identification number "8" is connected to the block without identification number. A block without an identification number corresponds in this case to a non-communication block.

As mentioned above, information is basically sent from lower level blocks to higher level blocks. Fig. 7 depicts the content of the information sent and the arrows indicating the direction of sending. Information transmitted from the block with identification number "3" to the block with identification number "2" is expressed as [3:J2(4)], for example. This is a signal formed in the format "Own identification number: identification number provided to the connection position of the block (identification number of the block connected to the connection position)" and indicates that the block with the identification number "4" is connected to The position with the identification number "J2" among the connection positions of the identification number "3". However, the drawings limit neither the format nor the content of the information.

The direction corresponding to a higher-level block can be determined by sorting or the like performed by a block having a role of a hub by searching a network formed by coupling blocks to each other. The networking technology in the device tree constituting the general information handling system can be applied to this process.

The block with identification number "4" in FIG. 7 is at the lowest level in the connection sequence to which the block with identification number "4" belongs. The block with identification number "4" therefore sends information to the block with identification number "3" at the level immediately above. Assuming that no other blocks are connected to the block with identification number "4", that the connection location of the block with identification number "4" is uniquely identified, and that the block with identification number "4" does not include any sensors, the information sent is only is to identify the block numbered "4" itself, and thus what is sent is expressed as "[4:-]". "-" indicates that neither sensor measurements nor connected blocks exist.

When the block with identification number "3" has received a signal from identification number "4", the block with identification number "3" sends a pass to the block with identification number "2" at the level immediately above that will serve as a connection The signal obtained by associating the identification number "4" with the identification number "3" of the block itself with the identification number "4" of the identification number of the position, the number of the terminal that has received the signal from the identification number "4" etc. . The transmitted content of this signal is [3:J2(4)], as described above. A block with identification number "2" is similarly generated by associating its own identification number with the identification number of the connection location (in the example of the drawing, "J5") and the identification number "3" of the connected block. signal (ie [2:J5(3)]). Furthermore, assuming that the block with the identification number "2" includes a sensor, the block with the identification number "2" also generates a signal obtained by associating a signal indicating the result of measurement of the sensor with its own identification number. In the example of the drawings, the measurement result is expressed as "result". However, in reality, specific numerical values are substituted according to the type of sensor.

The block with the identification number "2" sends the data thus generated and the data sent from the lower level block (ie [3:J2(4)]) to the block with the identification number "1" at the level immediately above . However, these signals are not always sent at the same time. For example, when the content of a once-transmitted signal is changed, only information on the change may be transmitted. Meanwhile, assuming that the blocks with identification numbers "6" and "7" connected to the block with identification number "5" do not include sensors, and the connection positions with identification numbers "6" and "7" are uniquely identified, then from These blocks send the signals [6:-] and [7:-] respectively to the block with identification number "5", as in the case of the block with identification number "4". The block with identification number "6" is connected with a further block. However, this block is a non-communicating block, and therefore no information is acquired from the non-communicating block.

A block with identification number "5" generates a signal obtained by associating its own identification number with the identification number of the connection location and the identification number of the block it is connected to, and sends this signal to the immediately above level with Block number "1" is identified. When multiple blocks are connected as described in the figure, these blocks are collectively indicated by [5: J3(6), J8(7)] and so on. Here, "J3" and "J8" are the identification numbers of the connection positions to which the blocks with the identification numbers in parentheses are connected.

Information about the core of the set of blocks is thus aggregated in a block with identification number "1". As for the other blocks, a block having an identification number "1" generates a signal obtained by associating its own identification number with the identification number of the connection position and the identification number of the block connected to the connection position. Then, the block with the identification number "1" transmits this signal to the information processing device 10 together with the signal transmitted from the lower-level blocks. The information processing device 10 can thus continuously acquire the identification numbers of the blocks constituting the core, the connection relationship of each block, and the measurement results in the blocks including the sensors.

When one block is thus set as a block having the role of a hub and information is thus aggregated and then transmitted to the information processing device 10, complexity of information and unnecessary communication processing can be prevented. On the other hand, in some cases, communication from a plurality of blocks to the information processing device 10 may be performed. For example, in the example of FIG. 7, a block with identification number "8" is coupled to a block with identification number "6" via a non-communicating block.

In this case, the block with the identification number “8” can directly transmit its own data to the information processing device 10 . When the block in question includes a position sensor, for example, the block's own identification number and the results of measurements made by the position sensor are directly sent to the information processing device 10 . The information processing device 10 can thus grasp the existence of the coupled block faster than the block with the identification number "6", further estimate the shape of the block in question, and approximate the connection situation of the block in question. The greater the number of sensors included in the block with identification number "8", the higher the accuracy of the information. When blocks from which a plurality of position information can be acquired are combined with each other, it is also possible to accurately identify the structure of blocks in the blind spot from the camera 122 .

FIG. 8 depicts an example of a data structure of basic information of a communication block, which is stored in the block information storage section 24 of the information processing device 10 . The communication block information table 160 includes an identification number field 162 , a shape field 164 , a size field 166 , and a connection location field 168 . The identification number field 162 describes the identification number given in advance to the communication blocks forming the block set. The shape field 164 describes the kind of shape of each communication block (ie, the type of block shown in FIG. 2, such as "quadrilateral prism", "cube", etc.)). Size field 166 describes the horizontal width, depth and vertical length of each communication block.

The connection location field 168 describes the connection location provided to each communication block in association with the identification number of the connection location. In the example of FIG. 8 , the connection position is described in the format "identification number of connection position (face number, x-coordinate in-plane, y-coordinate in-plane)". The face number is uniquely determined for each face of the block in advance. For example, a communication block having an identification number "1" is a quadrangular prismatic block having a horizontal width of 4 cm, a depth of 4 cm, and a vertical length of 8 cm. The connection position with the identification number "J1" is located at the position of coordinates (2, 2) in the first plane. The connection position with identification number "J2" is located at the position of coordinates (1, 2) in the second plane. However, the format of the comments is not particularly limited just to represent such information.

When the information processing device 10 holds such a communication block information table 160 , the parameters expressed in FIG. 3 are identified for the core based on the signal transmitted from the block set 120 . The structure analysis section 22 identifies the position, pose and shape of the entire set of blocks 120 including non-communicating blocks based on the states of the cores thus identified and the images captured by the camera 122 . FIG. 9 is a diagram of assistance in explaining a basic process for identifying a state of a set of blocks including non-communicating blocks. The upper left of the drawing depicts the state of the core 170 , which is identified based on the information received by the core information receiving section 20 . What is identified according to this information is the connection relationship of each communication block and the shape of the core (the shape is based on the connection relationship of each communication block). However, when a position sensor is provided internally, the position in real space is also identified.

At the same time, the structural analysis section 22 generates a depth image 172 from the image captured by the camera 122 . A depth image is an image showing objects within the field of view of the camera 122 with the distance from the camera as a pixel value. A stereo camera or the like may be used as the camera 122 to generate the depth image, as described above. The depth image 172 in the figure schematically represents an image whose brightness decreases as the distance increases. An image of the entire set of blocks 120 is shown in the depth image 172 with no distinction between communicating and non-communicating blocks. When an image of at least one part (such as eg a marker) belonging to a block of the core is detected in the depth image, the location coordinates of the part in three-dimensional space (including the distance of the part from the camera) can be identified. Then, the camera coordinate system relative to the core 170 is set such that the detected component exists at the position coordinates in the image in the previously identified state of the core 170 .

Incidentally, when an image of a kernel is detected, a color image captured by the camera 122 may be used. Then, the status of non-communicating blocks excluding components of the core in the block set can be identified by taking the volume difference between the core 170 viewed from the camera 122 side and the block set of images appearing as the depth image 172 . The shaded parts of the set of blocks 120 depicted on the right side of FIG. 9 are non-communication blocks acquired as differences. Thus, the location, pose and shape of the entire set of blocks 120, including core and non-communicating blocks, can be identified as depicted in the figure. Furthermore, when the image of the block set 120 can be identified by background separation, etc., and the image and core can be located based on the apparent size of the core, etc., it is possible to use only the normally captured image without relying on the depth image.

Incidentally, in the basic processing described in FIG. 9 , the information acquired on the non-communication block is only two-dimensional information in the field plane of the camera. Accordingly, when basic information of a non-communication block is stored in the block information storage section 24 and a search is performed for a block consistent with the apparent shape and size in the field plane, the accuracy of identifying the three-dimensional shape including the depth direction can be improved. FIG. 10 depicts an example of a data structure of basic information of non-communication blocks whose information is stored in the block information storage section 24 of the information processing device 10 .

The non-communication block information table 180 includes an identification number field 182 , a shape field 184 , a size field 186 and a color field 188 . The identification number field 182 describes the identification number given in advance to the non-communication blocks forming the block set. Blocks of the same shape, size and color may have the same identification number. The shape field 184 describes the kind of shape of each non-communication block (ie, the type of block shown in FIG. 2, such as "rectangular parallelepiped", "circular cylinder", etc.)). Size field 186 describes the horizontal width, depth (or diameter) and vertical length of each non-communication block. Color field 188 describes the color of each non-communication block.

The information of the shape field 184, the size field 186 and the color field 188 may be information about polygons, textures, etc., as is the case with data about object models of 3D graphics. Furthermore, the information held by the non-communication block information table 180 is not limited to the information described in the drawings. For example, when the shape of the recessed section etc. limits the connectable communication blocks and the identification numbers of the connectable communication blocks are saved, the non-communication blocks that can be connected to the communication blocks belonging to the already identified core can be narrowed down based on the communication blocks piece. Structural analysis section 22 refers to non-communicating block information table 180 and identifies non-communicating blocks that are individually consistent with images of components other than the core in depth image 172 depicted in FIG. 9 .

Incidentally, the exact shape of a block that is hidden by another block and thus cannot be seen is identified in a time-evolutionary manner by tracking the movement of the set of blocks after assuming that the block does not exist in the initial state. When a part is hidden and the shape of the part cannot be determined even when referring to the non-communication block information table 180, a certain candidate shape is assumed only for the hidden part, or a certain face is assumed, and gradually at a later time Make corrections to increase the accuracy of shape recognition.

FIG. 11 is a diagram of assistance in explaining a process of identifying a shape of a block set in a time-evolving manner. The axis of the ordinate in the figure indicates time. Assume that time elapses from time "T1" to "T2". Furthermore, as an example, as depicted in the uppermost row of the drawings, by combining the lower half of a communication block of quadrangular prism type equipped with markers (outlined) with a non-circular cylinder type (depicted in hatching) Communication blocks are assembled to form block set 190 . First, assume that at time T1, as in the drawing, the camera 122 photographs a block set 190 placed on a horizontal surface from the front. In this case, the side surfaces of each block only appear as images in the captured image 192a, as depicted in the figure.

When a depth image is generated from this image as described above and the volume difference from the shape-identified core is taken separately, the remaining part is clearly a quadrilateral, which is an image of the side surface of the non-communicating block (depth image 198a). That is, there is a possibility that the three-dimensional shape of the non-communication block cannot be identified at time T1. Depending on the resolution of the depth image, however, the distinction between circular cylinders and rectangular parallelepipeds may be made clear by the presence or absence of curves of the front surface. Furthermore, in the case where there is only one non-communication block whose size, aspect ratio, etc. are consistent among the blocks registered in the non-communication block information table 180, the shape of the non-communication block can be identified. Incidentally, the depth image 198a represents volume data generated by acquiring the volume difference between the image of the block set in the depth image acquired from the captured image and the kernel, and is not necessarily generated as an image. The same holds true for subsequent figures.

In other cases, structure analysis section 22 detects candidate non-communicating blocks from non-communicating block information table 180 and assumes that one of the candidate non-communicating blocks is coupled to the core. Alternatively, the same plane as that of the communication block whose shape is identified is assumed to be an indefinite plane of the non-communication block. As an example of the former case, the figure depicts the shape 200 of the set of blocks when the non-communicating blocks are assumed to be rectangular parallelepipeds. Incidentally, the shape 200 of the block set depicted in the drawing is a shape recognized at the time point of time T1 in the information processing device 10 and is not necessarily intended for display. For example, during execution of an application that displays the state of a collection of blocks as it is as a 3D object, the assumed shape described in the drawing may be presented as it is. Alternatively, the assumed shape may not be displayed in any way, but may only serve as a basis when correcting the shape in the next time step.

In order to identify the shape of a non-communicating block in a time-evolutionary manner regardless of whether it is displayed or not, or to enable the user to identify changes in the shape of the set of blocks in real-time and efficiently in the process of assembly, store the assumed shape for at least a predetermined period of time. related information, and this information is used for later processing. The shape of the block set is managed by giving structure identification numbers (hereinafter referred to as "element numbers") to communication blocks and non-communication blocks constituting the block set. In the drawing, a communication block in a block set is given a block element number "#C1", and a non-communication block in a block set is given an element number "#N1". In this example, communicating blocks and non-communicating blocks are distinguished from each other by the "C" and "N" of the alphabet. However, this does not limit the format of the component number.

These element numbers are associated with the identification numbers given to the respective blocks in advance (the identification numbers are described in FIGS. surface, attached position and orientation) are recorded together. Such information on the overall structure of the block set will be referred to as "structural data" hereinafter. Since the collection of blocks that appear as one subject in an image can be divided into units that can be attached and detached by the user, managing the structure of each block not only enables display as described above, but also improves the process of identifying the assembled The shape of the block collection or the efficiency of the shape of the block collection that has been modified. In the case of rendering the shape of the block set as it is, polygons and textures for rendering can be generated at this point in time. Then, in a subsequent shape identification process, the 3D model can be corrected, or blocks can be added or deleted.

After identifying the shape of the set of blocks, such as shape 200 , etc. at time T1 , the photographing and shape identification process continues. At this point, when the user tilts the apex of block collection 190 to one side of camera 122, as depicted in the figure, the top surfaces of communicating blocks 194 and non-communicating blocks 196 are slightly included in the captured image at time T2. Image 192b. When a depth image is generated from this image and the state-to-core volume difference at that point in time is acquired, the remaining components include the top surface of the circular cylinder of the non-communication block (depth image 198b). It can be determined from the shape of the image that the non-communicating blocks are most likely circular cylinders, rather than rectangular parallelepipeds assumed at time T1. By repeating these corrections as the pose of the block set changes, the reliability of recognition of the shape of the block set is increased.

At the point in time when the non-communicating block is found to be a circular cylinder, the structural analysis section 22 replaces the shape of the non-communicating block assumed to be a rectangular parallelepiped at time T1 with a circular cylinder. The exact shape 202 of the set of blocks is thus identified. This process is actually a process of correcting the identification number of the rectangular parallelepiped type block associated with the element number #N1 to the identification number of the circular cylinder type block. Alternatively, polygonal models can be corrected. The example of FIG. 11 shows a block set with a very simple structure for easy understanding. However, actually, for example, another non-communicating block may be connected behind a non-communicating block, or only a part of the non-communicating block can be seen in a case where the non-communicating block and another block overlap each other.

In this case, in addition to the method of assuming the shape of each non-communicating block and gradually identifying only a part of it as the user holds and tilts the block set or changes the orientation of the block set, as described above, it is also possible to Displays are made to prompt the user to rotate the block set relative to the camera 122 so that the block set can be photographed from multiple directions. In addition, candidate non-communication blocks may be extracted from the non-communication block information table 180 based on the shape and color of a part that is not hidden, and displayed in a list so that the user can designate a real block. Furthermore, when a two-dimensional barcode indicating the shape of the figure or a marker is attached to each non-communicating block, the shape can be identified from the captured image.

FIG. 12 is a diagram of assistance in explaining a process of identifying a shape of a block set that changes in structure during assembly or the like. The representation of the accompanying drawings is similar to that of FIG. 11 . Depicted in order from the left of the axis having the ordinate as the time axis are images 192b to 192d captured at respective times, depth images 198b to 198d of images of components excluding cores, and shapes 202 of recognized block sets, 210 and 216. Time T2 in the uppermost row in the drawing corresponds to time T2 in FIG. 11 . The same is true for the captured image 192b, the depth image 198b, and the recognized shape 202 of the set of blocks. Assume that from this state, at subsequent time T3, the user connects a new non-communication block 204 (the captured image 192c). A comparison of the depth image 198c at this time with the depth image 198b at a previous time T2 indicates the addition of a new image 206 of a non-communication block.

Structural analysis section 22 identifies connections of new non-communicating blocks 204 by comparing depth image 198b at previous time T2 with depth image 198c at current time T3. Here, the non-communicating blocks 196 that also appeared at a previous time may change in orientation relative to the camera, or the non-communicating blocks 196 may not be distinguished from newly connected blocks. Accordingly, by continuing to track the positions and orientations of non-communicating blocks that were once recognized, the structure analysis section 22 can recognize the same block as the same even when the positions and orientations of the blocks change. Ordinary tracking technology using an effective contour model or the like can be applied as tracking of position and orientation. Alternatively, a change in orientation of a non-communicating block in a state connected to the core may be derived from a change in position and orientation of the core, which can be identified by a signal from the core.

As a result of the comparison of the images after the volume difference, the connection of a new non-communicating block 204 can thus be detected, after which, by referring to the non-communicating block information table 180, for example in a manner similar to that described with reference to FIG. A shape that identifies a non-communication block. Then, the connection relationship to the core is identified based on the shape and pose of the core at the same time. Accordingly, the shape 210 of the set of blocks at time T3 can be identified as depicted in the figure. At this time, the structural data is updated by giving a new element number "#N2" to the added non-communication block, associating the new element number "#N2" with the identification number given in advance to the block in question, and Record the connection relationship to the communication block.

Incidentally, in order to determine whether non-communicating blocks appearing in the camera's field of view are connected or not yet connected, the relative velocity between the set of blocks including the core and the blocks appearing in the field of view can be monitored. In this case, when the relative velocity is zero, it is determined that the new block is connected. Assume that at time T4 subsequent to time T3, the user replaces the previously connected non-communication block 196 with a non-communication block 214 of another shape (the captured image 192d). A comparison of the depth image 198d at this time with the depth image 198c at a previous time T3 indicates that the shape of the image 215 of the non-communication block has changed.

When it can be detected that the non-communication block is replaced with another non-communication block of another shape as a result of such comparison of the images after volume difference with each other, by referring to the non-communication block information table 180 in a manner similar to that described so far Identify the shape of the new non-communication block. The connection relationship to the core is the same as that of the previously connected non-communication block to the core. Therefore, previous information can be used as it is. Therefore, by updating only the identification number of the previously connected block associated with the same element number "#N1" in the structure data to the identification number of the block identified at this time, thereby identifying the block set as described in the drawing in Shape 216 at time T4.

As an example of modifying a non-communicating block other than connecting a non-communicating block of another shape, there may be a case where the joint angle of a core included in the non-communicating block is changed. 13 is a diagram to aid in explaining the process of identifying the shape of a set of blocks modified due to a change in the joint angle of the core. The representation of the drawings is similar to that of Fig. 11 and Fig. 12 . Depicted in order from the left of the axis having the ordinate as the time axis are the images 192e-192g captured at the respective times, the depth images 198e-198g, and the shapes 218, 222, and 224 of the identified block sets. However, it is assumed that the shape of the set of blocks is different than that depicted in FIGS. 11 and 12 , and non-communicating blocks 228 and 230 are assembled respectively to include the upper and lower links of the communicating block 226 with markers and joints.

The non-communicating mass, after the volume difference, clearly appears as a subject in the depth image 198e generated based on the image 192e taken at time t1 when the joint was not bent. Accordingly, the structure analysis section 22 recognizes the shape 218 of the set of blocks as depicted in the drawing by assigning element numbers "#C1" and "#N1" respectively to one (series) of communication blocks assumed to be one block and non-communication blocks, associate the component numbers with the identification numbers of the respective blocks, and record the connection relationship between the communication blocks and the non-communication blocks as structural data.

Assume that at time t2, the user next bends the joints of the core (taken image 192f). A comparison of the depth image 198f at this time with the depth image 198e at a previous time t1 indicates a change in shape of the non-communicating block. The structure analysis section 22 has acquired the state of the cores contained by the non-communicating blocks based on the information about the cores which have been transmitted separately from the set of blocks. That is, the structural analysis section 22 also grasps the joint angles of the communication blocks. Therefore, when the joint angle of the communication block existing inside is changed at an angle corresponding to the modification of the non-communication block, it can be determined that the modification of the non-communication block is caused by the bending of the core.

It can thus be determined that there are actually multiple blocks that were initially identified as one block, rather than another block being reconnected. In this case, for example, by giving a new element number "#N2" to the non-communication block (the upper block in the drawing) whose inclination has been changed and using the element number as it is for the block whose inclination has not been changed "#N1", thereby correcting the structure data. When the component number is corrected, the connection relation to the core is corrected appropriately. The shape 222 of the set of blocks at time t2 can thus be identified as depicted in the figure.

In the example of the drawing, the distinction between the upper block and the lower block is clear according to the inclination angle and shape of the blocks. However, it may not be easy to detect splitting between the pieces when the angle of inclination is small or the pieces have a shape or material such that the gap therebetween is inconspicuous even when the angle changes. In this case, a block originally viewed as one block can be divided into two by, for example, setting a dividing plane 220 perpendicular to the axis of the core when the joint is not bent at the position of the joint and dividing the block by the dividing plane. blocks. In this case, the flat surfaces of the division plane 220 are assumed to be the upper surface of the lower block and the bottom surface of the upper block (the upper surface of the lower block and the bottom surface of the upper block initially contact each other), and in the subsequent shape identification process Make corrections.

As a result of the processing at time t2, the non-communication block is identified as consisting of two blocks. Therefore, even when returning to the state in which the joint of the block set at time t1 is not bent at subsequent time t3 (the captured image 192g and the depth image 198g), the information processing device 10 can recognize the state as two non-communicating A state in which blocks are linked to each other (block set 224). Since non-communicating blocks including links are individually managed, even when the joints of the core are subsequently bent and stretched, there is no need for a process of re-identifying the shape or the like.

In cases where multiple blocks are assembled into one body so that there is no temporal change in orientation or spacing, data management can be simplified by treating the blocks collectively as one block. On the other hand, even in the case where a plurality of blocks are assembled into one body, when the plurality of blocks are different from each other in color or texture and can thus be recognized as different blocks, it is possible to identify based on information about the color or texture of the block. Blocks are managed individually. When even the process of attaching and detaching blocks by the user needs to be correctly recognized, by individually managing each block, in particular, correction of the structure data can be limited to the changed parts. This is therefore advantageous in terms of processing efficiency.

In addition, when the state of the acquired block set is displayed as an object on the display device 16, or when structure data is saved as information on polygons and textures, low-polygon modeling with resolution coarsened can be performed to reduce processing load or memory consumption instead of reflecting the identified shape as-is. Alternatively, each non-communicating block may be further associated with another object model in the non-communicating block information table, and the object model associated with the parts of the non-communicating block may be rendered to replace the parts of the non-communicating block when displayed . Therefore, even when a block has a rough shape such as a rectangular parallelepiped or the like, the block is converted into a real object at the time of display and the object is displayed. Using the state of the block set, the processing performance of the information processing device 10, the memory capacity, and the like, according to the processing to be performed by the information processing section 30, whether or not the level of detail of the information is so lower, equal to, or higher is performed as appropriate Setting for the level of detail of the real block collection.

Incidentally, as depicted in FIGS. 11 to 13 , this embodiment captures temporal changes in block sets for both core and non-communication blocks. Therefore, various modes can be realized by storing changes in time as a history within a predetermined time and reading out and using the history as necessary. For example, when a user desires to restore an in-process block set being assembled to a state several stages ago, and the user makes a request via the input device 14, the state of the block set in the corresponding stage is displayed as a 3D object. Users can restore the collection of real blocks to a previous state while browsing the 3D object. When changing the virtual point of view relative to a set of blocks rendered as a 3D object through the input device 14, the previous state of the set of blocks can be checked from multiple directions. When the history is stored for a long time, block sets created in the past can be displayed as objects in each progress stage, and real block sets can be reproduced by assembling according to the objects.

Furthermore, information about blocks that were ever disassembled (such as the round cylinder type non-communicating block 196 in FIG. to manage. In this case, when another block is attached to the same position, the same element number is given to a plurality of non-communication blocks. However, flags can distinguish this information from past information. When the user later makes a request to restore the block set to a previous state, the objects of the block set in the previous state can be displayed by detecting and restoring the previous block from the structural data.

Next, a description will be made of the operation of the information processing device related to the identification of the state of the block set, which can be realized by the configuration described so far. 14 is a flowchart depicting a process for identifying a state of a set of blocks including non-communicating blocks. For example, the flowchart starts when the user powers on a block with a battery among the blocks in the block set 120 and inputs an instruction to start processing via the input device 14 by selecting an application on the information processing device 10 .

Incidentally, it is assumed that in parallel with the processing of the information processing apparatus 10 whose processing will be described below, a signal indicating the state of the core is transmitted at a predetermined timing from a block set that the user is assembling or lifting. Also, this flowchart focuses on changes in non-communication blocks, and assumes that changes in the state of the cores are acquired separately from signals indicating the states of the cores and the like. First, the structure analysis section 22 causes the camera 122 to start photographing a block set (S10). At the same time, a predetermined initial image is displayed on the display device 16 by cooperation between the information processing section 30 and the display processing section 32 (S12). The image displayed at this time may be a live video captured by the camera 112, an image created in advance as a part of the application (such as a game image), or the like.

In time step t=0 (t is an integer indicating the passage of time in ascending order), when the core information receiving section 20 receives the information transmitted from the block set, the structure analyzing section 22 identifies the core in the three-dimensional space based on the information pose and shape of (S14 and S16). The location of the nucleus can also be identified when the location of the nucleus is identified by means other than the captured image. Meanwhile, the structure analysis section 22 acquires the entire image and position of the block set by, for example, acquiring a captured image from the camera 122 and generating a depth image based on the captured image (S18). Things other than the collection of blocks, the background, the user's hand, etc. may appear in the captured image. Therefore, a process of removing these images is performed in a certain stage.

Ordinary methods such as foreground extraction, simultaneous localization and mapping (SLAM) methods, color segmentation, dictionary-based object recognition, etc. can be used for this process. Identify how the core is positioned relative to the overall image of the block set thus extracted (S20). Specifically, as described above, a part having a characteristic shape, color, or pattern, etc., such as a marker of a nucleus, etc., is detected from an image of a block set, and the position of the nucleus is determined with reference to the part. How the kernel is seen from the camera is then identified based on the shape and pose of the kernel identified in S16. This is an image of the block collection when there are no non-communicating blocks. Therefore, the image and position of the non-communicating block are acquired by taking the volume difference between the image of the block set in the absence of the non-communicating block and the overall image of the real block set (S22).

When the state of the non-communicating block is acquired in the previous time step t-1, the state of the non-communicating block in the previous time step t-1 and the current image of the non-communicating block are compared with each other (S24) to check whether there is a change (S26). When there is a change in the shape of the image (Y in S26), a process of identifying the shape of the non-communication block and updating the structural data of the block set is performed (S30). Incidentally, at time step t=0, the status itself of the non-communication block (the status acquired in S22) is regarded as changed, and the structural data is newly created based on the status of the non-communication block.

As for the processing of S30, as described with reference to FIGS. , and processing in the case of a change in the shape of the non-communication block due to bending of the joint of the core, and the processing to be performed for each case is performed. Specifically, when a non-communication block is added, element numbers are given and are each associated with a shape, connection direction, etc., and a connection relationship to the core is recorded. For example, when a non-communicating block is removed, the non-communicating block is managed by deleting the non-communicating block from the structural data or setting a flag indicating that the non-communicating block is removed. When performing replacement with another non-communication block, the shape, connection direction, etc. of the block corresponding to the corresponding element number are updated.

When the shape changes due to the bending of the joints of the core, when the non-communication blocks are not managed in a manner corresponding to the respective links with the joints inserted therebetween, these blocks are set as separate blocks, the new A component number is given to one of the blocks, and the shape and connection direction of each block are updated. The non-communication blocks are thus managed individually. Furthermore, even when the shape of the core or the non-communicating block does not change, the image of the non-communicating block changes according to the change in position or attitude. In this case, it is checked whether at least a part of the assumed shape of the part can be identified because the shape of the part cannot be determined due to hiding of the part or the like. When a portion of the assumed part's shape is identified, the structure data is updated accordingly. Shapes are thus identified in a time-evolving manner. As described above, structural data can be represented as model data of 3D graphics.

When there is no change in the shape of the image in S26 (N in S26), at least the state of the non-communication block does not change, and therefore, the processing of S30 is not performed. Until the user performs an input for ending the processing, the processing from S16 to S30 is sequentially repeated while the time step t is incremented (N in S32 and S28 ). For example, when an instruction input for ending the processing is performed by turning off the power to the block set, the processing ends (Y in S32 ). By the way, in the process of S20, when the entire core is hidden by non-communication blocks when viewed from the camera, positioning of the core cannot be performed. In this case, a display may be made that prompts the user to change the orientation of the set of tiles so that the kernel enters the camera's field of view. Alternatively, the camera coordinate system relative to the kernel can be assumed such that the overall image of the kernel does not protrude from the overall image of the block set, and can be corrected in subsequent time steps.

Incidentally, in the description so far, attention has been mainly paid to the change of the shape of the non-communication block. In case of a color or texture change of a non-communicating block, the structural data can also be updated by a similar process. However, in this case, first by locating the core in the overall image as well as in the depth image, thereby removing components of the core from the overall image of the block set, and then by feeding back the regions of the image of the remaining non-communicating blocks into the captured Color images to identify changes in color or texture.

Fig. 15 shows the relationship between a set of blocks and a display that can be realized by the modes described so far. In the example of the drawing, a block set 240 including communicating blocks and non-communicating blocks is presented as a 3D object 242 as recognized by the information processing device 10 and displayed on the display device 16 . Since the camera 122 acquires the shape, posture, color, etc. of the non-communication block, it is possible to recognize things not registered in the non-communication block information table (such as things made by the user himself/herself). For example, even when the user writes the face of the doll assembled as the block set 240 in the drawing by a felt-tip pen or pastes a sticker to the doll, the doll having a face similar to the written face may not stick Dolls that are glued to are also rendered as 3D objects.

In this example, a set of blocks existing in the field of view of the real camera 122 is flipped horizontally by mirror processing, and represented as an object. Because the set of blocks 240 faces the camera 122 on the display device 16, the 3D object 242 on the display is displayed such that the set of blocks 240 appears in the mirror. On the other hand, the information processing device 10 recognizes the three-dimensional shape of the block set 240 as described above, and can thus freely change the virtual viewpoint with respect to the 3D object 242 . Thus, even an image on the doll's back side (which at this point in time is the blind spot from camera 122 ) can be displayed without rotating block set 240 . However, depending on the shape identification process so far, details of the shape on the back side cannot be determined.

The description so far is based on identifying the state of the set of blocks actually assembled. However, it is possible to allow virtually connected things that are not actually connected. The screen shown in FIG. 15 displays images 244 and 246 of options for items to be provided with a 3D object 242 reflecting a collection of blocks 240 . When the user selects an image and causes the 3D object 242 on the screen to have a corresponding item via the input device 14 or by moving the set of blocks 240, for example, the information processing device 10 may update the structure data to indicate that the corresponding item is connected to the identified block The corresponding location of the set.

The three-dimensional shape of an item as an option is stored in the block information storage section 24 or the model data storage section 26 of the information processing device 10 . Therefore, in the subsequent processing, regardless of whether the item on the screen is actually connected to the set of blocks or connected virtually, the item can be represented so that it is held by the 3D object 242 according to the movement of the set of blocks in real space. to move in the virtual world. When the three-dimensional shape information is acquired by capturing a real thing or the like made by the user himself/herself with a 3D scanner, the three-dimensional shape information serves as a substitute for the above-mentioned items. That is, the non-communicating blocks can be displayed as if the non-communicating blocks are connected on the screen even when the real thing of the non-communicating blocks is not actually connected.

The external appearance itself of the collection of blocks can be created in the virtual world on the display in a similar way. FIG. 16 shows the relationship between a block set and a display when an appearance is set for the block set. In this example, the set of blocks 248 itself is a simple configuration (such as, for example, only communication blocks, etc.) compared to the case of FIG. 15 . When the state of this block set 248 is recognized, the information processing device 10 displays the recognized state of the block set as it is or as a 3D object 250 on the display device 16 after the block set is flipped horizontally. Then, as in the case of the article of FIG. 15 , images 252 a , 252 b , and 252 c are displayed as options of candidate 3D objects for each part (such as, for example, a face, etc.).

The user performs a selection input from the image 252a, 252b or 252c for each block in the set of blocks or each part formed of a plurality of blocks, and thereby virtually makes the connection to the 3D object 250 . The external appearance is thus complete. It is thus possible to freely create 3D objects with various external appearances, and further move the 3D objects or change the virtual viewpoint according to the movement of the real block set 248 . Furthermore, virtual 3D objects can be coupled to each other on the display by connecting multiple blocks thus creating their virtual appearance.

For example, it is possible to implement a mode in which when a block set and an external appearance corresponding to the block set are created for each part in the complete form of the target 3D object and finally the block set is connected, the display has a function in which even detailed parts are set The complete form of the external appearance. Alternatively, the process of storing a 3D object with a once-created virtual external appearance and using the same set of blocks to create a 3D object with another external appearance can be repeated, and thus created multiple 3D objects can be displayed only in the virtual world coupled with each other. In this case, since the 3D objects finally associated with the block set are interlocked with the block set, a mode in which all 3D objects coupled to each other move according to the block set can be realized.

In the examples of FIGS. 15 and 16 , the movement of the set of blocks is reflected in the 3D objects in the virtual world. Conversely, the movement of the 3D object can be reflected in the movement of the set of blocks. For example, a 3D object representing a set of blocks assembled by the user or a 3D object virtually created by the user so as to correspond to the set of blocks appears as a character in a game run by the information processing section 30 . When the user performs game operations using the input device 14 and the 3D objects move accordingly, the set of real blocks is also caused to move. In this case, the drive control section 34 transmits a signal for controlling the drive section 148 of the block set in a manner corresponding to the movement of the 3D object.

In the example of FIG. 16 , a mode in which a user applies a 3D object in units of one or more blocks is assumed. Therefore, the correspondence between real blocks and virtual objects is similar to that of non-communication blocks, and management can also be performed in a similar manner. That is, when the joint angle of the real block set is changed, the modified position caused by the change is also clear in the 3D object, and thus, the change can be easily reflected on the screen display. The same is true for the case of reflecting movement of 3D objects in a set of blocks. On the other hand, when a 3D object is associated with the whole of the assembled block set, the corresponding positions of the block set and the 3D object need to be properly set in order to properly interlock the block set and the 3D object with each other.

Fig. 17 shows the relationship between a block set and a display in the case where one 3D object is associated with the assembled block set. This example assumes a situation where a user creates a set of tiles 260 assumed to represent an elevator truck, and selects an elevator truck 262 as a 3D object on the display. The set of blocks 260 includes a number of joints 264a, 264b and 264c provided to the blocks assumed to represent elevators, and also includes wheels 266a, 266b, 266c and 266d fitted to the blocks acting as platforms.

When the user bends and stretches the joints of the block set 260 and moves the lift truck 262 on the screen in the same manner, for example, it is necessary to set the position of the lift truck 262 at which the bending and stretching of the joints 264a, 264b and 264c will be reflected . Furthermore, when the movement of the lift truck 262 on the display is to be reflected in the actual movement of the block set 260, the role of each wheel, rotation speed, steering angle, etc. needs to be determined from the movement on the display. The block set 260 in this embodiment is freely assembled by the user. Therefore, the parts that are expected to move as parts of the elevator, the front-rear relationship of the elevator wagon, and the like largely depend on the intention of the user. Accordingly, a method of setting the correspondence between the movement of such a freely assembled thing and the movement of a 3D object prepared in advance will be described.

FIG. 18 shows information necessary for associating a block set and movement of a 3D object with each other. The left side in the figure is a schematic diagram of a block set 270 corresponding to the block set 260 in FIG. 17 . The right side in the figure is a schematic diagram of a lift truck 272 corresponding to the lift truck 262 of the 3D object in FIG. 17 . As depicted in the drawing, a coordinate system used when the information processing apparatus 10 recognizes a shape is set for the block set 270 , and a local coordinate system with respect to the 3D model is set for the lift truck 262 . The two coordinate systems are set independently of each other. The orientation of the set of blocks 270 and the lift truck 272 in the corresponding coordinate system is thus changed.

For ease of understanding, the example of FIG. 18 shows the shape in a two-dimensional plane formed by the x-axis and the y-axis, that is, when the block set 270 and the lift truck 272 are defined in a state parallel to the x-axis of the corresponding coordinate system shape of the side surface. However, the orientations of the set of blocks 270 and the lift truck 272 with respect to the respective coordinate systems are opposite to each other with respect to the x-axis. Here, the three joints of block set 270 are denoted as RJ1, RJ2, and RJ3, and the three joints of lift truck 272 are denoted as VJ1, VJ2, and VJ3. Furthermore, the four wheels of block set 270 are denoted RA1, RA2, RA3, and RA4, and the track tracks of the lift wagon are denoted VA1 and VA2. Incidentally, the wheels RA3 and RA4 and the crawler track VA2 are on the side surface opposite to the display surface, and are normally hidden. However, in FIG. 18 , wheels RA3 and RA4 and track track VA2 are depicted offset as indicated by dashed lines, and reference numerals are depicted in parentheses.

The simplest method for interlocking the block set 270 and the lift wagon 272 to each other may be to associate the joints RJ1, RJ2 and RJ3 of the block set 270 with the joints VJ1, VJ2 and VJ3 of the lift wagon 272, respectively, and the wheels RA1 and RA2 are associated with track track VA2 not depicted in the figures, and wheels RA3 and RA4 are associated with track track VA1 . However, in this example, the location of the joints of the lifts differs between block collection 270 and lift wagon 272 . Thus, for example, when the corresponding joints of block set 270 and lift truck 272 are individually bent at the same angle, the movement expected by the user may not occur.

In addition, generally, the joint angle of the block assembly 270 has a physically movable range, and the joint of the lift truck 272 as a model also has a movable range. When these constraints are not considered, there may be situations where the 3D model bends at impossible angles or where the joint angle of a block collection reaches a limit and the block collection does not move further. Additionally, due to the different orientations relative to the coordinate system, there may be instances where the lift truck 272 on the display backs up when the collection of blocks 270 advances. In order to prevent this problem, the interlocking of real things and things in the virtual world is coordinated by performing processes of setting a common coordinate system, setting corresponding positions, and associating specific movements of the corresponding positions with each other.

FIG. 19 is a flowchart depicting a processing procedure in which the information processing device 10 associates a block set and a movement of a 3D object with each other. First, after the user assembles the set of blocks into a desired shape, the user inputs an instruction request via the input device 14 to start the association with the 3D object. When the information processing section 30 of the information processing device 10 receives the request (S40), the information processing section 30 acquires information on the structure of the assembled block set from the structure analyzing section 22 (S42). Then, a suitable model is extracted as a candidate from among 3D objects whose presentation data has been prepared in the model data storage section 26 based on the shape of the block set or the like (S44).

The model data storage section 26 associates various kinds of object models such as lift trucks and the like with metadata indicating characteristics of each model, and stores presentation data and metadata associated with each other. When roughly classified, metadata includes features as things, structural features, and external features. Features that are things include, for example, objects (such as people, animals, vehicles, food, etc.), proper nouns (such as the name of a movie, animation, or game in which a character appears, the name of a character, etc.), and associated time periods (such as original age, middle age, etc.) age, current age, future, specific year, etc.). Structural features include the number of joints, the movable angle and degree of freedom of each joint, the length and thickness of links, the connection relationship of joints, driving force, tire diameter, etc.

External features include color, surface shape, number or volume of non-communicating blocks, coverage ratio of non-communicating blocks in a block set, number of LEDs or display devices when provided, type of display device, etc. In the model data storage section 26, these various characteristics are associated with each model. The more features associated, the higher the accuracy in extracting suitable candidates. However, not all features are intended to be correlated. In S44, the information processing section 30 extracts a candidate model having a high degree of similarity to the real block set from information on the shape or structure of the block set (acquired in S42) based on structural features and external features.

For example, assuming that N _RJ is the number of joints of a block set, N _RA is the number of rounds of a block set, N _VJ is the number of joints of a 3D object, and N _VA is the number of rounds of a 3D object, the similarity is calculated by the following formula estimated value.

Estimated value of similarity = (N _RJ -N _VJ )×w _J +(N _RA -N _VA )×w _A

where _wJ and _wA are the weights for the estimate of the number of joints and the estimate of the round and are determined according to the importance of the estimate of the number of joints and the estimate of the round. The closer the estimate is to zero, the higher the similarity. Furthermore, when the estimate is positive, the estimate indicates a tendency for the set of blocks to have a large number of joints or rounds. When the estimate is negative, the estimate indicates a tendency for the 3D object to have a large number of joints or rounds. When there is a 3D object such that the estimated value is zero, the 3D object is extracted as the most likely candidate model.

Furthermore, when there are a plurality of 3D objects for which evaluation values having the same absolute value but different in sign from each other are obtained, it is preferable to extract a 3D object providing a negative evaluation value. This is because, as the number of joints or wheels of a 3D object increases, more detailed movement can be expressed on the screen, and movement of a collection of blocks can be expressed richer. Prior to estimation of such similarity, candidates may be narrowed down based on features selected by the user from among features that are things. Features as things, structural features, and external features may be combined with each other as appropriate to extract candidates, or the user may be allowed to also specify features other than features as things.

The information processing section 30 displays the plurality of candidate models thus extracted on the display device 16, and receives a selection input by the user via the input device 14 or the like (S46). An example of a model thus selected is the lift truck 272 depicted in FIG. 18 . Next, a coordinate system common to the block set and the objects in the image is set (S48). Thus, the parameters determining the pose and direction of travel of a collection of blocks can be handled similarly in 3D objects as well. For example, when the block set 270 in FIG. 18 advances in the negative direction of the x-axis, the coordinate system is set so that the lift truck 272 of the 3D object also advances in the negative direction of the x-axis. Furthermore, a common coordinate system allows for a common definition of joint angles and correct determination of which link moves relative to the joint when the joint angle changes.

Various kinds of parameters defined in the common coordinate system undergo coordinate transformation into values in the local coordinate system originally set for the 3D object, and are thus reflected in the rendering of the 3D object. When the shapes of the block set and the 3D object are similar to each other and their front and rear etc. are clear, the information processing section 33 sets a common coordinate system so that both the block set and the 3D object have the same orientation. Alternatively, the user may adjust the orientation of the 3D object on the screen so that the orientation of the 3D object becomes the same as that of the block set, and the information processing section 30 may set a coordinate system with this orientation as a reference. Therefore, even a set of blocks whose front and back are not clear can be interlocked with the 3D object in the orientation of the user's intention.

Next, corresponding positions of the block set and the 3D object are set (S50). When the number of joints and the connection relationship of the block set and the link of the 3D object coincide with each other, the processing may be jointly performed by the information processing section 30 . Specifically, based on the structure of the block set acquired in S42 and the structure of the object model selected in S46, the information processing section 30 geometrically derives the corresponding joints of the block set and the object model, and combines the block set and the object model Corresponding joints are associated with each other. Alternatively, a setting screen is displayed to allow the user to make settings. The associated locations in this case are typically joints or wheels in FIG. 18 . However, the user is allowed to freely set a corresponding point on the 3D object side for a position desired to reflect a change (such as bending, rotation, displacement, etc.) of a block set. This makes it possible to move animals without any joints (such as molluscs, etc.), or to have things that do not bend in an actuality bow.

As a side note, the associated locations do not necessarily need to be in a one-to-one relationship. That is, multiple joints of a set of blocks may be associated with one joint of a 3D object, or multiple joints of a 3D object may be associated with one joint of a set of blocks. The same holds true for wheels. In the case where the overall structure of the block set and the 3D object can be made consistent with each other by treating a plurality of joints as one joint even when the block set and the 3D object have different numbers of joints, the information processing section 30 can in this way Group connectors.

In this case, for example, one group of joints of one of the block set and the 3D object is associated with one joint of the other of the block set and the 3D object. At this time, a change in the joint angle of the latter is assigned as a movement of one group of joints of the former. A specific example will be described later. Regardless of whether the joints are grouped or not, the information processing section 30 sets the correspondence in consideration of not only the block set and the general structure of the 3D object but also the movable angles of the respective joints. For example, joints with the same movable angle are associated with each other. Accordingly, similarities between joints can be estimated from various viewpoints such as overall structure, movable angle, etc., and joints whose estimated values are higher than a threshold can be associated with each other. Information on the corresponding position set by the information processing section 30 or the user is stored in the corresponding information storage section 28 .

Next, movement correspondence is set for the positions thus associated (S52). That is, setting is made as to whether to reflect changes in joint angles as they are, or when joints are not associated on a one-to-one basis, a ratio in which changes in joint angles are allocated, etc. is set. When the structures of the block set and the 3D object coincide with each other and the movable angles of the corresponding joints are equal to each other, the joint angles can basically be regarded as similar to each other, and thus the information processing section 30 is set in this way. When structures are made similar to each other by grouping joint angles, a change in one joint angle is assigned to movement of joints belonging to one corresponding group, as described above. At this time, the allocation ratio may be determined according to the ratio between the movable angles of the corresponding joints.

When the movement correspondence can be set in this way with rules set in advance, the information processing section 30 can make the setting. In addition, a setting screen is displayed to allow the user to freely set the movement correspondence, or to correct the already set correspondence. Also, when the movement of the vehicle of the 3D object is reflected in the movement of the vehicle of the block set, the speed of the vehicle of the 3D object on the display and the rotational speed of the wheel shaft of the block set are associated with each other. This relationship changes according to the diameter of the wheel connected to the set of blocks. Thus, when the diameter of the wheel is known, the information processing section 30 may be operable to determine the rotational speed of the wheel axle of the set of blocks. When the diameter of the wheel is unknown, necessary parameters are acquired when the user actually moves the set of blocks, as will be described later. The processing of S50 to S52 is repeated for all positions to be associated (N in S54). When all positions are associated, the process ends (Y in S54).

FIG. 20 depicts an example of a screen displayed on the display device 16 to receive a model selection input by the user in S46 in FIG. 19 . The model selection reception screen 280 includes images of a plurality of models (ie, “Model 1 ,” “Model 2 ” and “Model 3 ”), a character string 282 prompting for selection input, and a cursor 284 movable by the input device 14 . "Model 1", "Model 2" and "Model 3" are candidate models extracted in S44 in FIG. 19 , and are as a result of filtering by at least one of features as things, structural features, and external features, as above. When the user specifies "elevator wagon" as a feature of a thing, for example, the extracted models are all lift wagons. In this case, before the model selection acceptance screen 280 is displayed, a screen for accepting selection of "elevator truck" is also displayed.

The extracted number is not limited to three. All things that match the criteria can be extracted. Alternatively, the models are ranked by using the similarity estimation values described above, and a predetermined number of high-ranked models may be extracted. In this case, the model selection receiving screen 280 may arrange images of models in descending order of ranking from the left. In FIG. 20 , Model 1”, “Model 2” and “Model 3” are all lift wagons, but the shapes of the lift parts of the models (such as the number of joints, etc.) are different from each other. For example, the user indicates the desired A model or a model similar to a set of blocks and by means of the input device 14 performs a definite input to select a model.

Furthermore, means other than means for selecting input via cursor 284 may be used to select a model. For example, there is proposed a technique of photographing a user by the camera 122 and detecting the position of the user's fingertip from a depth image or the like and further detecting the position indicated on the display screen. When this technique is used and the user indicates a model to be selected, the information processing device 10 can recognize the selected model. User input on the respective setup screens depicted in FIGS. 21 and 22 may be implemented in a similar manner.

FIG. 21 depicts an example of a screen displayed on the display device 16 to set the coordinate system common to the block set and the selected model in S48 in FIG. 19 . In this example, the user specifies a common coordinate system by adjusting the orientation of the 3D objects on the screen according to the orientation of the set of blocks. Therefore, as described above, the front-back orientation of the overall structure, the positional relationship of each link with respect to the joint, the definition of the joint angle, etc. can be unified. The model display orientation setup reception screen 290 includes: an image 292, where the front of the lift truck is oriented left; an image 294, where the front of the lift truck is oriented right; a string 296, which prompts a description of the orientation; and a cursor 298, which can be accessed by The input device 14 moves.

This example assumes that the user selects "Model 1" on the model selection reception screen 280 in FIG. 20 . Displays images of objects looking at the same model from both left and right directions. For example, the user places block set 260 as depicted in the figure, indicates by cursor 298 an image depicting a similar orientation to the block set or depicts an orientation that the user believes is the same as that of the block set, and via input device 14 Execute OK input. In the example of the figure, the left-oriented image 292 is selected. Then, the information processing section 30 sets a coordinate system for the block set 260 and the 3D object such that the orientation of the block set 260 (which is acquired by the structure analysis section 22) and the orientation of the 3D object in the selected image are defined to be the same orientation. By the way, as in this example, when the block set 260 and the 3D object front and rear can be estimated based on structures such as the position of the elevator relative to the platform vehicle, etc., the information processing section 30 can be set as described above coordinate system.

FIG. 22 depicts an example displayed on the display device 16 to set the corresponding positions of the block set and the 3D object in S50 in FIG. 19 . The corresponding position setting screen 300 includes: an image 302 of a 3D object; a command list section 304 ; a character string 306 prompting setting input; and a cursor 308 movable by the input device 14 . The 3D objects on the screen are in the same orientation as the block set after passing through the model display orientation setting reception screen 290 . Therefore, the user can easily understand the correspondence with the block set. The corresponding location setting screen 300 may further display an image 310 of the block set 260 .

The image 310 is an image captured by the camera 122 or an image in which the state of the block set 260 , which is recognized by the information processing device 10 , is represented as a 3D object. In the former case, when the user and the camera 122 are on opposite sides of each other with respect to the tile set 260 , the captured image is flipped horizontally to reproduce the tile set 260 as viewed from the user. In the latter case, when non-communication blocks make it difficult to confirm the structure of the core, only the image of the core may be displayed.

First, the user moves the desired associated joint (eg, arrow A) in tileset 260 by, for example, bending and stretching the joint. Movements are identified by the structural analysis section 22 . An input specifying the connector to be associated (for example RJ2) on the side of the block set is thus achieved. Next, the user designates a joint (for example, VJ3 ) to be associated on the 3D object side in the image 302 of the 3D object by the cursor 308 , and performs a determination input by the input device 14 . The joint RJ2 of the block set 260 and the joint VJ3 of the 3D object are thus associated with each other. The information processing section 30 records the correspondence of these joints and stores it in the corresponding information storage section 28 .

This is done for all joints of the block set 260 that the user desires to move joints. Also, similar settings can be made for wheels. When the block set 260 is actually moved based on such set correspondence, the user may be allowed to check the movement of the 3D object. Then, you can allow corrections as needed. For this, the command list section 304 displays a GUI such as a "prev" button for canceling the immediately preceding setting of the corresponding position and setting it again, for confirming the current setting and setting the next corresponding position "next" button, "stop" button for storing all the settings at the current time and ending the setting processing, etc.). In addition, a "menu" button for making a transition to a screen displaying various kinds of setting screens as a menu is also displayed. Also, after setting a movement correspondence to be described later, movement confirmation processing can be performed, and whether to cancel the setting can be confirmed by a "forward" button or a "backward" button accordingly.

By the way, when the joints of the 3D objects that can be associated with the specified joints of the block set 260 can be automatically narrowed according to the movable angles of the joints or structural constraints, the information processing section 30 can suggest the joints to the user as candidate. That is, at the point of time when joints to be associated on the block set side are specified, joints that are candidates for association in the image 302 of the 3D object displayed on the corresponding position setting screen 300 are changed in color or the like. , and allow the user to select one of the connectors. Therefore, even when the user is allowed to make free correspondence settings, impossible association can be avoided.

Furthermore, as described above, multiple joints of a set of blocks are allowed to be associated with one joint of a 3D object, or one joint of a set of blocks is allowed to be associated with multiple joints of a 3D object. In the example of FIG. 22 , two markers 312 indicating that the two joints of the set of blocks are associated with the joint VJ3 of the 3D model are provided to the joint VJ3 of the 3D model. This is achieved, for example, by the operation of associating the joint RJ1 of the block set 260 with the joint VJ3 of the 3D object by moving the joint RJ1 of the block set 260 and further associating the joint RJ2 with the same joint VJ3 of the 3D object by moving the joint RJ2.

On the other hand, the joints VJ1 and VJ2 of the 3D object are grouped into one group and surrounded by an ellipse 314 indicating that the joints VJ1 and VJ2 of the 3D object are associated with one joint of the block set. In order to thus group multiple joints of a 3D object into a group, the command list section 304 also displays a GUI of a "group" button for presenting an oval 314 indicating the group. For example, the user moves joint RJ3 of block set 260 . Then, the "Group" button on the screen is selected by the cursor 308, and an ellipse 314 is drawn so as to enclose the joints VJ2 and VJ1 of the 3D object. Joint RJ3 of block set 260 is thus associated with joints VJ2 and VJ1 of the 3D object.

The information processing section 30 may determine some suitable grouping patterns from viewpoints such as the overall structure of a block set of 3D objects, movable angles, etc., and display the grouping patterns as candidates on the corresponding position setting screen 300 . In this case, the user determines the group by selecting one pattern from the candidates. Alternatively, grouping pattern candidates may be created as metadata of the 3D object and displayed on the corresponding location setting screen 300 for user selection. Incidentally, when the image 310 of the block set is displayed on the corresponding position setting screen 300 , the joint of the block set can be designated on the image 310 by the cursor 308 . Furthermore, associated joints may be displayed in the same color in the image 302 of the 3D object and the image 310 of the block set, or their correspondence may be clearly indicated by connecting the associated joints to each other by means of straight lines.

Next, the movement correspondence of such associated joints will be described. In S52 in FIG. 19, correspondence is set by the information processing section 30, the user, or the cooperation of both the information processing section 30 and the user. As a basic pattern, FIG. 23 depicts a case where joints of block sets and 3D objects are associated with each other on a one-to-one basis and the joints move to the same angle. The left side of FIG. 23 depicts a joint component 320a of a collection of blocks, and the right side of FIG. 23 depicts a corresponding joint component 322a of a 3D object. That is, when the joint part 320a of the block set is bent by the angle θ from the state where the joint is not bent, the corresponding joint part 322a of the 3D object is also bent by the angle θ. Depending on the processing performed by the information processing section 30, when the joint part 322a of the 3D object is bent at an angle θ, the corresponding joint 322a may be driven by the actuator of the driving section 148 as a block set via the driving control section 34 etc. are bent according to the angle θ.

Based on this mode, various settings are realized according to constraints such as movable angles, the number of joints to be associated, and the like, user's intention, and the like. FIG. 24 depicts an example of the correspondence between the angles of the respective joints in the case where two grouped joints are associated with one joint. The left side of FIG. 24 depicts a joint component 320b of a collection of blocks, and the right side of FIG. 24 depicts a corresponding joint component 322b of a 3D object. In this example, two joint parts 320b of the block collection are associated with one joint part 322b of the 3D object. Then, when one of the joints of the block set is bent at angle _θ1 and the other joint is bent at angle θ2, the corresponding joint part 322b of the 3D object is bent at angle ( _{θ1 +} θ2 ₎ , which is angle _θ1 and angle _The sum of θ2. That is, the change of the joint angle by two joints of the block set is realized by one joint of the 3D object.

This mode is valid when the movable angle of each joint of the block set is less than the change in joint angle required by the corresponding joint of the 3D object. Changes in three or more joint angles of a collection of blocks can be summed. The angular correspondence is also similar in the case where multiple joints of a 3D object are grouped and associated with one joint of a collection of blocks. This mode is valid when the number of joints of the block collection is less than the number of joints of the 3D object.

Also, conversely, when the joint part 322b of the 3D object is bent at an angle (θ ₁ +θ ₂ ), the two corresponding joint parts 320b of the block set may be bent at corresponding angles θ ₁ and θ ₂ . In this case, the information processing section 30 may determine the angle _θ1 according to constraint conditions such as movable angles of corresponding joints, actual movement of what the block set is supposed to represent (ie, what the 3D object represents), etc. The ratio between and θ ₂ . When the movable angles of the two joints of the block set are in a ratio of 1:2, for example, the angles θ ₁ and θ ₂ are also set in a ratio of 1:2.

The same holds for the case where a change in the angle of one joint of a set of blocks is assigned to the angles of multiple joints of a 3D object. When assuming the lift truck depicted in Fig. 22, for example, it is natural that the hook connected to the joint VJ1 of the 3D object is oriented vertically downwards at all times. Therefore, the information processing section 30 calculates the angles θ ₁ and θ ₂ so as to maintain this state. This distribution rule is created together with the data of the 3D object. By the way, when the sum of the change of the angle of one of the block set and the 3D object is reflected in the change of the other's angle and the change of the angle of one of the block set and the 3D object is reflected so that Multiple joints of a block set may also be associated with multiple joints of a 3D object when modes of change of multiple angles assigned to one another are combined with each other.

FIG. 25 depicts another example of the correspondence between the angles of the respective joints where two joints are grouped and associated with one joint. The left side of FIG. 25 depicts a joint component 320c of a collection of blocks, and the right side of FIG. 25 depicts a corresponding joint component 322c of a 3D object. It is assumed in this example that the two joints of the block collection have different axes of rotation from each other, that one of the two joints changes angularly about an axis perpendicular to the plane of the drawing, and that the other of the two joints changes about the linked axis in change in angle. On the other hand, it is assumed that one joint of a 3D object whose joint is associated with two joints of a block set is a joint with two degrees of freedom enabling angles to be changed about two axes corresponding to the axes of the block set.

Thus, when one joint of the joint part 320c of the block set changes by angle Θ and the other joint changes by angle ψ, the corresponding joint part 322c of the 3D object changes by angle Θ and ψ about the corresponding axis. Also, conversely, when the joint part 322c of the 3D object changes by angles θ and ψ about the corresponding axis, the two corresponding joints 320c of the block set may change by θ and ψ, respectively.

FIG. 26 shows a case where changes in angles of joints associated with each other on a one-to-one basis are made different from each other. The left side of FIG. 26 depicts a joint component 320d of a collection of blocks, and the right side of FIG. 26 depicts a corresponding joint component 322d of a 3D object. In this example, when the joint of the set of blocks changes by angle Θ, the corresponding joint of the 3D object changes by angle 3Θ, which is three times the angle Θ. When a movable angle of a joint of a block set is smaller than a change in angle required by a corresponding joint of a 3D object, for example, even when the block set moves to a limit, the 3D object cannot move as desired. There are also cases where it is desired that a 3D object move dynamically without much work of moving a collection of blocks.

This problem can be solved by changing the joint angle of the 3D object by an angle obtained by multiplying the change of the joint angle of the set of blocks by a predetermined value greater than 1, as depicted in the figure. Alternatively, the joint angle of the 3D object may be changed by an angle smaller than the block set by multiplying the change in the angle of the block set by a predetermined value smaller than 1. For example, this mode is effective when a 3D object is expected to move minutely with higher precision than the movement of a hand moving a set of blocks, as in an operation such as a manipulator. It is also sufficient to make a similar setting in the case where a change in the angle of the joint of the 3D object is reflected in a change in the angle of the block set. Incidentally, the mode in which the change in the angle of one of the joints is multiplied by a predetermined value to obtain the change in the angle of the other joint may be combined with the modes depicted in FIGS. 24 and 25 . In this case, the joints for all groups may be multiplied by the same value, or may be multiplied by different values depending on the joints.

FIG. 27 depicts another example of associating one joint of a collection of blocks with joints of a 3D object. The left side of Figure 27 depicts the joint component 320e of the block set. The right side of Figure 27 depicts the corresponding joint part 322e of the 3D object. In this example, one joint part 320e of the block collection is associated with three joint parts 322e of the 3D object. Then, when the joints of the collection of blocks are bent by angle Θ, the three corresponding joints of the 3D object are each bent by the same angle Θ.

Therefore, even in the case where the block set has a simple configuration, the movement of the block set can be reflected in a wide area of the 3D object. For example, when one joint of a block set is associated with all joints of a snake having a shape formed by further connecting a plurality of joint parts 322e of a 3D object, the movement of the snake, such as forward movement, may be even It is represented by a block set composed of only the joint part 320e in FIG. 27 . As described above, when a moving image in which a serpent is moved in various ways is stored, the serpent can be used as a character in a game or animation at an arbitrary timing later. This progressive image expression can be achieved by this simple collection of blocks with joint component 320e. Incidentally, in this mode, the associated joints in the 3D object are not limited to the adjacent relationship with each other depicted in the drawings, but may be a plurality of joints existing at distant positions of the 3D object.

FIG. 28 depicts an example displayed on the display device 16 to set movement correspondence between a set of blocks and a 3D object in S52 in FIG. 19 . The movement correspondence setting screen 330 is a screen acquired by overlay displaying a dialog box 332 for inputting movement correspondence on the correspondence position setting screen 300 depicted in FIG. 22 every time the user sets a correspondence position on the screen. This example assumes that after the joints RJ1 and RJ2 of the block set are grouped and associated with the joint VJ3 of the 3D object, the allocation ratio in the case where a change in the angle of VJ3 is assigned to a change in the angle of RJ1 and RJ2 of the block set is set . To this end, dialog box 332 displays a string 334 prompting for a scale and a text box 336 for entering each numerical value of the scale.

When the user enters a specific numerical value into the text box via the input device 14, etc., the information processing section 30 further associates the ratio with the joint RJ1 and RJ2 of the block set and the joint VJ3 of the 3D object, and stores it in the corresponding information storage section 28 in. The information set in dialog 332 is not limited thereto. Necessary information among the associations in FIGS. 23 to 27 is allowed to be sequentially set according to whether or not joints are grouped. However, as described above, when the information processing section 30 can associate movements by rules set in advance, setting by the user can be omitted. Alternatively, the user may be allowed to correct the association performed by the information processing section 30 in a similar dialog box.

FIG. 29 depicts an example of a data structure of information on correspondence between block sets and corresponding positions of 3D objects and movements of the respective corresponding positions, which information is stored in the correspondence information storage section 28 . The corresponding information table 340 includes: a block set information section 342 depicting the corresponding position and movement of the block set side; and a 3D object information section 344 depicting the corresponding position and movement of the 3D object side, the corresponding position and movement of the 3D object side are The position of the block set corresponds to the movement, and the position and movement of the block set are entered in the section. In this example, the joints of the collection of blocks are associated with the joints of the 3D object, and other correspondence between changes in the angles of these joints is set.

Correspondence is basically entered in units of behavior. The corresponding "Joints" section of the block set information section 342 and the 3D object information section 344 shows, for example, that the joints RJ1 and RJ2 of the block set are associated with the joint VJ3 of the 3D object. When the movement of the block set is reflected in the 3D object in this correspondence, the sum of the changes in the angles of the joints RJ1 and RJ is defined as representing the change in the angle of the joint VJ3 of the 3D object.

The corresponding "angle change" section of the block set information section 342 and the 3D object information section 344 shows that when the movement of the 3D object is reflected in the block set inversely and the angle of the joint VJ3 of the 3D object is changed by θ, the joint The angles of RJ1 and RJ2 each change according to θ/2, that is, the distribution ratio is 1:1. The joint RJ3 of the block set is associated with the joints VJ2 and VJ1 of the 3D object, and when the movement of the block set is reflected in the 3D object, the angle change distribution ratio is 1:2. In the following sections, the correspondence between any of the movements described with reference to FIGS. 23 to 27 is similarly entered along with the corresponding positions.

Descriptions of specific examples related to the correspondence between the movements of the joints have been mainly made so far. Next, a description of the association of the movement of the wheels will be made. In the case of joints, although both the block collection and the 3D object have constraints (such as movable angles, overall structure, etc.), the joints are basically independent of each other, and therefore, the joints of the block collection and the joints of the 3D object can be equal effectively relate. In the case of wheels, on the other hand, there is actually the difficulty that even when the movement of the wheel is not precisely defined, the collection of blocks has more constraints than in the case of joints, while 3D objects can be Expressed as if the 3D object was driving. That is, in order for the set of blocks and the 3D object to appear interlocked with each other, it is sufficient to associate the direction of travel and the approximate speed with each other. These correspondences are acquired by setting a common coordinate system in S48 in FIG. 19 .

When the user moves the set of blocks 260 and this movement is reflected in the 3D object, the reflection of the movement of the set of blocks in the 3D object can be easily achieved based only on these correspondences. Specifically, it is sufficient to acquire the moving amount and traveling direction of the block set from the captured image of the camera 122 or the like and express the 3D object so that the 3D object also moves similarly based on the moving amount and traveling direction of the block set. On the other hand, when the movement of the 3D object is to be reflected in the set of blocks, setting according to constraint conditions is necessary. For example, when the wheels do not perform cooperative operations, the vehicle cannot be made to travel. Furthermore, the rotational speed of the wheel used to obtain the desired speed varies according to the diameter of the wheel. So, for example, when proper adjustments are not made, the vehicle may reach excessive speeds and collide with a wall.

Therefore, when associating the movement of the wheel, there is less room for free setting by the user than in the case of joints, since there are more constraints on the block set side. Therefore, the information processing section 30 mainly associates movements. Specifically, at the point in time when the front, rear, left, and right of the block set becomes clear, the driving wheels, driven wheels, and steering wheels are determined according to the driving system set in advance. In addition, grouping is performed so that the constraint condition that the left and right drive wheels and the left and right steered wheels have the same rotation speed and the same steering angle is satisfied. In addition, the rotation speed and steering angle of the wheels of the block set are associated with the virtual speed and direction change of the 3D object, so that the block set is at a suitable speed and a suitable direction corresponding to the speed and direction change of the 3D object expressed in the virtual world. Change to drive.

30 is a flowchart depicting a processing procedure for setting correspondence between movement of a 3D object and movement of a block set in a mode in which the movement of the 3D object is reflected in the block set. This processing corresponds to the processing of S50 and S52 in FIG. 19 . When the information processing section 30 first detects that the wheel is assembled to the block set based on the information from the structure analysis section 22, the information processing section 30 assigns the role of the wheel (such as driving the wheel , steering wheels, etc.) are assigned to the front or rear wheels (S58). In addition, at least two wheels are arranged parallel to each other, wherein the wheels are assigned a role of a driving wheel or a steering wheel and are grouped into a group such that the wheels perform a cooperative operation (S60).

Incidentally, when two wheels arranged in parallel to each other rotate about one wheel shaft, the two wheels naturally perform cooperative operations, and thus the processing of S60 can be omitted. Next, control parameters for obtaining an appropriate moving speed and an appropriate direction change reflecting the virtual travel of the 3D object are obtained. The control parameters are in this case the rotational speed of the actuator that rotates the axle of the drive wheel, the amount of movement of the actuator that changes the steering angle of the steered wheel, and the like. Incidentally, the plurality of actuators controlling the driving wheels and steering wheels grouped in S60 are controlled so that the same operation is performed according to the control signal from the information processing device 10 .

In order to acquire the control parameters, first, the user is made to actually move the set of blocks in a predetermined direction. The movement amount of the block set is measured, and the rotation amount and steering angle of the axle at the time of movement are measured (S62, S64, and S66). For example, a user may electronically actuate the block set through a separately prepared control mechanism, or may manually move the block set by pushing the body of the block set by hand. In the latter case, the axle and the mechanism for changing the steering angle are not controlled by the actuator. The movement amount of the block set may be acquired from a captured image of the camera 122 or a depth image generated from the captured image of the camera 122 . The amount of rotation can be obtained from a signal from a rotary encoder provided to the wheel. The steering angle can be acquired by a steering angle sensor provided to the wheels. Incidentally, when a sensor such as a rotary encoder or the like is not provided, for example, the amount of rotation and the steering angle can be calculated based on the diameter of the wheel, the travel distance, and the travel direction.

Then, based on the actual rotation amount and the actual steering angle regarding the movement amount including the direction change, a control amount for acquiring the movement amount (such as the rotation amount of the motor, the movement amount of the actuator, etc.) is acquired (S68). Convert this correspondence to a value per unit of time or per unit of angle. The speed of movement and the unit steering angle are thus associated with the values of the control parameters (S70). The moving speed in this case is an appropriate moving speed of the block set itself. However, when the block set is interlocked with the 3D object on the screen, the moving speed is a value determined by the moving speed in the virtual world. The same holds for the steering angle. Therefore, through the process of S70, the movement of the 3D object is associated with the movement of the block set.

By the way, when the diameter of the wheel connected to the block set is known from the identification number of said wheel, the correspondence between the moving speed and the control parameter can be obtained through operation. Therefore, the processing of S62 to S68 can be omitted. Furthermore, the examples described so far assume that the set of blocks assumed to represent the lift truck and the 3D object of the lift truck are associated with each other. The description of the association of the movement of the wheels has thus been made. However, even when the 3D object has no wheels, the flowchart depicted in FIG. 28 is similarly applied.

That is, even in the case where the 3D object is a person, an animal, an insect, a vehicle other than a car, etc., when the movement in the virtual world is reflected in the movement of the block set, by virtue of the processing procedure of FIG. The movement speed and steering angle are associated with the control parameters, and the set of blocks can be moved at a speed and direction corresponding to the virtual movement of the 3D object. Furthermore, this can be applied not only to a mode reflecting movement of a 3D object but also to a mode of simply moving a block set according to the result of processing performed by the information processing device 10 .

FIG. 31 is a diagram of assistance in explaining the case where the above-described setting related to the round of block sets is extended to composite links. In the composite link 350 depicted in the figure, the joints are not completely independent of each other, but rather the links interlock with each other. That is, when one joint of composite link 350 moves as indicated by the arrow, all four joints move (composite link 352). In this case, it is not possible for the actuator to move only one joint. Thus, as in the case of the drive wheel, the four joints are grouped and made to perform cooperative operations. When a synthetic link is incorporated into a block set, the user clearly indicates the existence of the synthetic link on the corresponding setting screen or the like, or when the synthetic link in the real thing of the block set moves, the information processing section 30 recognizes the synthetic link existence. Then, all joints included in the compositing link are grouped when the setting corresponds to the movement of the 3D object.

According to this embodiment described above, a block that can be freely assembled is used as an input device or an output device for processing in an information processing device. The details of the block after assembly, such as the skeleton and position of the block, etc. are acquired by using various kinds of sensors for acquiring the position and orientation and a communication block having a communication function. Furthermore, the surface shape of the block after assembly is acquired by using means for detecting the presence of the block in real space, such as an image taken by a camera or the like. Even when a non-communicating block having no communication function is used as a part of the block, the position, attitude, and shape of the block after assembly can be identified with high accuracy by integrating these information.

Therefore, the shape, material, color, etc. of the blocks are not limited, and even things made by the user himself/herself can be used, and an object with an external appearance serving the user's intention can be freely created. In addition, information processing devices can acquire position, attitude, and shape with high accuracy regardless of external appearance. Accordingly, various kinds of information processing can be performed with an assembling or moving action performed by a user as input information. Furthermore, as a result of information processing, assembled blocks can be moved.

For example, a 3D object may be displayed with the same external appearance as the assembled blocks, or a 3D object may be displayed with a more realistic external appearance (the 3D object corresponding to the assembled blocks). In the latter case, the user may form the 3D object itself by specifying parts of the 3D object for each part of the block, or may make the entire block correspond to one 3D object. At this time, by associating the positions and the like of the joints to be interlocked, not only changes in positions but also changes in attitude or shape can be mutually reflected. Setting corresponding position and movement associations can be set based on constraints such as shapes of block sets and 3D objects, number of joints, movable angles of joints, cooperative operation of wheels, etc. These associations are automatically established by the information processing device, or an environment that allows the user to make settings is provided. It is thereby possible to freely form a link between the real world and the virtual world while reducing the load on the user.

The present invention has been described above based on the embodiments of the present invention. The aforementioned embodiments are illustrative, and those skilled in the art should understand that combinations of constituent elements and processing procedures of the embodiments are susceptible to various modifications, and these modifications also fall within the scope of the present invention.

For example, the corresponding position setting screen 300 described with reference to FIG. 22 assumes a block set as an object to be associated with a 3D object on the screen. However, things other than block sets, such as, for example, the user himself/herself, etc., may be similarly associated. When associating a user, the camera 122 photographs the user, for example generates a depth image based on the photographed image, and estimates the position of the skeleton. The position of the skeleton can be tracked by applying conventional techniques such as skeleton tracking. When the user then moves a part (such as an arm, etc.) with respect to a joint desired to be associated (such as a shoulder, an elbow, etc.), the information processing device 10 recognizes the joint as in the case of a block set. Also, joints of 3D objects are specified on the screen, so that the user and the joints of 3D objects can be associated with each other.

A mode in which a 3D object moves according to a user's movement can thus be realized. Alternatively, when a block set is assembled in a human form and associated with a 3D object as in this embodiment, the joints of the user and the joints of the block set are indirectly associated with each other via the joints of the 3D object. When this state is utilized in this embodiment and combined with a mode of reflecting movement of a 3D object in a block set, a mode of moving the block set according to the user's movement can also be realized.

As a modification of this mode, an image of the user and an image depicting the position of the skeleton may be generated in real time from the captured image, and displayed on the corresponding position setting screen so that the joint of the user and the joint of the 3D object are mutually aligned on the screen associated. At this time, the two images may be displayed simultaneously or alternately to receive descriptions corresponding to the joints to be set one by one. Incidentally, these modifications assume that the user associates his/her own joints with those of the 3D object. However, when the 3D object is a person, a robot, or the like and the correspondence with the user's joint is obvious, the setting by the user can be omitted, and the information processing device can make all the settings.

Furthermore, in this embodiment, in a state where the front and rear of the vehicle are known, the wheels are fitted to the axles, and the information processing apparatus determines the roles of the driving wheels, driven wheels, and steering wheels accordingly. The movements of the actuators controlling the operation of the vehicle are then varied in accordance with these actions. This pattern can be extended to implement blocks that include actuators that change their movement depending on what they are connected to. For example, when assembling a part consisting of an axle and wheels, a vehicle of a block assembly is moved by rotating the axle. When assembling the parts consisting of the cam and the spring, etc., the spring is released by the rotation of the cam, and the arrow etc. are ejected in the set state. When assembling a part comprising the joint described above, the part is made to operate as a joint.

In this case, the information processing apparatus 10 appropriately changes the movement of the actuator by recognizing the kind of connected thing from the captured image and sending a control signal corresponding to the kind of connected thing to the block set. Therefore, the variety of blocks including actuators is increased, and a wide collection of blocks can be realized at a lower cost than making different blocks for different kinds.

[label list]

2 information processing system, 10 information processing equipment, 14 input equipment, 16 display equipment, 20 core information receiving section, 22 structure analysis section, 24 block information storage section, 26 model data storage section, 28 corresponding information storage area Section, 30 information processing section, 32 display processing section, 34 drive control section, 102a quadrangular prism block, 122 camera, 120 block assembly, 126a block, 128a battery, 130a communication mechanism, 132a memory, 134 position sensor, 136a Motion sensor, 138 angle sensor, 139a actuator, 141 rotary encoder, 142a first block, 143a first communication section, 144a component information acquisition section, 146a second communication section, 148 drive section.

[Industrial applicability]

As described above, the present invention is applicable to toys, game devices, built-in devices, learning materials, content display terminals, information processing devices, robots, and the like.

Claims (30)

1. An information processing device, comprising: a structure information acquisition section that acquires information on the position of the movable part in the overall structure of the external object; and an information processing section that displays a setting screen on a display device for receiving a setting input from a user related to an interlocking rule for a movable part for interlocking an object and an object model associated with the object, based on The received setting input is used to generate the information related to the interlock rule, and the information related to the interlock rule is stored in the storage device. 2. The information processing device according to claim 1, Wherein, the information processing section receives a setting input for associating the movable part of the object and the movable part of the object model with each other from the user through the setting screen so as to change the movable part of one of the object and the object model It is reflected in the movable part of the other of the object and the object model, and information on correspondence between the movable parts is generated based on the received setting input. 3. The information processing device according to claim 2, wherein, the structure information acquisition section acquires information on the position of a part moved by the user among the movable parts of the object, and The information processing section acquires, as movable parts to be associated with each other, parts moved by the user and parts of an object model specified by the user via an input device, and parts of the object model displayed on the setting screen, and by placing the movable parts are associated with each other to generate corresponding information. 4. The information processing device according to claim 2 or 3, Among them, the structure information acquisition section acquires information about positions of movable parts in an overall structure of an assembly device formed as an object by coupling separately prepared blocks to each other. 5. The information processing device according to any one of claims 2 to 4, Among them, the information processing section assists the setting input by the user by showing associable movable parts on the setting screen according to constraints individually given for the object and the movable parts of the object model. 6. The information processing device according to claim 2, Wherein, the information processing section displays the image of the object and the image of the object model on the setting screen based on the information on the position of the movable part in the overall structure of the object, acquires the images of the movable parts and the object model in the image of the object As the movable parts to be associated with each other, the movable parts in the image of the object and the movable parts in the image of the object model are specified by the user via an input device, and by linking the movable parts to each other Association to generate corresponding information. 7. The information processing device according to any one of claims 2 to 6, Among them, the information processing section further displays on the display device an object model selection screen showing images of a plurality of object models that are candidates to be corresponding to the object, and receives an instruction for selecting one object model to be corresponding to the object from the user. enter. 8. The information processing apparatus according to claim 7, Among them, the information processing section extracts from the plurality of object models prepared in advance by comparing the information on the movable parts of the object with the information on the movable parts of the plurality of object models and estimating the degree of similarity. The object model to be displayed as a candidate. 9. The information processing device according to any one of claims 2 to 8, Wherein, the information processing section allows the user to confirm the result of the association by moving the movable part on the object model side according to the movement of the associated movable part on the object side after the setting input, and is used for re-execution after the user executes The input of the association cancels the immediately preceding association. 10. An information processing method performed by an information processing device, the information processing method comprising the following steps: obtain information about the position of movable parts in the overall structure of the external object; A corresponding position setting screen is displayed on the display device for receiving a setting input for associating a movable part of an object with a movable part of an object model associated with the object from a user so that by linking the object and one of the object models Interlocking the object and the object model with each other by reflecting changes in the movable parts of the one in the associated movable parts of the other of the object and object model; and Correspondence information is generated based on the received setting input and stored in a storage device. 11. A computer program for enabling a computer to realize the following functions: obtain information about the position of movable parts in the overall structure of the external object; A corresponding position setting screen is displayed on the display device for receiving a setting input for associating a movable part of an object with a movable part of an object model associated with the object from a user so that by linking the object and one of the object models Interlocking the object and the object model with each other by reflecting changes in the movable parts of the one in the associated movable parts of the other of the object and object model; and Correspondence information is generated based on the received setting input and stored in a storage device. 12. A computer-readable recording medium on which a computer program is recorded, the computer program causes a computer to realize the following functions: obtain information about the position of movable parts in the overall structure of the external object; A corresponding position setting screen is displayed on the display device for receiving a setting input for associating a movable part of an object with a movable part of an object model associated with the object from a user so that by linking the object and one of the object models Interlocking the object and the object model with each other by reflecting changes in the movable parts of the one in the associated movable parts of the other of the object and object model; and Correspondence information is generated based on the received setting input and stored in a storage device. 13. The information processing device according to claim 1, wherein the information processing section receives a setting input for setting a movement reflection rule for each of a movable part of an object and a movable part of an object model, the movable parts being associated with each other, from the user through the setting screen , the movement reflection rule is used to reflect changes in the movable parts of one of the object and the object model in the associated movable parts of the other of the object and the object model, and based on the received setting input to Generate information about reflection rules. 14. The information processing apparatus according to claim 13, Among them, the structure information acquisition section acquires information about positions of movable parts in an overall structure of an assembly device formed as an object by coupling separately prepared blocks to each other. 15. The information processing device according to claim 13 or 14, Among them, the information processing section receives a setting input for setting a rule for reflecting a change of an angle between a joint of the object and a joint of the object model. 16. The information processing apparatus according to claim 15, Among them, the information processing section receives a setting input for changing one of the associated joints at an angle obtained by multiplying the change angle of the other joint by a predetermined value as a rule. 17. The information processing device according to any one of claims 13 to 16, Wherein, the information processing section acquires the orientation of the object in the real space from an image acquired by photographing the object, and sets a coordinate system common to the object and the object model by receiving an input for adjusting the orientation of the object model on the display , so that the orientation of the object model displayed on the display device is the same as that of the object, so that the traveling directions of the object and the object model are associated with each other. 18. The information processing device according to any one of claims 13 to 17, Wherein, the information processing section allows the user to confirm the result of the association by moving the movable part on the object model side according to the movement of the associated movable part on the object side after the setting input, and is used for re-execution after the user executes The input of the association cancels the immediately preceding association. 19. An information processing method performed by an information processing device, the information processing method comprising the following steps: obtain information about the position of movable parts in the overall structure of the external object; A movement correspondence setting screen is displayed on the display device for receiving a setting input from a user for setting a movement reflection rule for a movable part of an object and a movable part of an object model associated with the object, the movable parts being mutually related to each other. associating to interlock the object and the object model with each other by reflecting changes in the movable parts of one of the object and the object model in the associated movable parts of the other of the object and the object model; and Correspondence information is generated based on the received setting input and stored in a storage device. 20. A computer program for enabling a computer to realize the following functions: obtain information about the position of movable parts in the overall structure of the external object; A movement correspondence setting screen is displayed on the display device for receiving a setting input from a user for setting a movement reflection rule for a movable part of an object and a movable part of an object model associated with the object, the movable parts being mutually related to each other. associating to interlock the object and the object model with each other by reflecting changes in the movable parts of one of the object and the object model in the associated movable parts of the other of the object and the object model; and Correspondence information is generated based on the received setting input and stored in a storage device. 21. A computer-readable recording medium on which a computer program is recorded, the computer program causes a computer to realize the following functions: obtain information about the position of movable parts in the overall structure of the external object; A movement correspondence setting screen is displayed on the display device for receiving a setting input from a user for setting a movement reflection rule for a movable part of an object and a movable part of an object model associated with the object, the movable parts being mutually related to each other. associating to interlock the object and the object model with each other by reflecting changes in the movable parts of one of the object and the object model in the associated movable parts of the other of the object and the object model; and Correspondence information is generated based on the received setting input and stored in a storage device. 22. The information processing device according to claim 1, Wherein, the information processing section receives from the user a setting input for setting the correspondence between the number of the movable part of the object and the number of the movable part of the object model from the user through the setting screen, so that the object and the number of the movable part in the object model Changes in the movable parts of one are reflected in different numbers of the movable parts of the other of the object and object models, and information on correspondence between the movable parts is generated based on the received setting input. 23. The information processing device according to claim 22, Wherein, the information processing section receives a joint for associating a joint of one of the object and the object model with a plurality of joints of the other of the object and the object model among the joints of the object and the joints of the object model as a movable part setting input for , and The information processing section further receives an input for setting an allocation ratio when the change angle of the one joint is reflected so as to be allocated to change the angles of the plurality of joints, and records the allocation ratio in the corresponding information. 24. The information processing device according to claim 22, Wherein, the information processing section receives a method for causing each of the plurality of joints to move at an angle equal to a change angle of the one joint by associating a joint of the object with a plurality of joints of the object model as a movable part. Changed setting input. 25. The information processing device according to claim 22, Wherein, the information processing section receives a joint for associating a joint of one of the object and the object model with a plurality of joints of the other of the object and the object model among the joints of the object and the joints of the object model as a movable part And a setting input that causes the one joint to be changed by the sum of the changed angles of the plurality of joints. 26. The information processing device according to claim 22, Wherein the information processing section receives a setting input for associating a plurality of joints of the object with one joint of the object model, the joints individually changing at angles about a plurality of different axes, as the movable part, so The one joint has a plurality of degrees of freedom corresponding to a plurality of axes. 27. An information processing device as claimed in any one of claims 22 to 26, Among them, the structure information acquisition section acquires information about positions of movable parts in an overall structure of an assembly device formed as an object by coupling separately prepared blocks to each other. 28. An information processing method performed by an information processing device, the information processing method comprising the following steps: obtain information about the position of movable parts in the overall structure of the external object; A corresponding position setting screen is displayed on the display device for receiving a setting input for associating a movable part of an object with a movable part of an object model associated with the object from a user so that by linking the object and one of the object models Interlocking the object and the object model with each other by reflecting changes in the movable parts of the one in the associated movable parts of the other of the object and object model; receiving setting input for associating different numbers of movable parts between objects and object models with each other; and Correspondence information is generated based on the received setting input and stored in a storage device. 29. A computer program for causing a computer to realize the following functions: obtain information about the position of movable parts in the overall structure of the external object; A corresponding position setting screen is displayed on the display device for receiving a setting input for associating a movable part of an object with a movable part of an object model associated with the object from a user so that by linking the object and one of the object models Interlocking the object and the object model with each other by reflecting changes in the movable parts of the one in the associated movable parts of the other of the object and object model; receiving setting input for associating different numbers of movable parts between objects and object models with each other; and Correspondence information is generated based on the received setting input and stored in a storage device. 30. A computer-readable recording medium on which a computer program is recorded, the computer program causes a computer to realize the following functions: obtain information about the position of movable parts in the overall structure of the external object; A corresponding position setting screen is displayed on the display device for receiving a setting input for associating a movable part of an object with a movable part of an object model associated with the object from a user so that by linking the object and one of the object models Interlocking the object and the object model with each other by reflecting changes in the movable parts of the one in the associated movable parts of the other of the object and object model; receiving setting input for associating different numbers of movable parts between objects and object models with each other; and Correspondence information is generated based on the received setting input and stored in a storage device.