JP5292146B2 - Information processing apparatus and information processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To achieve various information processing by a simple operation. <P>SOLUTION: A standard image to which a link is set is displayed on a display device (S18). When a user operates an input device and the eye point enters a first link border (Y in S20), if there is no information to be resumed (N in S22), a first frame of moving image data is displayed as a still image on the display device (S26), and if there is resume information, the information is read out (Y in S22, S24), and a frame at the resume point is displayed as a still image on the display device (S26). When the eye point enters a second link border (Y in S28), reproduction and display of a moving image is started (S30). When the eye point goes out of the second link border (Y in S32), the reproduction of the moving image is stopped, to display a frame at a stop point as a still image, and the resume point is stored (S34, S36). When the eye point goes out of the first link border (Y in S38), the standard image which is displayed in S18 is displayed (S40). <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to an information processing technique for performing information processing in accordance with a user input instruction corresponding to a display image.

  Home entertainment systems have been proposed that not only execute game programs, but also play video. In this home entertainment system, the GPU generates a three-dimensional image using polygons (see, for example, Patent Document 1).

  On the other hand, a technique for enlarging / reducing a display image and moving up / down / left / right using a plurality of resolution tile images generated from a digital image such as a high-definition photograph has been proposed. In this image processing technique, an original image size is reduced in a plurality of stages to generate images with different resolutions, and images in each layer are divided into one or a plurality of tile images to represent the original image in a hierarchical structure. Usually, the image with the lowest resolution is composed of one tile image, and the original image with the highest resolution is composed of the largest number of tile images. The image processing apparatus quickly performs enlarged display or reduced display by switching the tile image being used to a tile image of a different hierarchy at the time of enlargement processing or reduction processing of the display image.

US Pat. No. 6,563,999

  In recent years, the size of a display screen has been expanded even in a portable terminal or the like, and it has become possible to display a high-definition image regardless of the type of information processing apparatus. Therefore, various contents appealing to the sight became more familiar. However, as the information to be displayed becomes complex and sophisticated, various knowledge is required to make full use of it. For this reason, the quality of information and the ease of operation often have a trade-off relationship. In addition, more specialized knowledge is required to create complex contents including images.

  The present invention has been made in view of such problems, and an object of the present invention is to provide a technique capable of realizing various processes by an easy operation.

  One embodiment of the present invention relates to an information processing apparatus. This information processing apparatus includes a storage device that holds hierarchical data formed by layering image data of different resolutions in the order of resolution, and a request for moving a viewpoint in a virtual space defined by the image plane and a distance from the image plane. An input information acquisition unit that accepts from, a display image processing unit that generates a display image from hierarchical data by changing a display area in response to a viewpoint movement request, and a viewpoint associated with a predetermined area in the image An object execution unit that executes the process when a condition for starting the process is satisfied, and the condition is a position between the predetermined area set on the image plane and an area reflected on the screen determined by the viewpoint It is set for the relationship.

  Another aspect of the present invention relates to an information processing method. The information processing method includes a step of reading out hierarchical data composed of image data of different resolutions in the order of resolution from a memory and displaying the hierarchical data on a display device, and a viewpoint in a virtual space defined by an image plane and a distance from the image plane. A step of accepting the movement request from the user, a step of changing the area to be displayed on the display device in response to the movement request of the viewpoint, and a condition for starting the processing set in association with a predetermined area in the image A step of executing the processing when the viewpoint satisfies, and the condition is set for a positional relationship between a predetermined area set on the image plane and an area shown on the screen determined by the viewpoint. Features.

  Another aspect of the present invention relates to a data structure of content. The data structure of this content is a condition for starting processing associated with hierarchical data formed by layering image data of different resolutions in the order of resolution and a predetermined area in an image generated using the hierarchical data. And set for the positional relationship between the predetermined area set on the image plane and the area of the image displayed on the screen determined by the viewpoint in the virtual space defined by the image plane and the distance from the image plane. It is characterized by being associated with a setting file that describes the conditions.

  Note that any combination of the above-described components, and the expression of the present invention converted between a method, an apparatus, a system, a computer program, a recording medium on which the computer program is recorded, and the like are also effective as an aspect of the present invention. .

  According to the present invention, various information processing can be realized by an easy operation.

It is a figure which shows the use environment of the image processing system concerning embodiment of this invention. It is a figure which shows the external appearance structure of the input device applicable to the image processing system of FIG. It is a figure which shows the hierarchical structure example of the image data used in this Embodiment. It is a figure which shows the structure of the information processing apparatus in this Embodiment. It is a figure which shows typically the flow of the image data in this Embodiment. It is a figure which shows typically the relationship of the several hierarchy data used as display object in this Embodiment. It is a figure which shows the structure of the control part in this Embodiment in detail. It is a figure which shows the example of a definition of the target rectangle and boundary rectangle in this Embodiment. It is a figure which shows the example of a definition of the magnification of the boundary rectangle in this Embodiment. It is a figure explaining the definition of parameter "overshot" showing the rate which a screen protrudes from a boundary rectangle in this embodiment. It is a figure explaining the definition of parameter "invisibility" showing the ratio which a boundary rectangle protrudes from a screen in this embodiment. It is a figure which shows the example description in XML of a setting file in this Embodiment. It is a figure which shows the relationship between the target rectangle in this Embodiment, a boundary rectangle, and a link boundary. It is a figure for demonstrating the change of a link boundary when changing parameter "max_overshot" in this Embodiment. It is a figure for demonstrating the change of a link boundary when changing parameter "max_invisibility" in this Embodiment. It is a figure which shows the example of the image displayed on a display apparatus by information processing apparatus in this Embodiment. It is a flowchart which shows the process sequence which implements the process of video reproduction as an embedded object in this Embodiment. It is a figure which shows the example of the image displayed when operating a store link as an embedded object in this Embodiment. In this Embodiment, it is a flowchart which shows the process sequence of the information processing apparatus at the time of performing selection reception display as an embedded object. In this Embodiment, it is a figure for demonstrating the aspect which returns a display area when a viewpoint is moved arbitrarily.

  FIG. 1 shows a use environment of an information processing system 1 according to an embodiment of the present invention. The information processing system 1 includes an information processing device 10 that processes content including at least one of functions such as image processing, video playback, audio playback, and communication, and a display device 12 that outputs a processing result by the information processing device 10. Prepare. The display device 12 may be a television having a display that outputs an image and a speaker that outputs sound. The display device 12 may be connected to the information processing device 10 by a wired cable, or may be wirelessly connected by a wireless local area network (LAN) or the like.

  In the information processing system 1, the information processing apparatus 10 may connect to an external network such as the Internet via the cable 14 and download and acquire content software including hierarchical compressed image data. Note that the information processing apparatus 10 may be connected to an external network by wireless communication.

  Note that the information processing apparatus 10 may be a game device, a personal computer, or the like, and may implement functions described later by loading software from various recording media. In the present embodiment, an image displayed on the display device 12 by a later-described image processing mechanism is used as a user interface. The user inputs an instruction to the information processing apparatus 10 by changing the display image using the input device. Processing performed by the information processing apparatus 10 in response to the instruction input may vary depending on the content. Specific examples will be described later.

  When the user inputs to the input device requesting enlargement / reduction of the display region and scrolling in the up / down / left / right direction while viewing the image displayed on the display device 12, the input device responds accordingly. A request signal for movement and enlargement / reduction is transmitted to the information processing apparatus 10. The information processing apparatus 10 changes the image in the screen of the display device 12 according to the signal. Such movement and enlargement / reduction of the display area can be considered as movement of the virtual viewpoint of the user, and will be collectively referred to as “movement of the viewpoint” hereinafter. The information processing apparatus 10 further executes a process associated with a predetermined area in the image according to a predetermined rule. For example, when the user zooms in on a predetermined area, the information processing apparatus 10 starts an operation associated with the area.

  FIG. 2 shows an external configuration example of the input device 20. The input device 20 includes a cross key 21, analog sticks 27a and 27b, and four types of operation buttons 26 as operation means that can be operated by the user. The four types of operation buttons 26 include a circle button 22, a x button 23, a square button 24, and a triangle button 25.

  In the information processing system 1, a function for inputting a movement of the viewpoint, that is, a request to enlarge / reduce the display area and a scroll request in the up / down / left / right directions is assigned to the operation unit of the input device 20. For example, the display area enlargement / reduction request input function is assigned to the right analog stick 27b. The user can input a display area reduction request by pulling the analog stick 27b forward, and can input a display area enlargement request by pressing it from the front. The input function for scrolling the display area is assigned to the cross key 21. The user can input a scroll request in the direction in which the cross key 21 is pressed by pressing the cross key 21. Note that the viewpoint movement request input function may be assigned to another operation means. For example, the scroll request input function may be assigned to the analog stick 27a.

  The input device 20 has a function of transmitting the input viewpoint movement request signal to the information processing device 10, and is configured to be capable of wireless communication with the information processing device 10 in the present embodiment. The input device 20 and the information processing device 10 may establish a wireless connection using a Bluetooth (registered trademark) protocol, an IEEE802.11 protocol, or the like. The input device 20 may be connected to the information processing apparatus 10 via a cable and transmit the viewpoint movement request signal to the information processing apparatus 10.

  FIG. 3 shows an example of a hierarchical structure of image data used in the present embodiment. The image data has a hierarchical structure including a 0th hierarchy 30, a first hierarchy 32, a second hierarchy 34, and a third hierarchy 36 in the depth (Z-axis) direction. Although only four layers are shown in the figure, the number of layers is not limited to this. Hereinafter, image data having such a hierarchical structure is referred to as “hierarchical data”.

  The hierarchical data shown in FIG. 3 has a hierarchical structure of a quadtree, and each hierarchy is composed of one or more tile images 38. All the tile images 38 are formed in the same size having the same number of pixels, and have, for example, 256 × 256 pixels. The image data of each layer expresses one image at different resolutions, and the original image of the third layer 36 having the highest resolution is reduced in a plurality of stages to obtain the second layer 34, the first layer 32, the 0th layer. Image data of the hierarchy 30 is generated. For example, the resolution of the Nth layer (N is an integer greater than or equal to 0) may be ½ of the resolution of the (N + 1) th layer in both the left and right (X axis) direction and the up and down (Y axis) direction.

  In the information processing apparatus 10, the hierarchical data is held in the storage device in a compressed state in a predetermined compression format, and is read from the storage device and decoded when the content is activated. The information processing apparatus 10 according to the present embodiment has a decoding function corresponding to a plurality of types of compression formats, and can decode compressed data in, for example, the S3TC format, JPEG format, and JPEG2000 format. In the hierarchical data, the compression processing may be performed in units of tile images, or may be performed in units of a plurality of tile images included in the same layer or a plurality of layers.

  As shown in FIG. 3, the hierarchical structure of the hierarchical data is set with the horizontal direction as the X axis, the vertical direction as the Y axis, and the depth direction as the Z axis, thereby constructing a virtual three-dimensional space. When the information processing apparatus 10 derives the movement amount of the display area from the viewpoint movement request signal supplied from the input device 20, the information processing apparatus 10 derives the coordinates (frame coordinates) of the four corners of the frame in the virtual space using the movement amount. The frame coordinates in the virtual space are used for display image generation processing. Instead of the frame coordinates in the virtual space, the information processing apparatus 10 may derive information for specifying a hierarchy and texture coordinates (UV coordinates) in the hierarchy. Hereinafter, the combination of the hierarchy specifying information and the texture coordinates is also referred to as frame coordinates.

  The image data of each layer included in the layer data exists discretely with respect to the Z axis of the virtual space. Therefore, when displaying an image at a scale ratio between hierarchies where there is no image data, image data in the vicinity in the Z-axis direction is used. For example, when the scale ratio of the display image is in the vicinity of the second hierarchy 34, the display image is created using the image data of the second hierarchy. In order to realize this, a source image switching boundary is set between the layers, for example, in the middle. When the scale ratio of the display image exceeds the switching boundary, the image data used to generate the display image is switched, and the image is enlarged or reduced for display.

  FIG. 4 shows the configuration of the information processing apparatus 10. The information processing apparatus 10 includes a wireless interface 40, a switch 42, a display processing unit 44, a hard disk drive 50, a recording medium mounting unit 52, a disk drive 54, a main memory 60, a buffer memory 70, and a control unit 100. . The display processing unit 44 has a frame memory that buffers data to be displayed on the display of the display device 12.

  The switch 42 is an Ethernet switch (Ethernet is a registered trademark), and is a device that transmits and receives data by connecting to an external device in a wired or wireless manner. The switch 42 is connected to an external network via the cable 14 and may receive a content file including hierarchical compressed image data and a setting file for realizing various functions using the image. . The content file further includes data necessary for executing various functions, for example, compression-encoded video data, music data, website name and URL (Uniform Resource Locator) correspondence data, and the like. It may be.

  The switch 42 is connected to the wireless interface 40, and the wireless interface 40 is connected to the input device 20 using a predetermined wireless communication protocol. The viewpoint movement request signal input from the user by the input device 20 is supplied to the control unit 100 via the wireless interface 40 and the switch 42.

  The hard disk drive 50 functions as a storage device that stores data. Various data received via the switch 42 is stored in the hard disk drive 50. When a removable recording medium such as a memory card is mounted, the recording medium mounting unit 52 reads data from the removable recording medium. When a read-only ROM disk is loaded, the disk drive 54 drives and recognizes the ROM disk to read data. The ROM disk may be an optical disk or a magneto-optical disk. The content file may be stored in these recording media.

  The control unit 100 includes a multi-core CPU, and includes one general-purpose processor core and a plurality of simple processor cores in one CPU. The general-purpose processor core is called a PPU (Power Processing Unit), and the remaining processor cores are called a SPU (Synergistic-Processing Unit).

  The control unit 100 includes a memory controller connected to the main memory 60 and the buffer memory 70. The PPU has a register, has a main processor as an operation execution subject, and efficiently assigns a task as a basic processing unit in an application to be executed to each SPU. Note that the PPU itself may execute the task. The SPU has a register, and includes a sub-processor as an operation execution subject and a local memory as a local storage area. The local memory may be used as the buffer memory 70.

  The main memory 60 and the buffer memory 70 are storage devices and are configured as a RAM (Random Access Memory). The SPU has a dedicated DMA (Direct Memory Access) controller as a control unit, can transfer data between the main memory 60 and the buffer memory 70 at high speed, and the frame memory and the buffer memory 70 in the display processing unit 44 can be transferred. High-speed data transfer can be realized. The control unit 100 according to the present embodiment realizes a high-speed image processing function by operating a plurality of SPUs in parallel. The display processing unit 44 is connected to the display device 12 and outputs a processing result according to a request from the user.

  The information processing apparatus 10 according to the present embodiment may load at least part of the compressed image data from the hard disk drive 50 to the main memory 60 in advance in order to smoothly change the display image as the viewpoint moves. . Further, an area to be displayed in the future may be predicted based on a viewpoint movement request from the user, and a part of the compressed image data loaded in the main memory 60 may be decoded and stored in the buffer memory 70. This makes it possible to instantaneously switch an image used for generating a display image at a necessary later timing.

  FIG. 5 schematically shows the flow of image data in the present embodiment. First, the hierarchical data included in the content file is stored in the hard disk drive 50. Instead of the hard disk drive 50, a recording medium mounted on the recording medium mounting unit 52 or the disk drive 54 may be held. Alternatively, the hierarchical data may be downloaded from an image server connected by the information processing apparatus 10 via a network. As described above, the hierarchical data here is subjected to fixed length compression in the S3TC format or the like, or variable length compression in the JPEG format or the like.

  Among the hierarchical data, a part of the image data is loaded into the main memory 60 in a compressed state (S10). The area to be loaded here is a predetermined rule such as the vicinity of the current display image in the virtual space, the area of the image, the area where a display request is predicted to be frequently made based on the user's browsing history, etc. Determined by. The loading is performed not only when a viewpoint movement request is made, but also at any given time interval, for example. This prevents the load process from being concentrated at one time.

  Next, among the compressed image data stored in the main memory 60, the image data of the region necessary for display or the image of the region predicted to be necessary is decoded and stored in the buffer memory 70 (S12). The buffer memory 70 includes at least two buffer areas 72 and 74. The size of each of the buffer areas 72 and 74 is set larger than the size of the frame memory 76, and when a viewpoint movement request signal is input from the input device 20, the buffer area 72 has a certain amount of movement. , 74, the display image can be generated.

  One of the buffer areas 72 and 74 is a display buffer that is used to hold an image used to generate a display image, and the other is a decoding that is used to prepare an image predicted to be necessary thereafter. Buffer. In the example of FIG. 5, it is assumed that the buffer area 72 is a display buffer, the buffer area 74 is a decoding buffer, and the display area 68 is displayed.

  Next, the image in the display area 68 among the images stored in the buffer area 72, which is a display buffer, is drawn in the frame memory 76 (S14). During this time, the image in the new area is decoded as necessary and stored in the buffer area 74. The display buffer and the decoding buffer are switched according to the timing of completion of storage, the amount of movement of the display area 68, and the like (S16). Thereby, a display image can be smoothly switched with respect to a movement of a display area, a change of a scale ratio, or the like.

  In the processing described so far, in order to move and enlarge / reduce the display area of a certain image, the frame coordinates according to the viewpoint movement request from the user in the virtual space constituted by one hierarchical data as shown in FIG. It was the aspect which moves. On the other hand, a plurality of hierarchical data may be prepared as display targets, and the display image may be moved back and forth between the hierarchical data. FIG. 6 schematically shows a relationship between a plurality of hierarchical data to be displayed in such an aspect.

  In the figure, two triangles indicate different hierarchical data 150 and 152. Each of the hierarchical data 150 and 152 actually has a configuration in which a plurality of image data having different resolutions are discretely present in the Z-axis direction in the figure as shown in FIG. When the user requests enlargement / reduction of the display area using the input device 20, the display area, and thus the viewpoint, moves in the Z-axis direction in the figure. On the other hand, when a request to move the display area up, down, left, or right is made, the horizontal plane in the figure is moved. In such a virtual space, image data in which two hierarchical data 150 and 152 overlap as shown in the figure is constructed.

  When the user continuously requests enlargement while displaying the image of the hierarchical data 150, the viewpoint moves as indicated by the arrow a and enters the area of the hierarchical data 152, that is, moves between the hierarchical data. When a different hierarchical data area is entered, data for generating a display image is switched from hierarchical data 150 to hierarchical data 152. This process can be realized only by changing the hierarchical data to be loaded into the main memory 60 in the image display processing procedure described so far.

  In order to construct image data composed of a plurality of hierarchical data as shown in FIG. 6, the resolution and position of the image when switching the hierarchical data 150 and the hierarchical data 152 are set in advance. This setting is represented by a line 154 in FIG. 6, whereby the degree of overlap of the hierarchical data can be determined. In the example shown in the figure, switching from the hierarchical data 150 to the hierarchical data 152 is performed at the resolution indicated by the z-axis z1 and the position represented by the line 154. Hereinafter, such switching between hierarchical data is referred to as “link”. The hierarchical data images to be switched may be the same image with different scale ratios or may be completely different images.

  Furthermore, instead of switching the display image to the hierarchical data 152, processing such as moving image reproduction, audio reproduction, display processing, and display area movement may be performed. Also in this case, as shown by the line 154, a link point is set for the hierarchical data 150, and when the viewpoint reaches the point, processing associated with the point is started.

  In this way, when the user approaches the viewpoint while viewing the image of the hierarchical data 150, information associated with the area can be displayed or a video can be played back. . If a plurality of such points are set in one image data, a menu screen that can be selected and operated from a plurality of processes can be constructed. In this way, processing that starts by moving the viewpoint is referred to as an “embedded object”. The association between the image data and the embedded object is also called “link”.

  FIG. 7 shows the configuration of the control unit 100 in detail. The control unit 100 includes an input information acquisition unit 102 that acquires information input by the user from the input device 20, a load unit 103 that loads display target hierarchical data from the hard disk drive 50, and a display region determination that determines a display region according to the input. Unit 104, a decoding unit 106 that decodes the compressed image data, and a display image processing unit 114 that draws a display image. The control unit 100 further includes a link determination unit 116 that determines whether the viewpoint satisfies a link condition for which a viewpoint is set, and an object execution unit 117 that executes an embedded object.

  In FIG. 7, each element described as a functional block for performing various processes can be configured by a CPU (Central Processing Unit), a memory, and other LSIs in terms of hardware. This is realized by a program loaded on the computer. As described above, the control unit 100 includes one PPU and a plurality of SPUs, and each functional block can be configured by the PPU and the SPU individually or in cooperation. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

  The hard disk drive 50 stores a content file including a setting file in which hierarchical data and information on links set in the hierarchical data are described. When displaying image data composed of a plurality of hierarchical data, a plurality of hierarchical data and a setting file corresponding to each of the hierarchical data are stored. The content file may further include moving image data, audio data, image module data, application programs such as games, and the like necessary for processing the embedded object as necessary.

  The input information acquisition unit 102 acquires information related to a request such as content activation / termination and viewpoint movement from the input device 20 in accordance with an operation by the user of the input device 20, and displays the information as required by a display area determination unit. 104, and notifies the load unit 103. When notified from the input information acquisition unit 102 that the content activation request has been made, the load unit 103 reads the hierarchical data of the initial image of the content and its setting file from the hard disk drive 50 and stores them in the main memory 60.

  The load unit 103 further reads out the hierarchical data of the movement destination of the viewpoint, data necessary for executing the embedded object, and the like from the hard disk drive 50 as necessary and stores them in the main memory 60. When notified by the input information acquisition unit 102 that the viewpoint movement request has been made, the display area determination unit 104 converts the movement amount of the viewpoint into coordinates in the virtual space of the hierarchical data, and the frame coordinates of the movement destination to be displayed To decide.

  The decoding unit 106 reads and decodes a part of the compressed image data from the main memory 60 and stores the decoded data in the buffer memory 70. The data decoded by the decoding unit 106 may be image data of a predetermined size including the display area. By decoding a wide range of image data in advance and storing it in the buffer memory 70, the number of readings from the main memory 60 can be reduced, and smooth viewpoint movement can be realized.

  The display image processing unit 114 acquires the frame coordinates of the region to be displayed determined by the display region determination unit 104, reads the corresponding image data from the buffer memory 70, and draws it in the frame memory 76 of the display processing unit 44. The link determination unit 116 refers to the setting file stored in the main memory 60 and associated with the hierarchical data being displayed, and determines whether or not the destination viewpoint satisfies the link condition. As described above, the link destinations set in this embodiment include a link to another hierarchical data and a link to an embedded object.

  When the link determination unit 116 determines that the link condition for the embedded object is satisfied, the object execution unit 117 executes the corresponding embedded object. If a program or data for executing the embedded object is not stored in the main memory 60, the load unit 103 loads the hard disk drive 50 to the main memory 60. The content of the process executed by the object execution unit 117 is not limited. Therefore, the object execution unit 117 may appropriately execute different types of processing according to the viewpoint position and the setting file settings.

  When processing an image being displayed as one of the embedded objects, the object execution unit 117 makes a request to that effect to the display image processing unit 114, and the display image processing unit 114 draws the processed image. Also, when moving the viewpoint of the image being displayed as one of the embedded objects, the object execution unit 117 makes a request to that effect to the display area determining unit 104, and the display area determining unit 104 moves the display area to which the image is to be moved. To decide. When performing moving image reproduction or audio reproduction, the object execution unit 117 reads out moving image data and audio data from the main memory 60, decodes them and outputs them to the display processing unit 44 or outputs them to the speaker of the display device 12.

  If the link determination unit 116 determines that the link condition to another hierarchical data is satisfied, the load unit 103 loads from the hard disk drive 50 if the link target hierarchical data is not stored in the main memory 60. The display area determination unit 104 acquires the determination result from the link determination unit 116, converts the frame coordinates to that of the link destination hierarchical data, and notifies the decoding unit 106 together with the identification information of the hierarchical data. Accordingly, the decoding unit 106 processes the linked hierarchical data as a processing target.

  Next, a setting example of the setting file will be described. 8 to 11 are diagrams for explaining an example of definition of each data in the setting file. FIG. 8 shows a definition example of the target rectangle and the boundary rectangle. In the figure, a target rectangle 82 is an area in which an object for setting a link is displayed in the image, or a rectangular area circumscribing the area. The boundary rectangle 84 is a rectangle that represents the range of the viewpoint in which the link is valid. In other words, when the viewpoint enters the boundary rectangle 84, the link to the target rectangle 82 becomes valid, and the display is switched to data at another link destination, or the embedded object is activated.

  In this example, a target rectangle 82 and a boundary rectangle 84 are set for a reference rectangle 80 representing a reference coordinate system in which the coordinates of the upper left vertex are (0, 0) and the coordinates of the lower right vertex are (W, H). doing. Here, the reference rectangle 80 can be regarded as the entire image to be displayed, and the unit of coordinates can be, for example, the number of pixels. First, the distance from the upper left vertex (0, 0) of the reference rectangle 80 to the center of the target rectangle 82 is a parameter “center”, the height of the target rectangle 82 is a parameter “height”, and the width is a parameter “width”. A target rectangle 82 is defined with three parameters.

  Further, the distance from the center of the target rectangle 82 to the center of the boundary rectangle 84 is a parameter “offset”, the magnification of the boundary rectangle 84 with respect to the target rectangle 82 is a parameter “scale”, and the boundary rectangle 84 is defined by these two parameters. At this time, as shown in the drawing, the height of the boundary rectangle 84 is height × scale and the width is width × scale. Note that the parameter “center” and the parameter “offsett” have components in the horizontal direction and the vertical direction, but are collectively shown in FIG.

  FIG. 9 shows a definition example of the magnification of the boundary rectangle. In the present embodiment, not only the display area is moved on a certain plane but also the movement in the Z-axis direction in the virtual space for enlarging or reducing the image is accepted. Therefore, by making use of the movement for link setting, it becomes possible to set links that are easy for the user to understand with abundant variations. As one of the setting methods, an upper limit and a lower limit can be set for the magnification of the boundary rectangle.

  In FIG. 9, a rectangle having a height h and a width w represents the screen 86 of the display device. At this time, the aspect ratio of the boundary rectangle 84, that is, the magnification of the rectangle 88 expanded in the vertical direction or the horizontal direction so as to be h: w with respect to the screen 86, the parameter “zoom” indicating the magnification of the boundary rectangle 84. And At this time, as shown in the drawing, the vertical and horizontal lengths of the rectangle 88 are h × zoom and w × zoom, respectively.

  10 and 11 show examples of the definition of the positional relationship between the boundary rectangle and the area shown on the screen (hereinafter simply referred to as “screen”). FIG. 10 is a diagram illustrating the definition of the parameter “overshot” that represents the ratio of the screen protruding from the boundary rectangle. In the figure, three patterns, a screen 86a, a screen 86b, and a screen 86c are shown as screens. In the figure, “a” is the length of the screen protruding from the boundary rectangle 84 in the horizontal direction (screens 86a and 86b), and “b” is the length of the screen protruding in the vertical direction (screen 86c). At this time, the parameter “overshot” is defined as max (a / w, b / h). That is, when it protrudes in either the horizontal direction or the vertical direction, the ratio of the protruding length (a or b) to the screen length (w or h) in that direction, and the ratio when it protrudes in both directions The larger one.

  FIG. 11 is a diagram illustrating the definition of the parameter “invisibility” that represents the ratio of the boundary rectangle protruding from the screen. In the figure, three patterns, a boundary rectangle 84a, a boundary rectangle 84b, and a boundary rectangle 84c are shown as boundary rectangles. In the figure, “c” is the length of the bounding rectangle protruding from the screen 86 in the horizontal direction (boundary rectangles 84a and 84b), and “d” is the length of the boundary rectangle protruding in the vertical direction (screen 86c). At this time, the parameter “invisibility” is defined as max (c / width × scale, d / height × scale). That is, when it protrudes in either the horizontal direction or the vertical direction, the ratio of the protruding length (c or d) to the length of the boundary rectangle (width × scale or height × scale) in that direction, it protrudes in both directions. When there is, it is assumed that the ratio is larger.

  FIG. 12 shows a description example in XML as a setting file using the above parameters. In the setting file example 200, lines (1) to (4) are descriptions related to the home position of the corresponding hierarchical data. When the home position is displayed by the parameter “zoom” in the “view” attribute of the (3) line. The magnification is set. Here, the “home position” is an initial image of content, and is a display area that can be displayed by pressing a predetermined button of the input device 20. In this example, an image at a magnification of 0.85 is set as the home position.

  Lines (6) to (9) are descriptions relating to the displayable range, and the viewpoint range is set by the “boundary” attribute in line (8). Here, the parameter “stretch” is a magnification representing how many pixels of one layer of the highest resolution among the layer data are extended on the screen, that is, a ratio between the resolution of the display device and the resolution of the image. For example, when displaying an area of 960 × 540 pixels out of a total image of 1920 × 10800 pixels on a display device having a resolution of 1920 × 1080 pixels, the parameter “strech” is “2.0”. In the example of FIG. 12, the upper limit for the parameter is set as “max_stretch”.

  By performing such settings, the maximum magnification that can be displayed on the screen is automatically adjusted according to the resolution of the display device, and an image can be displayed with the same image quality regardless of the resolution of the display device. . Further, “min_zoom” that is the lower limit value of the parameter “zoom” and “max_overshot” that is the upper limit value of the parameter “overshot” are set as a displayable range. Here, the lower limit of the magnification defined by the parameter “zoom” is “0.85”, and the ratio of the screen protruding from the image is “0.5”. The user can move and enlarge / reduce the display area within the range of these settings.

  Lines (11) to (16) are descriptions relating to links to other hierarchical data. In the (12) and (13) lines, the target rectangle of the link is set, the path of the setting file of the link destination is set by the “target” attribute, and the link destination is expanded or reduced by the “direction” attribute. It is set whether to switch to the hierarchical data. In the example of FIG. 12, the attribute file of the linked hierarchical data is “a.xml” and the link direction is “forward”, that is, the setting is switched when expanding. The attribute file and the hierarchical data may be associated by unifying file names excluding the extension, for example.

  Further, the position of the target rectangle is set by the horizontal component and vertical component “center_x” and “center_y” of the parameter “center”, and the size is set by the parameters “width” and “high”. doing.

  In the (14) and (15) lines, a “boundary” attribute sets a “link boundary” that enables the link. The “link boundary” is an extension of the “boundary rectangle” on the image plane in the Z-axis direction of the virtual space by the setting of the “boundary” attribute, and satisfies the link condition when the viewpoint enters the link boundary. To do. (14) By setting the “action” attribute to “jump” in the line, when the viewpoint moves within the link boundary of this setting, the hierarchical data is switched.

  FIG. 13 shows the relationship between the target rectangle 82, the boundary rectangle 84, and the link boundary. In the figure, the horizontal plane including the target rectangle 82 represents the image plane, and the vertical direction represents the enlargement / reduction of the figure, that is, the distance of the viewpoint with respect to the image plane. A rectangle 90 inscribed in the target rectangle 82 is an image of the linked hierarchical data. The boundary rectangle 84 is set as shown in FIG. 9 on the image plane of the target rectangle 82.

  In the present embodiment, the link is validated when the viewpoint moves in the space shown in the figure and approaches the target rectangle 82. Therefore, the validity / invalidity of the link can be set not only by the position on the image plane but also by the proximity of the viewpoint. Therefore, as shown in the figure, the link boundary is three-dimensional with respect to the image plane.

  Returning to FIG. 12, in the (15) line of the setting file, the size of the boundary rectangle is the above-mentioned parameter “scale”, and the position is “offset_x” which is the horizontal component and the vertical component of the parameter “offset”. , “Offset_y”. Further, in order to define the link boundary according to the proximity of the viewpoint, the screen sets an upper limit “max_overshot” of the ratio of the boundary rectangle and the minimum magnification “min_zoom” of the boundary rectangle. The effect of setting the parameter “max_overshot” will be described later.

  Lines (18) to (23) are descriptions relating to moving image reproduction of the embedded object. In line (19), the moving image file to be embedded with the “source” attribute is set as “b.mp4”. In the (20) th line, the target rectangle is set as in the (13) th line. In the (21) and (22) lines, a link boundary for executing moving image reproduction is set with the “boundary” attribute. The link boundary is the same as that shown in FIG. The video is displayed at the position and size inscribed in the center of the target rectangle.

  Here, the “action” attribute is set to “show” (21) line sets conditions for switching display to a preliminary image before actually displaying a moving image. As a preliminary image display, for example, the first frame of a moving image is displayed as a still image. The (22) line in which the “action” attribute is “play” sets conditions for reproducing a moving image and displaying it as a moving image. As a result, it is possible to realize a mode in which when the target rectangle displaying the moving image title is zoomed up, the still image of the first frame is displayed first, and when further enlarged, the moving image is reproduced.

  The setting example of FIG. 12 is such a case. When the parameter “min_zoom” is “0.20”, the first frame is displayed (line (21)), and when the parameter “min_zoom” is “0.35”. The video is played. By making it possible to set a two-stage operation in this way, there is no inconvenience that a moving image is displayed unnecessarily only by approaching the viewpoint. Further, the time lag until the moving image is displayed can be absorbed by the first stage still image display. When the viewpoint is removed from the link boundary of the video playback, the video playback is also stopped. At this time, as long as there is a viewpoint at the link boundary with respect to the first stage still image display, the frame at the time when playback is stopped is displayed as a still image.

  In the (21) and (22) lines, for each “action” attribute, an upper limit “max_overshot” of the ratio that the screen protrudes from the boundary rectangle and an upper limit “max_invisibility” of the ratio that the boundary rectangle protrudes from the screen are set. ing. The effect of setting the parameter “max_invisibility” will be described later.

  Lines (25) to (29) are descriptions relating to audio reproduction among the embedded objects. In line (26), the audio file embedded with the “source” attribute is set as “c.mp3”. In the (27) line, the target rectangle of the embedded object is set as in the (20) line. In line (28), as in the case of moving image reproduction, a “boundary” attribute is used to set a link boundary for performing audio reproduction. However, only “play” representing reproduction is set for the “action” attribute.

  Similarly to the video playback, the audio playback starts when the viewpoint enters the link boundary, and stops when it exits. Or, if another sound originally flows, the sound is switched. When playback of both video and audio is stopped, the stop point on the video or audio data stream is stored in the main memory 60 or the like, and when the viewpoint enters the link boundary again, playback is started from that position. Resume processing to be started may be performed.

  Lines (31) to (35) are descriptions relating to links (store links) to stores on the network among the embedded objects. In line (32), identification information such as a store name is set as “store: AAA” in the “source” attribute. In line (33), the store link target rectangle is set as before. It should be noted that another identification file is stored in the main memory 60 or the like for the location on the network such as the store identification information and the URL.

  In line (34), the “boundary” attribute is used to set a link boundary that enables movement to the store site as before. Here, by setting the “action” attribute to “show_link”, a setting for performing processing related to the link to the store is made. The process related to the link is, for example, processing of a display image such as display of an indicator indicating that movement to a store website is possible. Note that the setting of the (31) to (35) lines may not be a link to a store, but may be a link to a file stored on a hard disk or a site other than a store open on the network.

  Lines (37) to (67) are descriptions relating to selection acceptance display among the embedded objects. The selection acceptance display is a process of highlighting an area that can be enlarged, such as an area for which a link is set, as an option, and moving the image to the area when the user selects one of them. For example, when there are a plurality of link setting areas in the image, the color of each area is changed to indicate that the areas can be selected when the areas become selectable magnifications. In this state, when a direction instruction key such as the cross key 21 of the input device 20 is pressed, the selection target is moved. Then, when a selection operation is performed with the ○ button 22 or the like, the viewpoint is moved or zoomed up so that the selected area comes to the center of the screen. By doing so, the movement of the viewpoint can be simplified, and it is possible to move efficiently to a desired link destination or a part of an image.

  The (39) line sets “link boundary” for accepting selection with the “boundary” attribute. The parameters to be set are the same as described above. The (40) to (43) lines, the (44) to (47) lines, the (48) to (51) lines, and the (52) to (66) lines, respectively, are rectangles of choices based on the “option” attribute. Is specified. Each option specifies a color with a “frame_color” attribute. This highlights the option rectangle. In addition, the position and size of each rectangle are set in the same manner as the target rectangle.

  Furthermore, a viewpoint for display when the option is selected is set by the “view” attribute. The parameters used for setting are the same as described above. In addition, it is shown that there are subordinate options set in the (55) to (65) lines in the options by the settings in the (52) to (66) lines. By clearly indicating this, the lower choices can be excluded from the choices when the upper choice is accepted.

  Next, the parameter “max_overshot” and the parameter “max_invisibility” will be described. FIG. 14 is a diagram for explaining the change of the link boundary when the parameter “max_overshot” is changed. As shown in a frame 94, this figure shows a state in which the positional relation between the field of view 98 from the viewpoint 96 and the boundary rectangle 84 is viewed from the side, and the parameter “max_overshot” is 0.0, 0.25, 0.5. , 0.75, and 1.0, the change of the link boundary is shown.

  First, when “max_overshot” is 0.0, that is, when the screen is set so as not to protrude from the boundary rectangle, the bottom of the triangle representing the field of view at the viewpoint 96a, that is, the screen is in the area above the boundary rectangle 84. The screen does not protrude from the boundary rectangle, and the viewpoint 96a is within the link boundary. On the other hand, at the viewpoint 96b, since the base of the triangle representing the field of view protrudes from the area above the boundary rectangle 84, that is, the area around the boundary rectangle 84 enters the screen, the viewpoint 96b is outside the link boundary. is there. If the viewpoint is within the link boundary, “◯” is marked on the viewpoint, and if the viewpoint is outside the link boundary, “x” is marked on the viewpoint.

  In this way, the link boundary, which is the boundary of the viewpoint set where “max_overshot” satisfies 0.0, has a shape like a link boundary 92a. The same applies when the parameter “max_overshot” is 0.25, 0.5, 0.75, and 1.0. When “max_overshot” is 0.25, the link is valid even if the screen protrudes from the boundary rectangle 84 to ¼. That is, even if the periphery of the boundary rectangle 84 is visible when viewed from a higher viewpoint, the link boundary 92b having the parameter “max_overshot” of 0.25 has a higher shape than the link boundary 92a.

  Similarly, when “max_overshot” increases to 0.5, 0.75, and 1.0, the shape changes like link boundaries 92c, 92d, and 92e. That is, as the viewpoint becomes higher, the boundary rectangle 84 becomes easier to enter the screen, and if the protrusion amount of the screen is allowed, the range of viewpoints that satisfy the condition increases as the viewpoint becomes higher. Become. In this way, by introducing the parameter “max_overshot”, the link is effective against the apparent change in the bounding rectangle and the area where the link is set due to the movement of the viewpoint in the three-dimensional space including the height direction. / Invalid boundaries can be provided.

  When the analog stick 27a or the like of the input device 20 is operated so as to move the display area in the vertical and horizontal directions, the shake width of the image varies depending on the height of the viewpoint even with the same operation amount. With the parameter “max_overshot”, the range in which the link is valid can be easily changed according to the height of the viewpoint, and an operation in consideration of the height of the viewpoint becomes possible.

  FIG. 15 is a diagram for explaining the change of the link boundary when the parameter “max_invisibility” is changed. The way of representing this figure is the same as in FIG. First, when “max_invisibility” is 0.0, that is, when the boundary rectangle is set so as not to protrude from the screen, the boundary rectangle is included in the bottom of the triangle representing the field of view at the viewpoint 96c, that is, the lower area of the screen. The boundary rectangle does not protrude from the screen, and the viewpoint 96c is within the link boundary. On the other hand, at the viewpoint 96d, since the boundary rectangle 84 protrudes from the area below the bottom of the triangle representing the field of view, that is, the area around the boundary rectangle enters the screen, the viewpoint 96d is outside the link boundary. .

  Similarly to the parameter “max_overshot”, the parameter “max_invisibility” is a parameter for enabling the link when the boundary rectangle is on the screen, but the change in the range of the link boundary with respect to the change in the height of the viewpoint is the parameter “ It is different from “max_overshot”. For example, when “max_invisibility” is 0.0, if the viewpoint becomes low and the boundary rectangle 84 is raised too much, the boundary rectangle protrudes from the screen, so that the viewpoint is not included in the link boundary. On the contrary, the higher the viewpoint is, the more difficult it is for the boundary rectangle to protrude from the screen, so the range of viewpoints that fall within the link boundary becomes wider.

  In this way, the link boundary that is the boundary of the viewpoint set that satisfies the parameter “max_invisibility” of 0.0 has a shape like the link boundary 92f. The same applies when the parameter “max_invisibility” is 0.25, 0.5, 0.75, and 1.0. As the parameter “max_invisibility” increases to 0.25, 0.5, 0.75, and 1.0, the boundary rectangle 84 is allowed to protrude from the screen even when the viewpoint is lowered, so the width of the link boundary increases. The shape changes like link boundaries 92g, 92h, 92i, and 92j.

  In the present embodiment, the viewpoint can be freely moved in a three-dimensional space including the height direction within a predetermined range. Therefore, the display area may be moved in the horizontal direction with a large enlargement ratio. If the screen accidentally enters the area where the embedded object is set by such an operation, the moving image may be played back at an unintended timing or the store link screen may be displayed. The protrusion ratio of the boundary rectangle to the screen is an index for determining whether or not the user has intentionally aligned the screen with the boundary rectangle.

  When the user wants to operate the embedded object, the user normally intends to adjust the screen to the boundary rectangle. Therefore, the parameter “max_invisibility” is introduced in setting the link boundary of the embedded object. On the other hand, since switching to another hierarchical data image is desirably performed seamlessly even when the viewpoint is moved in the horizontal direction at a high magnification, the parameter “max_overshot” is effective.

  Next, the operation of the information processing apparatus 10 configured as described above will be described. FIG. 16 shows an example of an image displayed on the display device 12 by the information processing apparatus 10. The display image 300 is an initial image of content, for example. For example, various contents such as software for games, contents for promotion display such as music and movies, contents of virtual department stores consisting of stores on the network, contents of virtual libraries displaying the contents of books, and contents combining them Embodiments are possible.

  By setting the display image 300 as the home position in the setting file, the image is set as the initial image. In the display image 300, six links of rectangular areas 301, 302, 304, 306, 308a, and 310, that is, options are set. The type of link may be display switching to another hierarchical data image or an embedded object. The target rectangle is set for each link in the setting file so as to circumscribe these rectangular areas. The boundary rectangle and the link boundary are also set as described above, but are not explicitly shown in the display image 300.

  The user moves the viewpoint using the input device 20 while viewing the display image 300, and zooms up the rectangular area of interest. When the viewpoint enters the link boundary set for the rectangular area, processing corresponding to the rectangular area, such as switching of hierarchical data and operation of an embedded object, is performed.

  FIG. 17 shows a processing procedure for executing a moving image reproduction process as an embedded object. In the example of the setting file shown in FIG. 12, the moving image playback object is implemented in two stages: a stage in which a certain frame is displayed as a still image and a stage in which a moving image is displayed. A link boundary set for still image display is called a “first link boundary”, and a link boundary set for moving image display is called a “second link boundary”. First, in a state where the viewpoint is outside the first link boundary, a predetermined standard image such as a moving image title image or advertisement is displayed (S18). This standard image is a part of the hierarchical data of the currently displayed image such as the initial image, and corresponds to an image in the rectangular area 301 in the display image 300 of FIG.

  In this state, the link determination unit 116 monitors whether the viewpoint has entered the first link boundary (S20). When the user operates the input device 20 and the viewpoint enters the first link boundary (Y in S20), the object execution unit 117 is notified to that effect, and the object execution unit 117 refers to a predetermined area of the main memory 60. It is confirmed whether there is information to be resumed (S22). If the same moving image has been played before, the stop point at that time is stored in the main memory 60 as a resume point. If there is such information (Y in S22), the information is read (S24). If there is no information to be resumed (N in S22), the first frame of the moving image data read from the main memory 60 is drawn, and if there is resume information (Y in S22, S24), the frame of the resume point is drawn as a still image. The information is displayed on the display device 12 (S26).

  At the same time, the link determination unit 116 monitors whether the viewpoint has entered the second link boundary (S28). When the viewpoint enters the second link boundary (Y in S28), the object execution unit 117 starts playback and display of the moving image (S30). Again, if the resume point is read in S24, the moving image is reproduced from the point. Note that the moving image reproduction process may be speculatively started while displaying a still image before the viewpoint enters the second link boundary. Accordingly, the moving image can be smoothly displayed when the viewpoint enters the second link boundary in S28.

  In parallel with the moving image display, the link determination unit 116 monitors whether the viewpoint has come out of the second link boundary (S32). When the viewpoint comes out of the second link boundary (Y in S32), the object execution unit 117 stops the reproduction of the moving image and displays the frame when stopped as a still image (S34). Further, the stop point is stored in the main memory 60 as a resume point (S36). Furthermore, the link determination unit 116 determines whether the viewpoint has come out of the first link boundary (S38). When the viewpoint comes out of the first link boundary (Y in S38), the information is notified to the display area determining unit 104, and the original hierarchical data image, that is, the standard image such as the title image displayed in S18 is displayed ( S40). With the above operation, it is possible to realize a mode in which the display is switched in two stages and the moving image is played back in response to the zoomed-up area in which the link is set.

  Note that the link determination unit 116 always monitors the position of the viewpoint in practice and makes an internal / external determination with respect to the set link area. Therefore, for example, while the frame of S26 is displayed, the viewpoint does not enter the second link boundary. When it comes out of one link boundary, the process proceeds to the standard image display step of S40 as it is.

  The processing procedure for audio reproduction is the same. At this time, as shown in the example of the setting file in FIG. 12, switching between voice playback and non-playback may be performed with one link boundary, or a plurality of stages are provided to play back voice data, volume, etc. May be switched. In addition, when sound is reproduced as BGM even when the image of FIG. 16 is displayed, processing may be performed so as to crossfade when the sound is switched.

  In both cases of moving images and sound, when the same moving image or sound is reproduced in a plurality of areas in which embedded objects are set in the image of the original hierarchical data, the resume point may be used as common information. For example, in the display image 300 of FIG. 16, even if any of the rectangular areas 301 to 310 is zoomed up, the music is switched to the same music, and the resume point is shared, so that the music continues regardless of the rectangular area. Sexuality is maintained and a sense of unity is born.

  Whether or not to resume or whether or not to make a resume point common may be set by a user or a content creator for each embedded object. In addition, when a certain frame is displayed in S26 of FIG. 17, it may be displayed so as to crossfade with the standard image that has been displayed in the same rectangular area until then. Similarly, even when the viewpoint is moved to the adjacent rectangular area, the displayed still image or moving image may be crossfade.

  FIG. 18 shows an example of an image displayed when a store link is operated as an embedded object. The display image 312 is an image when the viewpoint enters the link boundary set for the rectangular area 308a for which the store link is set in the display image 300 of FIG. By setting the “action” attribute in FIG. 12 to “show_link”, the object execution unit 117 switches the image in the rectangular area 308a in FIG. 16 to the image in the rectangular area 308b in FIG. 18 when the viewpoint enters the link boundary. .

  In the image of the rectangular area 308b, an indicator 314 indicating “GO to store!” Indicating that it is possible to move to the website of the store, and “○ The indicator 316 described as “Please” is displayed in addition to the store information originally displayed. The content and arrangement of the image to be added are determined as appropriate in consideration of the design and the input device. For example, the indicator 314 may be displayed at the stage of the display image 300 in FIG. 16, and when the rectangular area 308a is zoomed up, only the color is changed to indicate that the user can move to the link destination. Good.

  The user zooms up the rectangular area 308a until it becomes possible to move to the link destination. When the image of the rectangular area 308b is obtained, the user presses the “○ button” or the like of the input device 20 according to the display of the indicator 316. The website screen of the store is displayed. General techniques can be applied to access and display the website.

  In the display image 300 of FIG. 16, the rectangular areas 301 to 310 can be considered as “options” in which some kind of link is set. Therefore, in accordance with the setting of the selection reception display illustrated in FIG. 12, highlighting such as changing the colors of the rectangular areas 301 to 310 at a certain timing may be performed to indicate that these are options. The highlighting may be performed when the viewpoint enters the set link boundary or when a predetermined input is performed by the input device 20.

  By clarifying options on the image displayed on the screen and accepting a selection operation by the user, the user can efficiently select a desired rectangular area. In addition, by automatically moving the viewpoint such as moving the selected rectangular area to the center of the screen or zooming up, it is possible to immediately switch hierarchical data and start processing embedded objects with minimal operations. Can do.

  In such an operation, the user selects a rectangular area from the image and performs a shortcut in the direction of increasing the enlargement ratio of the image. However, the shortcut may be performed bidirectionally with respect to the enlargement ratio. For example, after automatically zooming up the rectangular area selected by the user, if a predetermined key such as the X button 23 of the input device 20 is operated, the enlargement ratio is lowered to return to the original display. By doing so, the efficiency of the operation increases when it is desired to select another rectangular area.

  In this case, each time a selection input is made, the object execution unit 117 causes the main memory 60 to store parameters representing the display area such as the frame coordinates at that time. When the user inputs to return the display, the parameter indicating the previous display area is read from the main memory 60 and notified to the display area determination unit 104 to display the corresponding area. When multiple selections are made, such as when there is a lower option for a certain option, parameters representing the display area at that time are stored in order, and read out using the LIFO (Last In First Out) method. Go. Hereinafter, a list of parameters representing the display area stored in this way is referred to as a “breadcrumb list”.

  FIG. 19 shows a processing procedure of the information processing apparatus 10 when selection acceptance display is performed as an embedded object. First, in the state where the display image 300 shown in FIG. 16 is displayed, the user moves the viewpoint using the input device 20. At this time, the link determination unit 116 monitors whether or not a viewpoint has entered the link boundary for selection acceptance display (S50). When the viewpoint enters the link boundary (Y in S50), the object execution unit 117 performs highlighting such as changing the colors of the rectangular areas 301 to 310 according to the setting of the setting file (S52). Then, the user waits for one of the rectangular areas to be selected by operating the direction instruction key such as the cross key 21 and the determination button such as the circle button 22 of the input device 20 (S54).

  When the selection input is made (Y in S54), the object execution unit 117 writes the parameter relating to the display area at that time into the breadcrumb list of the main memory 60 (S56). Then, by notifying the display area determining unit 104 of information on the movement of the viewpoint set in the setting file, the display area is moved, such as zooming up the selected rectangular area (S58). At this time, not only simply moving the display area, but also a predetermined process that is set in association with the rectangular area, such as an explanation of the selected rectangular area, an additional advertisement image, or a color change. May be applied. The image data to be additionally displayed is included in the content in association with the rectangular area. Thereafter, when the user gives an instruction to move the viewpoint such as further zooming up the rectangular area, processing corresponding to the instruction is appropriately performed, but the illustration is omitted in FIG.

  When the user inputs an instruction to return the display area by operating a predetermined key such as the X button 23 of the input device 20 (Y in S60), the object execution unit 117 has recently written from the breadcrumb list in the main memory 60. By reading out the parameters (S62) and notifying the display area determining unit 104, the display is moved to the display area determined by the parameters (S64). By such an operation, when there is a clear intention display of a user's selection input, it is possible to perform display that places importance on efficiency.

  In addition, by marking the display area other than the home position according to the user's selection input, avoiding inefficiency such as losing sight of the destination in high-magnification image display or starting over from the beginning. Can do. As a result, the degree of freedom increases in the input method, and desired information can be efficiently reached even if options are provided in a deep hierarchical structure.

  Although the operation of FIG. 19 relates to the selection reception display, the process of returning the display area may be performed regardless of the selection reception display. For example, the display area determination unit 104 marks a point to be returned together with the hierarchical data switching process. FIG. 20 is a diagram for explaining a mode in which the display area is returned when the viewpoint is arbitrarily moved. The representation of the figure is the same as in FIG. 6, and the hierarchical data 400 is set with links to the hierarchical data 402, 404, and 406. In addition, the hierarchical data 404 is set with a link with the hierarchical data 407. In such content for displaying image data, information on the image 410 at the home position is first stored in the breadcrumb list 408 of the main memory 60. The information stored here may be frame coordinates or hierarchical data identification information.

  It is assumed that the user moves the viewpoint in the virtual space as indicated by arrows A and B using the analog sticks 27 a and 27 b of the input device 20 and enters the area of the hierarchical data 404. As a result, when the display image is switched to that of the hierarchical data 404, the information of the image 412 having the lowest resolution, which should be called the home position of the hierarchical data 404, is added to the breadcrumb list 408. Further, when the user moves the viewpoint as indicated by arrows C and D and enters the area of the hierarchical data 407, the information of the image 414 having the lowest resolution in the hierarchical data 407 is added to the breadcrumb list 408.

  When the user inputs to return the display, the latest image information is read from the breadcrumb list 408 by the LIFO method, and the viewpoint is moved as indicated by an arrow E. At this time, the viewpoint may be literally “moved” while displaying the progress, or the display image may be switched immediately. By doing so, the display can be switched to the hierarchical data 404 continuously, and the viewpoint can be moved again in the hierarchical data 407. In the example of FIG. 20, after the image 414 is displayed, the user switches the display to the hierarchical data 404 by reducing the image (arrow F).

  When the input for returning the display is performed again, the information of the image 412 is read from the breadcrumb list 408 and the viewpoint is moved as indicated by the arrow G. When the user further reduces the image, moves the viewpoint in the image of the hierarchical data 400 as indicated by the arrow H, and inputs to return the display, the information of the image 410 is read from the breadcrumb list 408, and the arrow I Move the viewpoint as follows.

  By such an operation, it is possible to display an image with the lowest resolution in the same layer data by only inputting an instruction to return the display regardless of what viewpoint the user has moved in the same layer data. The low-resolution image is an image that makes it easy to overlook the entire hierarchical data and that can immediately switch the display to the hierarchical data one level higher. The user can quickly reduce the image to a unified image for each hierarchical data without sudden interruption of thinking due to a large image change such as suddenly returning to the initial image, and efficiently and without losing sight of the destination You can reach the area. Further, if the instruction to return is input twice in succession, it is possible to return to the original step by step while roughly following the viewpoint movement history, such as returning to the previous hierarchical data.

  In the example of FIG. 20, the information of the image with the lowest resolution among the hierarchical data is stored as a breadcrumb list, but it is only necessary to define the reference display area with the lowest resolution for each viewpoint range. A plurality of reference display areas may be defined within the same hierarchical data. Also in this case, when the viewpoint enters a new range, a similar mode can be realized by writing information related to the reference display area corresponding to the range into the breadcrumb list. Moreover, you may combine with the aspect of FIG. 19 which highlights an option. In this case, in addition to the information on the images 410, 412, and 414, information on the display area at the time when the selection input is made is stored as a return destination.

  According to the present embodiment described above, the display is switched to another hierarchical data, video playback, audio playback, and a link to another site on the network triggered by movement of the viewpoint including enlargement / reduction of the display image. , And processing for highlighting options. Thus, the user can cause the information processing apparatus to perform a desired process with a sensuous and easy operation of moving the viewpoint while enjoying the image. In addition, such multi-functional content can be created with easy settings using a setting file.

  In addition, since attention is paid to the viewpoint of the image, fine settings can be made depending on the position of the viewpoint, and it is possible to perform stepwise processing rather than simply binary determination of function selection / non-selection. By acquiring the ratio of the screen protruding from the rectangular area, the ratio of the rectangular area protruding from the image, and the like, a desired process can be operated at an appropriate timing according to the content of the process. In processing switching, by performing processing such as cross fading, images and sounds before and after switching can be seamlessly connected. As a result, the user can feel free to try moving image reproduction on the extension of the image display. For these reasons, it is possible to realize content that is superior in design and user-friendly as compared with the case where processing is selected using a cursor.

  Further, by highlighting the options and making them selectable, even if it is desired to directly perform an operation, it is possible to realize a mode according to the user's needs and circumstances without a large change in appearance. Regardless of whether or not such a direct selection has been performed, the display area of the return destination is stored at a resolution lower than that of the currently displayed image, and the display area is directly returned to by a predetermined operation of the input device. By making it possible to move within the image, the efficiency of reaching a desired region increases as a result.

  The present invention has been described based on the embodiments. Those skilled in the art will understand that the above-described embodiment is an exemplification, and that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there.

  For example, in the present embodiment, the movement of the display screen in the three-dimensional virtual space formed with the image plane and the resolution has been described in association with the movement of the viewpoint with respect to the image. In the case where the display screen is changed by changing the center position of the image and the size of the area, the display screen can be regarded as a movement of the virtual viewpoint of the user. That is, the present embodiment can be applied to the movement of the display area on the two-dimensional plane, and the same effect can be obtained.

  DESCRIPTION OF SYMBOLS 1 Information processing system, 10 Information processing apparatus, 12 Display apparatus, 20 Input apparatus, 30 0th hierarchy, 32 1st hierarchy, 34 2nd hierarchy, 36 3rd hierarchy, 38 tile image, 44 Display processing part, 50 Hard disk drive , 60 main memory, 70 buffer memory, 76 frame memory, 80 reference rectangle, 82 target rectangle, 84 boundary rectangle, 92 link boundary, 100 control unit, 102 input information acquisition unit, 103 load unit, 104 display area determination unit, 106 A decoding unit; 114 a display image processing unit; 116 a link determination unit; 117 an object execution unit;

Claims (14)

A storage device that holds hierarchical data configured by layering image data of different resolutions in the order of resolution;
An input information acquisition unit that receives a request to move a viewpoint in a virtual space defined by an image plane and a distance from the image plane;
A display image processing unit that generates a display image from the hierarchical data by changing a display area in response to the viewpoint movement request;
An object execution unit that executes the process when the viewpoint satisfies a condition for starting a process associated with a predetermined region in the image;
With
The information processing apparatus is characterized in that the condition is set with respect to a positional relationship between the predetermined area set on an image plane and an area displayed on a screen determined by the viewpoint.   The information processing apparatus according to claim 1, wherein the condition is that a ratio of a region that protrudes from the predetermined region in the region displayed on the screen is equal to or less than a set value.   The information processing apparatus according to claim 1, wherein the condition is that a ratio of a region that protrudes from the region displayed on the screen among the predetermined region is equal to or less than a set value.   The object execution unit reads out, reproduces and outputs from the memory at least one of moving image data and audio data associated with the area when the viewpoint satisfies a condition set in association with a predetermined area. The information processing apparatus according to claim 1, wherein the information processing apparatus is characterized in that:   The object execution unit reproduces and outputs one frame data included in the moving image data when the viewpoint satisfies a first condition set in association with the predetermined area, and the viewpoint further includes the viewpoint 5. The information processing apparatus according to claim 4, wherein when the second condition set in association with a predetermined area is satisfied, the reproduced moving image data is output.   When the viewpoint deviates from the condition, the object execution unit stops the reproduction of data that has been reproduced until then, records the stop point in the memory, and sets a condition in which the viewpoint is set in association with the same area. 5. The information processing apparatus according to claim 4, wherein when the condition is satisfied again, the information is reproduced and output from the stop point.   The object execution unit records the stop point common to the areas when the same moving image data or audio data is associated with a plurality of areas in the memory. Information processing device.   The object execution unit displays a display indicating that access to a location on the network associated with the area is possible when the viewpoint satisfies a condition set in association with the predetermined area. The information processing apparatus according to claim 1, wherein the information processing apparatus is applied to the information processing apparatus. Reading out hierarchical data composed of image data of different resolutions in the order of resolution from a memory and outputting them to a display device;
Receiving from the user a request to move the viewpoint in a virtual space defined by the image plane and the distance from the image plane;
Changing a region to be displayed on the display device in response to the movement request of the viewpoint;
Executing the process when the viewpoint satisfies a condition for starting the process set in association with a predetermined region in the image;
Including
The information processing method, wherein the condition is set with respect to a positional relationship between the predetermined area set on an image plane and an area displayed on a screen determined by a viewpoint. When the viewpoint satisfies the first condition set in association with the predetermined area, the step of executing the process reproduces and outputs one frame data included in the moving image data associated with the area And steps to
Outputting the reproduced moving image data when the viewpoint further satisfies a second condition set in association with the predetermined area;
The information processing method according to claim 9, further comprising: A function of reading out hierarchical data composed of image data of different resolutions in the order of resolution from a memory and displaying them on a display device;
A function of accepting a request to move a viewpoint in a virtual space defined by an image plane and a distance from the image plane from a user;
A function of changing a region to be displayed on the display device in response to the movement request of the viewpoint;
A function for executing the process when the viewpoint satisfies a condition for starting the process set in association with a predetermined area in the image;
A computer program characterized by causing a computer to realize
The computer program according to claim 1, wherein the condition is set with respect to a positional relationship between the predetermined area set on the image plane and an area displayed on a screen determined by a viewpoint. A function of reading out hierarchical data composed of image data of different resolutions in the order of resolution from a memory and displaying them on a display device;
A function of accepting a request to move a viewpoint in a virtual space defined by an image plane and a distance from the image plane from a user;
A function of changing a region to be displayed on the display device in response to the movement request of the viewpoint;
A function for executing the process when the viewpoint satisfies a condition for starting the process set in association with a predetermined area in the image;
A computer-readable recording medium on which a computer program is recorded,
The recording medium having a computer program recorded thereon, wherein the condition is set with respect to a positional relationship between the predetermined area set on an image plane and an area displayed on a screen determined by a viewpoint. Hierarchical data configured by layering image data of different resolutions in the order of resolution;
This is a condition for starting processing associated with a predetermined area in an image displayed using the hierarchical data, and the predetermined area set on the plane of the image and the distance from the image plane and the image plane A setting file that describes the conditions set for the positional relationship with the area shown on the screen of the image determined by the viewpoint in the virtual space defined by
And the computer displays the image using the hierarchical data on a display device, and the positional relationship is described in the setting file by the movement request for the viewpoint received from the user. A data structure of a content file , wherein processing corresponding to the predetermined area is started when the condition is satisfied . 14. A recording medium on which the data structure of the content file according to claim 13 is recorded.

Images