JPH08263683A - User interface device using display of human body image - Google Patents

Abstract

PURPOSE: To effectively conduct interaction between a user and a device by representing a human body image accompanied with action and expression to the user. CONSTITUTION: An action representation script 112 generated by a computer is analyzed by an action representation script analyzing part 101, and on the basis of an analyzed result, an action pattern generating part 102 refers to an action pattern data base 103, and generates action pattern information 114, and an action parameter modifying part 104 gives modification or adjustment to the action pattern information by referring to an action parameter modification rule 105 and user information 119 or device state information 120, and generates post-processing action pattern information 115. A representation form selecting part 106 refers to a representation form rule 107 and the user information or the device state information on the basis of the analyzed result of the action representation script, and generates display control information 117. A human body image generating part 110 generates the video signal 118 of the designated human body image on the basis of the action pattern information and the display control information, and sends it to a human body image output part 111, and displays the human body image to the user.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a user interface of a computer system or an apparatus incorporating a computer, and more particularly to a human image for presenting a human image accompanied by motions and facial expressions to the user and having a dialogue between the user and the computer. A user interface device by display.

[0002]

2. Description of the Related Art Regarding the technique of a user interface that uses a person's movements and facial expressions, a facial expression generated by texture mapping of computer graphics (CG) is used to make a dialogue with a computer natural. (Artificial Intelligence Society Study Group materials, SIG-SLUD-920
4-2, pp. 9-16, 1992) or a facial expression other than facial expression, such as a listening attitude or a talking attitude, for facilitating dialogue with a computer (8th. Times, Human Interfa
ce symposium, pp. 105-110, 1992
Years).

[0003]

When a user has a dialogue (input / output) with a computer system or a device incorporating a computer, the characteristics and quality of the user interface, which is the dialogue part, are convenient for the system and the entire device. Influence. When image information is used for the user interface, the listability and visibility of the information presented can be improved compared to the dialog using only text information.
It can be expected to improve work efficiency by helping intuitive understanding. By using a person image for this image information, the dialogue with the computer can be made closer to the dialogue between humans in everyday situations,
It is possible to provide a user interface that has low resistance and is easily accepted by a user who is unfamiliar with the operation of the computer or who is uncomfortable with using the computer. In addition, inputting / outputting commands and information via a person image also leads to unifying the windows for transmitting commands to computers and receiving information to the computer. You only need to pay attention to the part of. For example, a user who is unfamiliar with the handling of a system or device can intuitively understand that a human image is related to input / output of information. Also works as.

The user's impression of the user interface greatly changes depending on which display format is selected as the person image. For example, when an inorganic CG human image composed of a simple format is displayed, the dialogue with the human image easily reminds the user of computer-likeness and exchange of symbolic information. In addition, when a real image of a person is displayed, it gives an impression that bidirectional and natural language exchange can be performed, such as a dialogue with humans. Therefore, it is desirable to be able to freely select the display format of the person image according to the type and purpose of the work, the user and the surrounding environment, the operating state of the system and the device, and the like.

However, the above-mentioned conventional techniques cannot change the expression and posture of a human figure or the expression form of a human figure dynamically and freely, and provide a flexible expression according to the use and situation. There is a problem that cannot be done. The monotonous and inappropriate presentation of a person's image not only makes the user feel uncomfortable and deteriorates the operability, but also lowers the impression and evaluation of the system or device and reduces the user's willingness to use, resulting in a decrease in work efficiency. There is a problem that it will end up.

Further, conventionally, when the display of a human figure is used as a user interface, it has been necessary to design depending on the display format of the human figure. Furthermore, there is no relation between the human images in different display formats, and the meaning and content of the information to be output remains the same, and the display format alone cannot be changed. For example, if an application program is designed to display an illustration image as a person image, in order to switch the person image to a CG image or a photographed image by polygon display, or to synthesize the person image, the CG or photographed image is required for each application program. It was not possible to change the display format easily, because it was necessary to add a subroutine and a function for processing, and to individually create the image data of the person necessary for that. If different user interfaces are constructed for each application program like this, even programs that display similar human figures lose uniformity in the meaning of their operation and display, and not only the efficiency of creating application programs is poor. , It also contributes to confuse users who use multiple types of completed programs.

An object of the present invention is to make it possible to display human images in various display formats while accommodating actions and facial expressions in accordance with the operating conditions of the user and the computer system.
In addition, by making it possible to easily design and use these person image displays as the user interface of the device, the user interface that uses the person image display has consistency and unity, and effective and comfortable device operation is possible. To realize the sex.

[0008]

The present invention, the first invention,
Specifying the type and movement of the person displayed on the computer screen,
In a user interface device by a human figure display used for a computer that creates a behavioral expression script in which an operation procedure and the like are described, controls the display of the human figure by the behavioral expression script, and interacts with a user. Action expression script analysis that generates and outputs action information that specifies the action of a person image to be displayed by analyzing the action display script and expression information that specifies elements other than the action such as the attributes and types of the person image and effect effects Section, a motion pattern database that defines a correspondence between a motion pattern that models a motion of a person by numerical values or symbols, and the motion information, and the motion pattern database is collated to generate motion pattern information based on the motion information. And an operation pattern generation unit that collects information about the user and outputs it as user information. And user information collection unit that performs a paid,
A device state information collecting unit that checks the operation state of the computer and outputs or stores it as device state information, and a case of modifying or adjusting the operation pattern information based on the device state information and the user information An operation parameter modification rule that defines a relationship, an operation parameter that refers to the operation parameter modification rule and the device state information or the user information, modifies or adjusts the operation pattern information, and outputs the processed operation pattern information. A modification unit, an expression format rule defining a correspondence relationship between the user information or the apparatus state information or the expression information and the type or attribute of the person image to be displayed or the display format, the expression format rule, the user information, and By referring to the device status information, the type and attributes of the person image to be displayed and the display method are determined based on the expression information, and the display control information is determined. And a person image generation unit that generates a person image specified based on the processed operation pattern information and the display control information and outputs the person image as a video signal. And

According to a second aspect of the present invention, in the first aspect of the present invention, the human image generation unit controls the image generation and the image output processing based on the motion pattern information and the display control information. A generation control unit, a CG person image information storage unit in which information on a human image in the expression form of computer graphics is stored for each operation pattern, and information on a human image in the expression form for an illustration is stored for each operation pattern. An illustration person image information storage unit, a photographed person image information storage unit in which information of a person image in an expression form of a photograph is stored for each operation pattern, and the CG person image information storage unit under the control of the person image generation control unit. And a CG image generation unit for generating a video signal of a human image in a computer graphics expression format, and the illustration person under the control of the human image generation control unit. An illustration image generation unit that generates a video signal of a human image in an expression format of an illustration with reference to the image information storage unit, and an actual image representation with reference to the actual image information storage unit under the control of the human image generation control unit. An illustration image generation unit that generates a video signal of a human image in a format, and a human image output process that outputs a video signal of a human image specified by the display control information to a human image output unit under the control of the human image generation control unit. It is characterized in that it is composed of a section.

According to a third aspect of the present invention, in the second aspect, the person image output processing section applies to a video signal sent from the CG image generating section, the illustration image generating section or the photographed image generating section. An image processing unit that corrects or adjusts or adds a special effect, an image composition switching unit that switches or combines the video signals sent from the image processing unit, and a video signal sent from the image composition switching unit And a final-stage image processing unit for making corrections, adjustments, or adding special effects.

According to a fourth aspect of the present invention, in the second or third aspect, the person image generation unit refers to the voice information storage unit storing voice information and the voice information storage unit, and the person image generation control is performed. And a voice generation unit that generates a voice signal according to the control of the unit, wherein the person image generation control unit is
It is characterized by further performing control of voice generation and voice output processing.

[0012]

DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following description, "dialogue" refers to the exchange of information between a user and a device, that is, the input / output of information. The CG or CG image refers to an image generated based on coordinate values. An illustration (ILLUST) or an illustration image is an image created as a set of pixels by using drawing software with an input device such as a stylus pen, mouse, digitizer, or keyboard, or
A picture drawn by using a drawing means such as a pen or a brush and converted into a set of pixels by an input means such as a scanner or a camera (for example, expressed as bitmap data). What is a live-action (IMAGE) or live-action image?
It refers to an image of the actual space captured by a camera and converted into a set of pixels and rasters. The difference between the illustration and the live-action image is the difference in whether or not the content of the image is drawn by a human hand. Also, when the video signal is represented as fluctuations in physical quantities such as current or voltage amplitude, phase, and frequency (for example, video signals according to the NTSC standard)
In addition, when represented as logical information such as a numerical value or a symbol (for example, this is an image representation by RGB values represented as discrete gradations. Also, drawing based on a given vector value. In the case of outputting to a display device that performs the above, the vector coordinate value and vector amount necessary for drawing are also included. Further, in the following, a computer system or a device incorporating a computer will be simply referred to as a "computer". In carrying out the present invention, in addition to a computer, a device for manually inputting and outputting a motion expression script, or storing and storing the motion expression script in advance
It can also be implemented for a device that can store and reproduce / output the action expression script. An example of the latter is a device (a device that reads information from a memory card, magnetic card, etc.) that stores an action expression script in memory and sequentially outputs it by some trigger.
It is also effective for a purpose of displaying only a specific person image and a purpose of presenting a large number of specific information (for example, a publicity advertisement accompanied by displaying a moving image of a person image).

First, the handling of a person image and its operation in the present invention will be described. In the present invention, processing, control, and management of a human image are performed by distinguishing between "attribute" and "motion" of the human image. The attributes represent, for example, sex, age, occupation, characteristics of physique and appearance. Actions are defined and managed by action patterns. That is, the motion of the person image is expressed by setting a combination of motion patterns and parameters of each motion pattern. A motion pattern is a typical motion of a human being classified and classified into a finite type. Specifically, it is expressed as the direction and amount of displacement of each part of the human body, and the time change of those displacements. In particular, a part that numerically represents the quantitative elements of the operation such as the amount of operation, the speed, and the change with time is called an operation parameter. That is, the operation pattern is described by the type of operation and the operation parameter. The types of motions are, for example, motions classified into "swing head", "raise hands", "stand up", "close eyes", and "laugh". Therefore, the operation pattern is
It is a description of a human motion such as "moving your hand up 10". In the example of FIG. 6, “HEAD F
The sentence "ORWARD 0.3" forms one motion pattern. When different parts of the body are operated at the same time, a plurality of operation patterns can be expressed by a description method such that parentheses are difficult to group together. It is desirable to use an abstract expression with numbers and symbols so that the operation pattern can be easily processed, but this is just an example, and various expressions are possible. Information about the actual part of the human body is stored inside the person image generation unit, and the movement pattern is associated with the person image generated as a video signal and the movement thereof. The type of motion pattern to be prepared and the body part described as a motion are set according to the content and purpose of the work. This is because it is difficult to provide and reproduce all human motions because the human motions are complicated, and the amount of information required to generate a human figure becomes enormous. By limiting the scenes and uses, it is possible to reduce the types of motion patterns and the image information of human images,
It becomes possible to naturally express the movement of a desired human figure with limited resources (for example, storage area and information processing ability). The operation patterns are set in common even when a plurality of types of person images are used. Further, each of the different types of person images has information on each operation pattern. When the resources are abundant and the processing capability is high, it is more effective to register a large number of motion patterns so that a more natural and diverse human image can be displayed.

Here, the action expression script will be described. The action expression script is a series of character or symbol information describing the type of person image displayed on the screen of the computer, the designation of the action, the procedure of the action, and the like. The action designation is based on expressions that directly specify the actions of the human body, for example, a description that corresponds one-to-one with a motion pattern such as "raising a hand", and then a more complicated action such as "greeting,""apologizing," or "highly laughing" It is also possible to describe an expression (in order to use a complicated expression, it is associated with an operation pattern in an operation pattern database). An example of the action expression script is shown in FIG. The part surrounded by VAR_START and VAR_END is a block that defines a variable for controlling the display operation of the human figure (that is, a block of character information).
Is. METHOD_TYPE is an expression format of motions and facial expressions, for example, which expression format of CG, ILLUST (illustration) and IMAGE (actual image) is used, or the expression format is dynamically determined by referring to the state of the user or the system. It is a variable that defines or decides. MODEL
_TYPE is a constant that determines the type of person image to be displayed and whether or not the type is dynamically changed or selected. In FIG. 5, the value of the variable MODEL_TYPE is DYNA.
This is an example of setting in the MIC, but this is a command for instructing to dynamically determine the attributes of a person image based on user information and device state information. When the attribute of the person image is specifically described in this part (for example, AGED_WOMAN,
The written content is preferentially handled in (BOY, SECRETARY, etc.), and the attribute of the person image is determined in consideration of the user information and the device state information. When it is desired to specify the attributes of a person image in more detail, information having the same level of detail as expression information can be described in this block, that is, in the action expression script. MOTION_S
The block enclosed by TART and MOTION_END is a part that defines the type of operation. This includes the definition of the type of action (TYPE) and the time of action (TIME). There is also a method of directly describing the motion pattern in the motion expression script. The method of describing the motion pattern is effective in the case of finely specifying and controlling the motion of the human image displayed on the application software side. In the action expression script, actions and attributes of the person image displayed as such a block are described in order. In the action expression script, not only the action of the person image to be displayed and the type of the person image, but also the purpose of displaying the person image, the effect to be added at the time of displaying, and the sound output in accordance with the movement of the person image Can describe information about audio output.

By describing the display of the human figure by the action expression script, the programmer can particularly display the human figure without being aware of the difference in the expression format and the attributes. In other words, the programmer can not only display a person's image by finely specifying the expression format and attributes, but also
The determination of the expression format and attributes is left to the present invention, and an appropriate person image can be displayed only by instructing and describing the purpose of displaying the person image, the general operation, and the purpose and purpose with the action expression script. In addition, it becomes possible to give an instruction to display a person image having a different expression format in a single description format called an action expression script, and it is possible to reduce the load when creating a program.

Next, an embodiment of the first invention will be described. FIG. 1 is a block diagram showing an embodiment of the present invention. The motion expression script analysis unit 101 analyzes the motion expression script 112 generated on the computer side to generate motion information 113 and expression information 116. The motion information 113 is expressed by the motion pattern generation unit 102, and the expression information 116 is expressed by the expression information 116. Format selection section 1
It is sent to 06. The motion pattern generation unit 102 refers to the motion pattern database 103, generates motion pattern information 114 based on the motion information 113, and sends it to the motion parameter modification unit 104. The operation parameter modification unit 104 includes an operation parameter modification rule 105 and a user information collection unit 108.
The operation pattern information 114 is modified or adjusted by referring to the user information 119 generated in (1) and the device status information 120 generated in the device status information collection unit 109, and the processed operation pattern information 115 is sent to the person image generation unit 110. send. The expression format selection unit 106 refers to the expression format rule 107, the user information 119 and the device status information 120 based on the expression information 116 to generate the display control information 117 and sends it to the person image generation unit 110. The person image generation unit 110 generates a video signal 118 of a person image designated based on the processed operation pattern information 115 and the display control information 117, and outputs the person image output unit 1
Send to 11. The person image output unit 111 displays the person image to the user.

The action expression script analysis unit will be described. As an example, the case where the action expression script as shown in FIG. 5 is received will be described. The action expression script analysis unit first describes VAR_ that describes the characteristics and attributes of the human image.
The definition of the variables in the START block is analyzed and initialized (for example, clearing of specific variables managed by the computer or the present invention and setting of initial values) are executed, and the expression information as the analysis result is displayed. Send to the expression format selection section. Next, MOTION_STAR, which is a block describing the operation
The T portion is analyzed, the type of operation and the time to execute the operation are read out, and operation information is generated and sent to the operation pattern generation unit.

As an example of the expression information, there is one in which the content of the block in which the attribute is described is extracted from the action expression script and the data format and format are adjusted. Similarly, as an example of the motion information, the contents of the block in which the motion is described are extracted from the motion expression script and the data format and format are prepared. A method of converting to another language system or notation may be used in the generation of motion information or expression information.

The operation pattern database will be described. In the motion pattern database, as shown in FIG.
The movement of each part of the body required for one movement, that is, the movement pattern necessary for realizing the designated movement is defined. In the example of FIG. 6, the operation of GREETING is HEAD (head), NECK (neck), and WA.
It is defined by a combination of motion patterns with IST (abdomen). Further, in the example of FIG. 6, HEAD is the part of the motion, FORWARD is the designation of the motion direction of the part,
0.3 is a parameter indicating the operation amount. The operation parameters defined here are general numerical values for operation,
That is, the default value of the operation is defined. The adjustment of the operation parameter is performed in the operation parameter modifier.

The user information collecting unit and the user information will be described. The user information refers to information about the user such as the user's name, age, sex, skill level regarding computer operation work, past device operation history, user preference and physical characteristics. User information input behaviors are also included in the user information. User information includes static information that does not need to be changed or updated once registered, and dynamic information that constantly changes while using the computer. The static information is registered in advance in the computer or the user information collecting unit as text information, and is output as user information as needed. Examples of dynamic information include real-time measurement values such as the user's input action, current position, posture, and motion. Dynamic information collection may be measured at preset fixed time intervals or may be measured every time user information is referenced. In order to collect dynamic information, various sensors are connected to the user information collecting unit or the computer according to the characteristics of the collected information to collect the user information.

The device status information collecting unit and the device status information will be described. The device status information is information indicating the operating status or operating status of a computer or the present invention. For example, the usage time of a computer or program, the elapsed time of work, the name of a process or program being started and executed, information about computer resources such as CPU registers or memory, information processing capacity, information about communication buffers or I / O ports, etc. Input information, presence / absence of various interrupt requests, and information regarding computer operation check including the connection status of peripheral devices. The device status information collecting unit collects these pieces of information, represents and stores them as device status information, and outputs the information in response to a request.

The operation parameter modifier will be described.
The operation parameter modification unit extracts the operation parameter from the received operation pattern information, modifies (or modifies or adjusts) the operation parameter by referring to the user information and the device status information, and modifies the operation parameter portion of the operation pattern information. Substituting the modified numerical value for reconstructing the motion pattern information and outputting it to the person image generating unit. For example, when referring to the device status information including the screen display size information of a person, the output size information on this screen can be used to set parameters for increasing or decreasing the behavior of the operation. It can be carried out. As an example of the actual modification, the original operation parameter is M, the new operation parameter is N,
Let S be the area where the person image obtained from the device state information is output to the screen, and calculate the modified numerical value N such as N = M {2- (S / B) ² }. However, B
Represents the size of all screens acquired from device status information and the like. In this calculation (modification method), the width of motion is changed depending on the ratio of the display area of the human image in the entire screen to improve the visibility. That is, if the display of the person image is small, the movement of the person image is increased.

The expression format selection unit will be described. In the action expression script shown in FIG. 5, the expression format (METHO
D_TYPE) is CG, ILLUST, or IMAG
If a specific designation such as E is specified, the designation is converted into expression information and sent to the expression format selection unit. The expression format selection unit generates display control information instructing to display the specified format according to the expression information. Also,
If there is a designation to dynamically select and determine the attribute DYNAMIC as the expression format, the expression format selection unit refers to the user information and the device state information and determines the expression format of the person image based on the expression format rule.

The expression format rule will be described. The expression format rule defines the correspondence between the expression information and the display control information, or the correspondence between the user information or the device state information and the display control information. FIG. 7 shows an example of the definition in the expression format rule. Rule1 is the age of the user (USER
If _AGE) is 15 years old or younger, this is a description example of a rule that an illustration is selected as a representational form of a human figure.
Rule 2 is a rule for selecting CG and live action when the gender (USER_SEX) of the user is female. Rule 3 is an example in which the correspondence between the elapsed time (USER_TIME) in which the user is using the device and the display format is defined as an example of the device status information. In this example, when the usage time is within 10 minutes, the illustration is selected as the expression format.

The display control information will be described. The display control information includes information that specifies data to be referred when generating a video signal such as an expression format, a type, and an attribute of a person image to be displayed. Further, it also includes information for designating a display method such as an instruction to make adjustments to the generated video signal or add effect effects. If the same expression format, for example, the expression format is an illustration, but data with different person images is provided, specify which image data of the person image represented is used to generate the video signal. .

The person image generator will be described. The person image generation unit generates a person image based on the processed operation pattern information and display control information. Data of a human image necessary for generating a video signal is stored in the human image generation unit in association with an operation pattern, and a function of reading the data of the human image and generating a video signal is provided. Is equipped with. For example, in the case of displaying a CG person image, the person image data is stored as information such as coordinate values or polygons in association with each operation pattern, and the CG image is designated according to the display control information. The information of the human image is sequentially read in accordance with the processed operation pattern information to generate a video signal which is a human image. The generated video signal is output to the person image output unit, but when the video signal is directly output to the signal input terminal of the CRT or the liquid crystal display,
O or display memory area inside the computer (eg VRAM)
In some cases, the person image generator is designed to generate a video signal according to the signal input form of each output destination.

Next, an embodiment of the second invention will be described. FIG. 2 is a block diagram showing an embodiment of the present invention. In the human figure generation unit 110 of the first invention, the processed operation pattern information 115 and the display control information 117 are analyzed by the human figure generation control unit 201, and a control signal 209 for controlling each unit of the human image generation unit 110, 211, 213, 215
Is generated and sent to each part. The CG image generation unit 202 uses C
The G person image information storage unit 205 is referred to, and a video signal which is a designated CG image of a person is generated in accordance with the control signal 209. The illustration image generation unit 203 refers to the illustration person image information storage unit 206 and generates a video signal which is a person image in an illustration format designated according to the control signal 211. The photographed image generation unit 204 refers to the photographed person image information storage unit 207 and generates a video signal which is a photographed person image specified according to the control signal 213. In the portrait image output processing unit 208, according to the control of the control signal 215, the CG image generating unit 202, the illustration image generating unit 203, or the photographed image generating unit 20.
The specified video signal of the video signals output from the No. 4 is output to the person image output unit 111. The expression format does not need to include all of CG, illustration, and live-action, and may be configured alone or in combination as necessary. For example, CG
It will be carried out in a chisel, CG and illustration, live-action and illustration.

The person image generation controller will be described. The person image generation control unit analyzes the processed operation pattern information and the display control information to specifically determine the person image to be actually generated. For example, from the display control information, "the image to be generated is a CG image, male, age 30 years old, height 17
The attributes of the person's image are determined, such as a slim figure at 0 cm, a suit, and a Japanese. There are cases where such a detailed human image attribute is specified by the expression format selection unit 106 and cases where it is determined by the human image generation control unit 201. In the latter case, information and rules for determining the detailed attributes of the person image from the display control information are built in the person image generation control unit in advance. The information designating the attribute of the person image to be displayed is numerically and symbolically transmitted to the designated image generation unit. Specifically, it is sent as an index, address information, pointer information, etc. for designating the information of the person image stored in the person image information storage unit. These pieces of information are sent as control signals to the image generation unit of the specified expression format, but when combining images of a plurality of expression formats, they are sent to a plurality of image generation units that generate the images that are the basis of the combination. . The operation pattern information after the processing is converted into a control signal or included in the control signal and sent to the image generation unit specified by the display control information. The person image generation control unit controls the person image output control unit, for example, controls such as sending and adjusting a video signal and converting a signal format.

The image generation unit and the person image information storage unit will be described. The image generation unit refers to the information of the human image stored in the human image information storage unit according to the control signal to generate a human image as a video signal. The information on the stored person image is stored in a unique format for each expression format. The control signal including information about the attributes and motion patterns of the human image is information necessary for reading information from the human image information storage unit in each image generation unit according to the storage state of the information of the human image of each expression format (for example, , Memory address value, part name and number of human body part, file name and file number, field and frame number, index number,
Time stamp information etc.). The information of the human body may be stored by managing the entire human body or the half body as one unit and storing the information by dividing the body part into fine pieces. When dividing, for example, the head or chest,
Neck, each part of the face (eyes, nose, mouth, eyebrows, etc.), hands (fingers, backs, etc.), arms, torso, legs, etc. are divided and managed and stored. In addition, the information of these person images is stored for each person image having different attributes. In order to make it easy to refer to specific information about a person from the information about the attribute, the correspondence between the attribute and the person image information is defined. For example, it is conceivable that this definition may be stored as table information in the image generation unit or the person image information storage unit and referred to as necessary.

For example, when the expression format is CG, CG
It is represented as information of a human image stored in the human image information storage unit, a set of coordinate values of points, line segments, polygons, and the like, and texture information attached to the polygon.
When representing the movement of a person image, it is stored as information (for example, velocity or acceleration) regarding the displacement of each coordinate value and its time variation. The CG image generation unit generates a person image as computer graphics based on the coordinate values.

Further, for example, when the expression format is an illustration, the information of the human image stored in the illustration human image information storage section is represented as a set in units of planar images. For example, bitmap format, JPEG format (or TIF
Information format capable of describing a plane image such as F or MAG) A person image is stored as a set of plane images. It may be a format for moving images such as MJPEG or MPEG format. The information of these plane images is sequentially read from the illustration image generation unit to generate a video signal.

Further, for example, when the expression format is a live-action image, the information of the human image stored in the real-imaged human image information storage section is stored as a set of planar images as in the case of the illustration, but MJPEG or In addition to the recording format that is intended to be saved as a file format on a recording medium such as the MPEG format, there is a case where it is saved in a format that is aware of raster scanning such as frames or fields. In the latter case, the image information may be stored as an analog signal on a magnetic tape, a magnetic disk, an optical disk, or the like. Information of these images is sequentially read from the photographed image generation unit to generate a person image.

The handling of operation patterns and operation parameters in the image generator will be further described. Expression format is C
In the case of G, taking the information of the motion pattern shown in FIG. 6 as an example, a model of a human image composed of numerical information and polygon data is called based on the display control information, and the head (HEAD ) To the front (FORWAR
An image is generated which displaces to D) by the operating parameter 0.3. Further, when the image is generated while performing the displacement stepwise (for example, the displacement from 0.0 to 0.3 is 0.0
(For example, when performing at 5 intervals and generating an image for each step)
And when generating an image that is immediately displaced (for example,
In the case of generating only an image displaced by 0.3), the former method is used when the motion of the human image is smoothly expressed. The meaning of the motion parameter (in this case, the numerical value of 0.3) means, for example, that the coordinate points grouped as the head in the model of the human image and the center of gravity of the head are moved by a displacement of 0.3. Is done. In this case, the value of 0.3 may be defined as a value depending on the unit of the coordinate system forming the model, or may be defined as a value depending on the subjectivity of the user who observes the human image. It is more appropriate to use the latter definition when the parameters are common to the illustration and the live-action format. For example, when considering the movable range of the head, the stage from the reference point to the limit operating point is defined as 0 to 1,
There is a method of locating a parameter designating the stage. Taking the case of the head as an example, the state in which the head is straight is defined as 0, the state in which the head is completely bent is defined as 1, and so on.

Taking the case where the display format is an illustration as an example, it is assumed that the information of the person image is stored as an image file in the illustration person image information storage section and is sent to the operation pattern information shown in FIG. explain. The illustration image generation unit sequentially reads image files for moving the head forward from the illustration person image information storage unit and generates an image. In this case, there are a method of updating the entire human figure and a method of first reading out a reference image and sequentially overwriting only the portion where the operation occurs. For example, a method is used in which images depicting only the head are overwritten on the whole-body image one after another. With such a partial drawing update method, it is possible to reduce the amount of information on the human image to be stored, and since there are few overwritten drawing portions, it is possible to perform drawing at high speed, resulting in a smooth animation image. It is effective because it can be generated. Also, in the case of moving a plurality of parts in the illustration image, for example, the head and the waist at the same time, information on a person image that moves the head and the waist at the same time is prepared in advance, or The method of overwriting the part of is taken at the same time. The operation parameters are reflected when selecting the image file to read.
An image file showing a large movement is read. For this reason, all the information of the human image that can express the motion within the range that can be set by the operation parameter is stored in the illustration human image information storage unit in advance, or the operation parameter The image to be represented should be able to be generated by image processing such as interpolation or transformation, enlargement, reduction. Further, when the operation parameter represents the speed of the operation, it is used for adjusting the time interval for reading the image file and drawing the information.

When the expression format is a live-action image and the information stored in the live-action portrait image information storage section is in the same file format as the illustration, the operation patterns and operation parameters are the same as in the illustration format. Can be treated to. On the other hand, in the case where an image of a live-action portrait image is recorded as an analog video signal on a magnetic tape, a magnetic disk, an optical disk, etc. in the live-action portrait image information storage unit, a partial drawing, for example, a whole-body screen is drawn first, It may be difficult to overdraw and draw only the head later, and if partial overwriting and combining of such images is performed in the live-action representation format, the drawing position may be noticeable and the user may feel discomfort. is there. In that case, it is effective to store all the images that can be specified by the operation pattern and operation parameters.

The person image output processing section will be described. The simplest process of the person image output processing unit is a switching operation of video signals sent from the CG image generation unit, the illustration image generation unit, or the photographed image generation unit. This is realized by switching the video signal so that the designated video signal is sent to the human image output unit in accordance with the control signal sent from the human image generation control unit. In the person image output processing unit, in addition to switching, adjustment of video signals, for example, brightness, contrast, hue, video sync signal adjustment, video resolution conversion, video size and display position, display timing adjustment,
Furthermore, it is possible to have a function of applying a special effect to an image. Further, there may be a case where a function of changing the format of the video signal output according to the signal format input by the person image output unit is provided. Further, when the formats of the video signals output by the CG image generation section, the illustration image generation section, and the photographed image generation section are different, the CG image generation section, the illustration image generation section, and the photographed image generation section are provided with a function of converting into the same signal input format of the same person image output section.

An embodiment of the third invention will be described. FIG. 3 is a block diagram showing an embodiment of the present invention. In the person image output processing unit 208 of the second invention, the video signal 210, 212 or 214 generated by the CG image generating unit, the illustration image generating unit or the photographed image generating unit is the image processing unit 3.
According to the control signal 215 input to 01, 302, or 303 and output from the person image generation control unit, correction or adjustment or addition of a special effect is performed on each video signal.
The processed video signal is sent to the image composition switching unit 304, and the video signal is switched or combined under the control of the person image generation control unit 201. The processed image is sent to the final stage image processing unit 305, where the final image signal is corrected or adjusted and special effects are added under the control of the human image generation control unit, and the processed image signal 118 is output as a human image. Sent to the department.

The image processing unit corrects or adjusts the sent video signal or adds a special effect.
As corrections and adjustments, brightness, contrast, hue,
Corrections such as hue and gamma corrections that depend on the characteristics of various video equipment and related corrections or cancellation of those corrections,
Examples include adjustment of video synchronization signals, conversion of video signal types including conversion of video resolution, adjustment of video size, display position, and display timing. The special effects include reduction, enlargement, deformation, cutout, inversion of images, application of effector functions and filters for video signals such as wipe and mosaic, and various video signals.

The image composition switching unit performs switching so as to output a designated video signal from a plurality of input video signals according to the control signal. Switching may be instantaneous, or video may be switched in stages, such as overlap or wipe. When synthesizing multiple images, when synthesizing images based on the video signal that is a key (for example, chroma key synthesizing using a color component as a key or synthesizing using brightness information as a key), There are cases where it is used to simultaneously add and display a plurality of images as in the case of overlapping and mixing methods, and cases where window splitting and wipes are used to embed another screen on a reference screen and synthesize.

Under the control of the control signal 215, the final stage image processing unit performs correction and adjustment or addition of special effects to the sent video signal in the same manner as the image processing unit. It also has a function of converting a video signal into a video signal format on the image output unit side, if necessary.

An embodiment of the fourth invention will be described. FIG. 4 is a block diagram showing an embodiment of the present invention. Second or third
In the human figure generation unit of the invention, the human figure generation control unit 20
According to the control of No. 1, the voice generation unit 401 causes the voice information storage unit 40 to
2, the audio signal 404 is generated and the human image output unit 11 is generated.
Send to 1. When presenting video and audio to the user, the person image output unit includes an audio output unit in addition to the image output unit.

A fourth invention is a device for outputting a voice to a user together with the display of a human figure. The voice in the present invention includes not only a voice emitted by a person but also general sound information such as a sound existing in nature, a sound emitted by a machine, and a sound effect. A control signal is output from the person image generation unit to select the audio to be output,
Based on the control signal, the voice generation unit reads necessary voice information from the voice information and outputs the voice. The specification of the voice output is determined in the human figure generation control unit from the post-processing action pattern information and the display control information. Or display control information to the person image generation control unit, and the person image generation control unit defines the processing content related to the output control of voice. Also,
It is also possible to select an output of voice and issue an instruction as display control information when the effect for displaying a human figure is determined by the expression format selection unit. When outputting a word as a voice, a method of describing the output content as text information in the action expression script is also possible. In that case, the text information is sent to the voice generation unit and converted into voice in the voice generation unit. When outputting a typical sound or word, a number or a symbol is added to the voice information to be output and managed, and the voice information can be called by specifying the symbol for the number. The voice information is stored in the voice information storage unit, and the storage format of the information depends on the voice generation method of the voice generation unit. For example, if the voice generation unit adopts a voice generation method such as PCM, the voice information is recorded as a coded voice waveform. Further, when a sound is generated by designating various parameters like the FM sound source, those parameters are stored. In the case of a speech synthesizer that generates a human voice, text information, phoneme, intonation, and information regarding the characteristics of a human voice are stored. Further, by referring to the operation parameter and the display control information, it is possible to adjust and correct the generated sound such as volume and pitch, and determine the type of effect and the depth of effect.

In the case of outputting a voice in accordance with the movement of a human image, the analysis result of the operation pattern and the operation parameter and the image generation status of the image generating unit are sent from the human image generating unit to the voice generating unit as a control signal, and the voice is generated. Timing is generated to generate and output a signal.

The effectiveness of outputting a voice together with a person image will be described. Human behavior is always accompanied by the utterance of voice and noise. In addition, as is the case with production techniques such as movies and television, by presenting sound effects together with video, it is possible to add meaning, emphasis, and production of motion. In this case, it is necessary to output the video and audio in conjunction with each other, but by centrally managing the output of the video and audio in the person image generation control unit, the video and audio can be easily synchronized. it can. Further, by outputting voice, not only visual channels but also auditory channels can be used as a user interface, and it is possible to reduce the need for always paying attention to the human figure. In addition, even when a person image is displayed as a two-dimensional image on the display, the presence of the person image and the realism of the user interface can be enhanced by outputting the sound, particularly by using the stereophonic method such as stereo. Can be increased. By thus outputting the video and audio together, an effective and versatile user interface can be provided.

[0045]

As described above, according to the user interface device for displaying a human image provided by the present invention, a human image of various display formats can be displayed in accordance with the operation status of the user or the computer system, along with the motions and facial expressions. It becomes possible to display them while displaying them, and these person image displays can be easily designed and used as the user interface of the apparatus. In addition, consistency and uniformity are created in the user interface that uses the person image display, and effective and comfortable device operability is realized.

[Brief description of drawings]

FIG. 1 is a diagram showing an embodiment of a configuration of a user interface device for displaying a human figure of the first invention.

FIG. 2 is a diagram showing an embodiment of a configuration of a user interface device for displaying a human figure of the second invention.

FIG. 3 is a diagram showing an embodiment of a configuration of a user interface device for displaying a human figure of the third invention.

FIG. 4 is a diagram showing an embodiment of a configuration of a user interface device for displaying a human figure of a fourth invention.

FIG. 5 is a diagram showing an example of an action expression script according to the present invention.

FIG. 6 is a diagram showing an example of information stored in an operation pattern database of the present invention.

FIG. 7 is a diagram showing an example of definitions in the expression format rule of the present invention.

[Explanation of symbols]

 101 Behavior Expression Script Analysis Unit 102 Behavior Pattern Generation Unit 103 Behavior Pattern Database 104 Behavior Parameter Modification Unit 105 Behavior Parameter Modification Rule 106 Expression Format Selection Unit 107 Expression Format Rule 108 User Information Collection Unit 109 Device State Information Collection Unit 110 Human Image Generation Unit 111 person image output unit 112 action expression script 113 action information 114 action pattern information 115 action pattern information after processing 116 expression information 117 display control information 118 person image signal 119 user information 120 device state information 201 person image generation control unit 202 CG image generation unit 203 Illustration image generation unit 204 Real image generation unit 205 CG person image information storage unit 206 Illustration person image information storage unit 207 Real image person image information storage unit 208 Person image output processing unit 209, 211, 213, 215, 403 Control signal 210, 212, 214 Video signal 301, 302, 303 Image processing unit 304 Image synthesis switching unit 305 Final stage image processing unit 401 Audio generation unit 402 Audio information storage unit 404 Audio signal 405 Video output unit 406 Audio output unit

Claims (4)

[Claims] 1. A motion expression script in which a type of a human image to be displayed on a computer screen, an operation, a procedure of the operation, and the like are described, and the display of the human image is controlled by the operation expression script. In a user interface device for displaying a human image used in a computer that interacts with a user, motion information that specifies the motion of the human image to be displayed by analyzing the motion display script, the attribute and type of the human image, and effect effects Action expression script analysis unit that generates and outputs expression information that specifies elements other than actions, and an operation pattern that defines the correspondence between the operation pattern that models the action of a person by numerical values or symbols and the action information. A motion pattern raw for generating a motion pattern information based on the motion information by collating the database with the motion pattern database Unit, a user information collecting unit that collects information about the user and outputs or stores it as user information, a device state information collecting unit that checks the operating state of the computer and outputs or stores it as device state information, and the device Based on the state information and the user information, the operation parameter modification rule that defines the correspondence when modifying or adjusting the operation pattern information, the operation parameter modification rule and the device state information and the user information. With reference to the operation pattern information, the operation parameter modification unit that modifies or adjusts the operation pattern information and outputs the processed operation pattern information, and the user information or the device state information or the expression information and the type or attribute of the person image to be displayed. And an expression format rule defining a correspondence relationship with a display format, the expression format rule, the user information, and Referring to the placement state information, the type and attributes of the person image to be displayed based on the expression information and determining the display method, the expression format selection unit that generates and outputs display control information, the operation pattern information after the processing and the A user image device for displaying a human image, comprising: a human image generation unit that generates a human image designated based on display control information and outputs the image as a video signal. 2. A person image generation control section for controlling the generation of an image and an image output process based on the motion pattern information and the display control information, and a person in a computer graphics expression format. CG person image information storage unit in which image information is stored for each motion pattern, illustration person image information storage unit in which image information in an expression form for each motion pattern is stored, and live-action expression form A person image in a computer graphics representation format with reference to the CG person image information storage unit under the control of the person image generation control unit A CG image generation unit that generates a video signal of the image, and an illustration table with reference to the illustration person image information storage unit under the control of the person image generation control unit. An illustration image generating unit for generating a video signal of a human image in a format, and an illustration for generating a video signal of a human image in an expression form of a live image by referring to the real image data storage unit under the control of the human image generation control unit. An image generation unit and a person image output processing unit that outputs a video signal of a person image designated by the display control information to a person image output unit under the control of the person image generation control unit. The user interface device according to claim 1, wherein the user image is displayed. 3. An image process in which the person image output processing unit performs correction, adjustment, or addition of a special effect to a video signal sent from the CG image generation unit, the illustration image generation unit, or the photographed image generation unit. Section, an image composition switching section that performs switching or composition of the video signal sent from the image processing section, and an end that performs correction, adjustment, or addition of special effects to the video signal sent from the image composition switching section. The user interface device for displaying a human image according to claim 2, wherein the user interface device comprises a step image processing unit. 4. The person image generation unit further includes a voice information storage unit that stores voice information, and a voice generation unit that refers to the voice information storage unit and generates a voice signal under the control of the person image generation control unit. And the human image generation control unit,
The user interface device for displaying a human image according to claim 2 or 3, further comprising controlling voice generation and voice output processing.