JP2000156849A - Portable information terminal equipment - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a portable information terminal equipment capable of stably displaying an image without blurring even when user's hands move, allowing a user to recognize the unsuitableness of a photographing angle, automatically switching display to a target photographing image, and saving power consumption. SOLUTION: A subject image whose area is extracted by an area extraction part 21 is moved to a prescribed position of a display frame and sent to the opposite party to display a stable picture having no blurs. When the area extracted subject image is brought into contact with the end of the display frame or extruded from the frame, the photographed image is displayed on the user side to allow the user to recognize it. Since a change in the direction of a camera part 4 is detected by a camera direction sensor part 28 and display is automatically switched to a target photographing image, operability can be improved.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a portable information terminal device which is portable and can access information at an arbitrary place.

[0002]

2. Description of the Related Art Conventionally, there is a videophone function as one use form of a portable information terminal device. This function is to make a conversation with a remote party at a remote location by using an image and a voice via a communication line. Generally, a person who has a portable information terminal device displays the other person's face or upper body on the display screen of the portable information terminal device wirelessly outdoors, and also displays the user's face or upper body on the display screen of the other information terminal device. Display the upper body and talk with each other. This videophone function enables a more realistic conversation by seeing the face of the other party than a conventional mobile phone. Also,
The user can take a picture of a scene around the user or a specific object with a camera and send it to the other party, so that a more specific and rich conversation is possible.

[0003]

By the way, in the conventional portable information terminal device, the user holds the portable information terminal device while talking with the other party and uses the camera attached to the portable information terminal device to shoot the subject of the user. 82 is sent to the information terminal device of the other party, and is displayed on the display unit. On the other hand, an image as shown in FIG. 83 taken by the information terminal device of the other party is received and displayed on the display unit of the portable information terminal device. When the user talks with the other party, only the video of the other party is displayed, so it is not known how the user's subject is photographed and sent to the other party. In addition, since the portable information terminal device is held in a hand, a camera shake or a user's body shakes, and an image shot in a shaken state may be sent to the other party. However, since there is no means for monitoring the video being sent to the other party, the user does not know, and the shaking screen is sent as it is, which may make the other party uncomfortable. Therefore, a method of always moving the image of the target object, that is, the image of the face of the user, to the center of the screen by image processing, even if the image is shaken, is considered. For example, consider a case in which the face has shifted to the left side of the screen due to camera shake or body shake as shown in FIG.

[0005] As shown in FIG. 85, a subject and a background 100 of a user enter a camera IF unit (camera interface unit) 25 through a lens 70. The camera IF unit 25 digitally processes the incoming video and then stores it in the image memory 45 for each frame. The data stored in the image memory 45 is output to the outside as video information 160. On the other hand, luminance information and hue information 161 are extracted from the data stored in the image memory 45 and applied to the shape extraction unit 20. In the shape extraction unit 20, the luminance information and the hue information 1
From 61, the user's face shape is extracted using skin color information or hair information, and the like, and shape information 162 is obtained. The optical axis of the lens 70 is mechanically changed by the panning / tilting drive unit 71 so that the face shape is at the center of the display screen. By this loop, the user's subject and background 100
, The stable video information 160 can be obtained. However, this method has a drawback that the optical axis of the lens 70 is mechanically driven and the cost is increased or the reliability is reduced because mechanical parts are used. As another method, as shown in FIG. 86, the subject and the background 100 of the user enter the camera IF unit 25 through the lens 70 and are digitized. The digitized video signal is stored in the image memory 45 in frame units. The luminance information and the hue information 161 are extracted from the digital video stored in the image memory 45 and applied to the shape extracting unit 20. The shape extraction unit 20 extracts the user's face shape from the luminance information and the hue information 161 using skin color information or hair information. Then, the address control unit 72 changes the address of the image memory 45 and moves the entire digital image so that the face shape comes to the center of the screen. Then, the controlled digital video is output to the outside as video information 160.

[0005] However, if the digital image stored in the image memory 45 is within the range of the display frame, if the image moves around the face shape as shown in FIG. 87, the background also moves and the image is missing at the edge of the screen. The portion 123 occurs. Then, a partially missing video is sent to the other party. Therefore, in order to improve this drawback, the digital video stored in the image memory 45 as shown in FIG.
An image is taken in an area larger than 1, that is, an image taking frame 120. Then, there is a method of extracting the face shape 112 by the shape extraction unit 20 and cutting out an image from the photographing frame 120 so that the face shape 112 comes to the center of the display frame 111. However, this method uses a shooting frame 120 larger than the display frame 111.
Therefore, there is a disadvantage that the capacity of the image memory 45 is larger than that required for the normal display frame. Then, as shown in FIG. 89, if the face shape 112 is extremely close to the end of the shooting frame 120, the display frame 111 has a drawback that the image of the area 124 that protrudes from the shooting frame 120 is lost. Further, even in a place where the user is taking a picture with a camera and the user is involved in privacy, in the above-described conventional method, the subject and the background of the user are sent to the other party together. Had to be sent to the other party.

[0006] Further, when photographing the surrounding scenery or an object during a conversation and sending it to the other party, it is necessary to monitor the state of the photographed image by using a portable information terminal device. There is. However, since the subject of the other party is displayed on the display unit, the user has to switch from displaying the subject of the other party once to the image of the surrounding scenery, and the operation is troublesome. Also, conventionally, even when the surrounding scenery is temporarily displayed on the display screen, processing for decoding the video sent from the other party is performed, and power in the decoding processing unit is wastefully consumed. Was. The present invention has been made in view of such a problem, and an object of the present invention is to provide a portable information terminal device that can display images stably without shaking on the other party even when the housing is shaken due to camera shake or the like. Aim. It is another object of the present invention to provide a portable information terminal device that allows a user to recognize when a user's shooting angle is not appropriate. Furthermore, the present invention provides a portable information terminal device that can automatically switch the image displayed on the user side to the image of the object when the direction of the camera is changed to the object such as the scenery. The purpose is to.

Still another object of the present invention is to provide a portable information terminal device capable of saving power consumption.

[0008]

According to a first aspect of the present invention, there is provided a portable information terminal device according to the present invention, wherein: a photographing means for photographing a subject and a background of a user by a camera in a state where a housing is held by the user; A region extraction unit that extracts a region of a subject photographed by the photographing unit by a predetermined process to obtain a region-extracted subject image; and a direction in which the subject image obtained by the region extraction unit moves from a predetermined position in a predetermined range. Detecting means for detecting a moving distance, a means for moving a subject image to a predetermined position by using a detection result of the detecting means, and a communication means for communicating with a counterpart information terminal device via a predetermined communication line. Video transmission control means for transmitting the subject video moved to a predetermined position to the partner information terminal device by the communication means. With such a configuration, even if the image shot by the camera shakes due to hand shaking or shaking of the user's body, a predetermined area is extracted and moved to a predetermined position and then sent to the other party. In, the image is displayed stably without significant shaking. Since the background video is not sent to the other party, the video at the edge of the screen is not lost. Further, since the region to be transmitted is extracted from the video, the amount of transmitted information can be reduced.

According to a second aspect of the present invention, in the portable information terminal device according to the first aspect, the object image contacts the end of the predetermined area or at least a part of the object image is in the predetermined area by the detecting means. A feature is provided in which the user is provided with a means for displaying the subject and the background of the user when the protruding part is detected. According to such a configuration, hand shake or shaking of the user's body becomes severe, and when the subject image comes into contact with a predetermined range, for example, the edge of the display screen, or protrudes from the predetermined range, for example, from the display screen, the subject automatically self-contains. Of the portable information terminal device can be changed so that it can be photographed at a normal angle by changing the way of holding the portable information terminal device. According to a third aspect of the present invention, in the portable information terminal device according to the first aspect, when a subject image moved to a predetermined position in a predetermined range is enlarged and protrudes from a predetermined range, the image includes a subject and a background. Means for transmitting an image to the other information terminal device. According to such a configuration, when the subject image protrudes from a predetermined range, for example, from the display screen, the area of the background portion becomes extremely narrow, and there is no need to delete the background portion. Send the raw video to the other party. Therefore, the region extracting means can omit the region extracting process.

According to a fourth aspect of the present invention, in the portable information terminal device according to the first or second aspect, means for accumulating a subject image immediately before the subject image protrudes from a predetermined range; Means for reading out the stored subject video as a still image when the image is outside the range, and transmitting the still image to the counterpart information terminal device via the communication line during the period outside the predetermined range.
According to this configuration, while the subject image protrudes from a predetermined range, for example, the edge of the display screen, and sends the latest normal subject image to the other party while the user changes the holding style of the portable information terminal device, Does not need to send the shaky subject image taken while the user is correcting the orientation of the portable information terminal device to the other party, and the other party can always see a stable subject image on the user side. A portable information terminal device according to a fifth aspect of the present invention includes: a photographing means for photographing a subject and a background of a user with a camera in a photographing area wider than a display area while the housing is held by the user; Area extracting means for extracting a subject in a photographing area photographed by the means by a predetermined process to obtain a subject image extracted from the area; means for moving the display area so that the subject image fits in the display area; Detecting means for detecting a direction and a moving distance of the subject image obtained by the extracting means from a predetermined position in the display area, and means for moving the subject image to a predetermined position by using a detection result of the detecting means; Communication means for communicating with the information terminal of the other party via a predetermined communication line, and transmitting the video of the subject moved to a predetermined position to the information terminal of the other party by the communication means Characterized by comprising a image transmission control means.

With such a configuration, by extracting a user's subject in the photographing region and obtaining a subject image, the movable range of the subject image is expanded. According to a sixth aspect of the present invention, in the portable information terminal device according to the fifth aspect, it is detected that the subject video has touched an edge of the shooting area or at least a part of the subject video has protruded from the shooting area. In this case, the user is provided with means for displaying the subject and the background of the user. With such a configuration, hand shaking or shaking of the user's body becomes severe, and when the subject image comes into contact with the edge of the shooting area or protrudes from the shooting area, the image shot by oneself is automatically displayed. It can be viewed and can be modified so that the portable information terminal device can be photographed at a normal angle by changing the holding method. According to a seventh aspect of the present invention, in the portable information terminal device according to the fifth or sixth aspect, means for accumulating a subject image immediately before the subject image protrudes from the shooting region, and a subject image protruding from the shooting region Means for reading out the stored subject video as a still image and transmitting the still video to the partner information terminal device via the communication line while the video is out of the shooting area.

[0012] With this configuration, while the subject image protrudes from the shooting area and the user changes the way of holding the portable information terminal device, the latest normal subject image is sent to the other party, so that the user can carry the portable information terminal device. It is not necessary to send the subject image with severe shaking taken while the direction of the information terminal device is being corrected to the other party, and the other party can always see the stable subject image of the user. The portable information terminal device according to the present invention as defined in claim 8, wherein a photographing means for photographing a subject and a background of the user with a camera in a state in which the housing is held by the user, and a subject photographed by the photographing means. Area extracting means for extracting an area by the processing of the above, obtaining an object image of the extracted area, means for detecting the mounting direction of the camera in the housing, and the means for mounting the camera in a direction deviating from the shooting direction of the subject by this means. When it is detected that there is, a means for displaying an image captured by the camera on the user side, a communication means for communicating with the information terminal device of the other party via a predetermined communication line, In the case of, the camera shoots the image of the subject from which the area has been extracted. Characterized by comprising a video transmission control means for transmitting to the other information terminal device by the communication unit images were.

With this configuration, when the user is talking with the other party and wants to show the surrounding scenery video to the other party, for example, when the direction of the camera is changed, the portable information terminal device on the user side is displayed. Since the video is automatically switched from the video of the other party to the video of the landscape, the user can operate the camera to adjust the angle to an appropriate angle while watching the video of the landscape, thereby improving operability. According to a ninth aspect of the present invention, in the portable information terminal device according to the first aspect, an infrared radiating unit and an infrared camera provided in a housing, and an infrared camera that reflects reflected infrared light radiated from the infrared radiating unit. A means for extracting shape information of a subject by photographing, and a shape control means for controlling shape information such that a ratio of a subject image to a predetermined range is a predetermined ratio. The subject photographed by the means is subjected to region extraction using the shape information controlled by the shape control means, and a subject image in which the region is extracted is obtained. With such a configuration, even if the image captured by the camera shakes due to hand shaking or shaking of the user's body, the subject image is extracted to a certain size, moved to a predetermined position, and then moved to a partner. Since the image is transmitted, the image is displayed stably without much shaking on the other side.

According to a tenth aspect of the present invention, in the portable information terminal device according to the first aspect, an ultrasonic transmitting unit and an ultrasonic receiving unit provided on a housing, and an ultrasonic wave transmitted from the ultrasonic transmitting unit. The apparatus comprises means for extracting shape information of a subject by receiving a sound wave pulse by an ultrasonic receiving unit, and shape control means for controlling shape information such that a ratio of a subject image to a predetermined range is a predetermined ratio. The area extracting means extracts an area of a subject photographed by the photographing means using the shape information controlled by the shape control means, and obtains an image of the subject from which the area is extracted. With such a configuration, even if the image captured by the camera shakes due to hand shaking or shaking of the user's body, the subject image is extracted to a certain size, moved to a predetermined position, and then moved to a partner. Since the image is transmitted, the image is displayed stably without much shaking on the other side. Claim 11
The portable information terminal device according to the present invention described in any of (1) to (3), further comprising: photographing means for photographing a subject and a background of the user with a camera in a state where the housing is held by the user; Means, a means for displaying the subject and background of the user for a predetermined period on the user side when the displacement of the shaking detected by the means exceeds a predetermined threshold value, and a counterpart information terminal via a predetermined communication line Communication means for communicating with the device and video transmission control means for transmitting a video including the user's subject to the information terminal of the other party by the communication means are provided.

[0015] With this configuration, when the user is holding the portable information terminal device and displaying the image of the other party on the display means to have a conversation with the other party, the housing is greatly shaken by hand shaking or the like. Then, by switching the display content displayed on the display means to the user's subject and the background, the user can confirm how much the user's subject has deviated from the display frame, so that the image of the subject is in a normal position. The orientation of the portable information terminal device can be corrected. According to a twelfth aspect of the present invention, there is provided a portable information terminal device according to the present invention, wherein a photographing means for photographing a subject and a background of a user by a camera in a state where the housing is held by the user, and a user's subject and a background determined by the user. Means for displaying at a predetermined period and a predetermined period, communication means for communicating with the other-side information terminal device via a predetermined communication line, and transmitting an image including the subject of the user to the other-side information terminal device by the communication means. Video transmission control means. With such a configuration, when the video of the other party is displayed, the subject and the background of the user can be displayed periodically, and the displayed video can be checked at any time, and the video of the subject that is out of the display frame can be checked. By changing the orientation of the portable information terminal device each time, the image of the subject can be corrected to a normal position.

According to a thirteenth aspect of the present invention, the portable information terminal device according to the present invention includes a photographing means for photographing a subject and a background of the user with a camera in a state where the housing is held by the user, and a predetermined communication line. Communication means for communicating with the other party's information terminal device, video transmission control means for transmitting a video including the user's subject to the other party's information terminal device by the communication means, and detecting a silent section of the voice uttered by the user. ,
First detecting means for detecting a pause period exceeding a predetermined length from the silent section, and receiving a voice transmitted from the partner information terminal device via the communication means, detecting a silent section of the voice, and A second detecting means for detecting a pause period exceeding a predetermined length from the section, and a means for displaying the user's subject and background on the user side during the pause periods detected by the first and second detection means. It is characterized by having. With such a configuration, in the silent section of the voice uttered by the user or the other party, the display is switched to the display of the video on the user side when the duration is longer than a predetermined length, that is, when the conversation is paused, so that there is sufficient time. Can be switched. A portable information terminal device according to the present invention according to claim 14, wherein a photographing means for photographing a subject and a background of the user with a camera in a state where the housing is held by the user,
Communication means for communicating with the other information terminal device via a predetermined communication line; video transmission control means for transmitting a video including the user's subject to the other information terminal device by the communication means; The apparatus is characterized by comprising means for detecting an utterance period when the power exceeds a predetermined threshold and continuing for a predetermined time, and means for displaying a user's subject and background on the user side during the utterance period.

With such a configuration, by switching to the display of the user's subject and the background while the user is uttering, there is an advantage that the user can pause while watching the uttering user's subject. A portable information terminal device according to the present invention as set forth in claim 15, further comprising: photographing means for photographing a subject and a background of the user with a camera in a state where the housing is held by the user; Communication means for communicating with the terminal device; video transmission control means for transmitting a video including the subject of the user to the other information terminal device by the communication means; and a user side for starting communication with the other information terminal device by the communication means. And means for displaying a subject and a background of the user. With such a configuration, in the first step of starting communication, after confirming that the subject of the user is normally contained in the display frame, the conversation with the other party is started. Therefore, the transmission of the video including the subject of the user who does not fit in the display frame does not occur.
The portable information terminal device according to the present invention according to claim 16,
Photographing means for photographing a subject and a background of the user by a camera while the housing is held by the user; communication means for communicating with a counterpart information terminal device via a predetermined communication line; and an image including the subject of the user Video transmission control means for transmitting to the information terminal of the other party by communication means, means for detecting the period during which the video of the user is displayed on the user side, and the time period is transmitted from the other information terminal apparatus by the communication means Means for stopping the supply of power to the decoding means for the decoded video information.

[0018] With such a configuration, while the video on the user side is being displayed, the supply of power to the decoding means for the video information transmitted from the partner information terminal device is stopped, so that the portable information terminal is stopped. The power consumption of the device can be saved. The present invention according to claim 17 is:
In the portable information terminal device according to any one of claims 2, 6, 11 to 15, when the user displays the subject and the background of the user, the portable information terminal device is transmitted from the partner information terminal device. Means for displaying the image in a multi-window together with the image. With such a configuration, the video of the user and the video of the other party are displayed in a multi-window together with the display means of the portable information terminal device of the user, so that the video of the other party is not interrupted and the subject of the user is also displayed at the same time. And let the user
It is possible to check how much the user's subject is out of the display frame, and correct the orientation of the portable information terminal device so that the image of the subject is displayed at a normal position. The present invention according to claim 18 is the portable information terminal device according to claim 11, wherein when displaying the subject and the background of the user on the user side, the multi-window is displayed together with the video transmitted from the partner information terminal device. Means to display with,
A threshold is divided into a plurality of grades, and means for applying the displacement of shaking to the grade, and means for changing the size of the display of the subject and the background of the user according to the grade are provided.

With such a configuration, when displaying in the multi-window, the size of the multi-window display of the image on the user side is changed according to the size of the shaking of the housing due to hand shaking or the like, so that the shaking is large. At this time, the display becomes large, and it becomes easy to adjust the orientation of the portable information terminal device so that the user's subject is displayed at a normal position. According to a nineteenth aspect of the present invention, in the portable information terminal device according to the twelfth aspect, a means for setting the period and the period is provided. With such a configuration, the user can freely set the period and cycle for displaying the subject and the background of the user. Claim 20
According to the present invention, in the portable information terminal device according to the twelfth aspect, a means for detecting a displacement of the shaking when the housing is shaken, and the cycle is set in accordance with the displacement of the shaking detected by the means. Changing means. With such a configuration, the period at which the subject and the background of the user are displayed can be changed in accordance with the magnitude of the shaking. When the shaking is large, the period is shortened and the subject and the background of the user are frequently displayed. This makes it easy to adjust the orientation of the portable information terminal device so that the user's subject is displayed at a normal position.

The present invention described in claim 21 is directed to claim 11.
Alternatively, in the portable information terminal device according to claim 12, when displaying the subject and the background of the user on the user side, the displacement of the shake when the housing is shaken is performed by the user performing an operation of correcting the hand shake. Means for extending the display of the subject and the background of the user during a period in which the threshold value exceeds a predetermined threshold value. With such a configuration, when the user's subject is out of the display frame when inserting the user's video in the process of displaying the video of the other party, the user changes the orientation of the portable information terminal device. By displaying the subject and the background of the user on the display means while the adjustment is performed so as to fit in the display frame, the user can change the direction of the portable information terminal device accurately with sufficient time. . The portable information terminal device according to the present invention as set forth in claim 22, further comprising: photographing means for photographing a subject and a background of the user with a camera in a state where the housing is held by the user; Communication means for communicating with the terminal device, video transmission control means for transmitting an image including the user's subject to the other information terminal device by the communication means, and detecting displacement of hand shake when the housing is shaken by hand shake Detection means, and means for canceling the hand-shake component in the video by adding a hand-shake displacement and a hand-shake component detected by the detection means to the image of the user's subject and the background captured by the camera, A means for detecting a shaking component of the user's subject from the image in which the hand shaking component has been canceled by this means, and a magnitude of the shaking component of the subject detected by this means that a predetermined threshold value is set. When was example, it is characterized by comprising a means for displaying the user of the object and the background in the user side.

With such a configuration, the hand-shake component is canceled out, for example, by adaptively adjusting the optical axis of the camera lens, from the shake component obtained by combining the hand-shake video component and the subject shake component, and the user's subject is removed. By extracting the swing component of the user, when the swing component of the subject exceeds a predetermined threshold, the user's subject and the background are displayed on the display means of the portable information terminal device, so that how much the user's subject is out of the display frame It can be off or alert the user.

[0022]

Embodiments of the present invention will be described below in detail with reference to the drawings. In the drawings, the same reference numerals indicate the same or corresponding parts. (Embodiment 1) First, Embodiment 1 of a portable information terminal device according to the present invention will be described. FIG. 1 is a block diagram illustrating a configuration of the portable information terminal device according to the first embodiment. In this figure, what is denoted by reference numeral 1 is the terminal body, and 3
Is an earphone microphone, and 4 is a camera unit. The terminal body 1 includes a main control unit 11, a video decoder 12, a video LC
D control circuit 13, LCD 14 for video, LC for text
D control circuit section 15, text LCD 16, demultiplexing section 17, PHS line IF section (PHS line interface section) 18, antenna 19, area extraction section 21, audio codec 23, earphone microphone terminal 24, camera IF section (camera Interface section) 25, camera terminal 26,
It has a video encoder 27, a camera orientation sensor unit 28, an operation input control circuit unit 29, a touch panel 30, a scroll dial 31, operation buttons 32, a power button 34, and a power unit 36. Among them, the main control unit 11, the video decoder 12, the video LCD control circuit unit 13, the text LCD
Control circuit unit 15, demultiplexing unit 17, PHS line IF unit 1
8, area extraction unit 21, audio codec 23, camera IF
Unit 25, video encoder 27, camera orientation sensor unit 2
The operation input control circuit unit 29 and the power supply unit 36 are connected to each other via a main bus 37.

The video decoder 12, the demultiplexer 1
7, the audio codec 23 and the video encoder 27
They are connected to each other via a synchronous bus 38. The main control unit 11 includes a CPU, a ROM, a RAM, and the like, and realizes an operation as a portable information terminal device by comprehensively controlling each unit of the terminal body 1. The video decoder 12 decodes the encoded video data and outputs the reproduced video data to the video LCD control circuit unit 13.
Give to The video LCD control circuit 13 controls the video LCD 14 to display the video indicated by the video data supplied from the video decoder 12. LCD 14 for video
Is a color LCD having a resolution sufficient to display a video (for example, an MPEG4 video), and displays a video under the control of the video LCD control circuit unit 13. The text LCD control circuit unit 15 controls the text LCD 16 so as to display the text image indicated by the text data provided from the main control unit 11. LCD 16 for text
Is a black-and-white LCD having a larger area and a lower resolution than the video LCD 14, and the text LCD control circuit 1
The text image is displayed under the control of No. 5.

The demultiplexing unit 17 has two operation modes, a multimedia communication mode and a voice communication mode, and operates in a mode specified by the main control unit 11.
In the multimedia communication mode, the demultiplexer 17 encodes video data supplied from the video encoder 27 via the synchronization bus 38,
8 and the other data provided from the main control unit 11 are converted into a predetermined multiplexing method (for example, H.221 of the ITU-T recommendation or H.221 of the ITU-T recommendation is modified). ), And the transmission data obtained by this is supplied to the PHS line IF unit 18. In the multimedia communication mode, the demultiplexer 1
7 separates coded video data, coded audio data, and other data from the transmission data supplied from the PHS line IF unit 18 and separates these data into the video decoder 12, the audio codec 23, and the main control unit 11. Give to each of the. On the other hand, in the voice communication mode, the demultiplexing unit 17 transmits a signal from the voice codec 23 to the synchronization bus 38.
PHS the encoded voice data given via
It is given to the line IF unit 18. In the voice communication mode, the demultiplexing unit 17 directly supplies the transmission data (coded voice data) provided from the PHS line IF unit 18 to the voice codec 23.

The PHS line IF unit 18 wirelessly transmits a PHS (Personal 1 Hand) through an antenna 19.
Yphone System) network and PHS
In addition to performing various types of call processing for performing communication via a network, transmission / reception of transmission data is performed via a communication bus set via the PHS network. The region extracting unit 21 operates in the multimedia communication mode, and cuts out the image of the head, that is, the image of the face, from the image supplied from the camera IF unit 25 in frame units. The voice codec 23 has two operation modes, a multimedia communication mode and a voice communication mode, and operates in the mode specified by the main control unit 11. Audio codec 2 in multimedia communication mode
3 digitizes an audio signal output from the earphone microphone 3 connected via the earphone microphone terminal 24 and performs a predetermined low-rate audio encoding method (for example, I / O).
The encoded audio data is obtained by encoding according to TU-T recommendation G729). The audio codec 23 supplies the encoded audio data to the demultiplexer 17 via the synchronization bus 38. In the multimedia communication mode, the audio codec 23 decodes the low-rate audio code in the encoded audio data supplied from the demultiplexing unit 17 and converts it into an analog signal to obtain an audio signal. The audio codec 23 supplies the audio signal to the earphone microphone 3.

On the other hand, in the voice communication mode, the voice codec 23 digitizes the voice signal output from the earphone microphone 3 connected via the earphone microphone terminal 24 and simultaneously converts the voice signal into a 32 kbps ADPCM system (ITU-M). The encoded audio data is obtained by encoding according to G recommendation T726 of T Recommendation). The audio codec 23 converts the encoded audio data via the synchronization bus 38 into the demultiplexer 1.
Give to 7. In the voice communication mode, the voice codec 23 decodes the ADPCM code in the coded voice data supplied from the demultiplexing unit 17 and converts it into an analog signal to obtain a voice signal. The audio codec 23
This audio signal is provided to the earphone microphone 3. Note that the earphone microphone 3 converts the surrounding sound into an audio signal and gives it to the audio codec 23,
Output as a voice. The earphone microphone 3 is detachable from the terminal body 1. The camera IF unit 25 takes in a video signal output from the camera unit 4 connected via the camera terminal 26 and digitizes the video signal to obtain video data. Camera IF section 2
5 is a video encoder 27 and an area extraction unit 2
Give to 1.

The video encoder 27 includes a camera IF unit 25
Alternatively, encoded video data is obtained by encoding the video data provided from the area extracting unit 21. Video encoder 27
Supplies the encoded video data to the video decoder 12 and the demultiplexer 17. The region extracting unit 21 cuts out a region image of a subject portion from the input image. The clipped video is sent to the video encoder 27. The camera unit 4 uses a CCD camera or the like. The camera unit 4 is detachable from the terminal body 1. In addition, the camera unit 4 captures the image in the same direction as the side on which the display surfaces of the video LCD 14 and the text LCD 16 are provided, and the rear side with respect to the side on which the display surfaces of the video LCD 14 and the text LCD 16 are provided. It is possible to mount the camera in two states, that is, a state in which the direction is photographed. Camera orientation sensor unit 28
Detects whether the camera unit 4 is mounted and which of the two mounting states the camera unit 4 is mounted in. The operation input control circuit unit 29 includes a touch panel 30,
The scroll dial 31, the operation button 32, and the power button 34 are respectively connected. The operation input control circuit unit 29 receives a user's instruction operation on the touch panel 30, the scroll dial 31, the operation button 32, and the power button 34, and notifies the main control unit 11 of the content of the instruction operation.

The touch panel 30 is an LCD 1 for text.
LCD for text, which is placed on the display surface of No.6
It is for receiving various inputs corresponding to the display contents of No. 16. The scroll dial 31 is for receiving instructions such as cursor movement and scrolling of a display screen. The operation button 32 is for receiving an input of a determination instruction or a cancellation instruction. The power button 34 is for receiving an instruction for turning on / off the operation of the portable information terminal device. The power supply unit 36 has, for example, a battery as a power source, and supplies power to each unit of the terminal body 1. The power supply unit 36 turns on / off power supply to each unit under the control of the main control unit 11. However, the power supply unit 36 always supplies power to at least the main control unit 11 and the operation input control circuit unit 29. FIG. 2 shows the terminal body 1 and the camera unit 4
FIG. (A) of this figure is a front view,
(B) is a left side view, (c) is a right side view, and (d) is a top view. The same parts as those in FIG. 1 are denoted by the same reference numerals. As shown in this figure, the terminal main body 1 has a box-shaped housing 60, and the above-described components of the terminal main body 1 are accommodated in the housing 60.

LCD 14 for video and LCD 1 for text
Reference numeral 6 is provided in a state where the display surface is exposed to the outside of the housing 60 from one surface of the housing 60. The scroll dial 31 includes an image LCD 14 and a text LCD 16.
Of the four surfaces (hereinafter, referred to as side surfaces of the housing) intersecting with the surfaces (hereinafter, referred to as the front surfaces of the housing), respectively (shown on the right side in FIG. 2A here). On the side of the housing). Further, the operation button 32 and the power button 34 are provided on a side surface of the housing 60 (here, the side surface of the housing shown at the upper side in FIG. 2A) adjacent to the surface of the housing 60 on which the scroll dial 31 is provided. I have. Note that the scroll dial 31 and the operation buttons 32 are
The relative position is determined such that the scroll button 31 can be operated with the thumb of the same hand and the operation button 32 can be operated with the remaining fingers of the same hand while the end of the zero is placed on the palm. . The earphone microphone terminal 24 is provided on the same side surface of the housing as the one on which the scroll dial 31 is provided. The position of the earphone microphone terminal 24 is determined so that the earphone microphone 3 (not shown in FIG. 2) does not hinder the operation of the scroll dial 31 even when the earphone microphone 3 is mounted.

The camera terminal 26 is provided on the side of the housing opposite to the side of the housing on which the scroll dial 31 is provided (here, the side of the housing shown on the left side in FIG. 2A). I have. The camera unit 4 includes a camera unit body 4
a and the support portion 4b are connected by a hinge portion 4c, and the terminal is inserted by inserting the support portion 4b into a concave portion 61 provided on the same side of the housing where the operation button 32 and the power button 34 are provided. It is attached to the main body 1. Further camera unit 4
Is electrically connected to the terminal body 1 by inserting a plug provided at the end of a connection wire (not shown) into the camera terminal 26. The concave portion 61 is a concave portion 61-1 near the front of the housing.
And a concave portion 61-2 near the surface of the housing 60 opposite to the front surface of the housing (hereinafter, referred to as the rear surface of the housing). As a result, as shown in FIG. 2, by inserting the support 4b into the recess 61-1 near the front of the housing, the camera 4
In addition to setting the photographing direction of the camera unit 4 to the front side of the housing, the camera unit 4 is photographed by inserting the support unit 4b into the recess 61-2 near the rear surface of the housing with the direction of the camera unit 4 reversed from that of FIG. The direction may be the rear side of the housing. The camera body 4a
Is rotatable around the hinge 4c, and the photographing angle can be changed.

Next, the operation of the portable information terminal device of this embodiment configured as described above will be described. First,
When the power is off, the main control unit 11
Is waiting for the button to be pressed. And the main control unit 11
Indicates that the power supply unit 3
From 6, the power supply to each unit is started, and the power supply is turned on. The portable information terminal device of this embodiment has a telephone mode and a videophone mode as main operation modes. Immediately after shifting to the power-on state as described above, the main control unit 11 enters a standby state. In the standby state, the main control unit 11 controls the text LCD control circuit unit 15 so that the text LCD 16 displays a main menu screen for selecting one of the telephone mode and the videophone mode. Further, on the main menu screen, a cursor is displayed so as to overlap the operation mode name that is currently a candidate for selection (a predetermined operation mode name in an initial state). And with such a main menu screen displayed,
It waits for a selection operation (an instruction to change a selection candidate and an instruction to determine) in any mode. The main menu screen may be larger in size than the text LCD 16, and the display area is changed to a scroll dial 3 described later.
It may be changed in accordance with the operation 1. When the scroll dial 31 is operated in this state, the main control unit 11 determines that an instruction to change a selection candidate has been issued. At this time, the main control unit 11 controls the scroll dial 31
The operation input control circuit 29
And changes the selection candidate in an order corresponding to the rotation direction for each predetermined amount of rotation, and moves the cursor so as to always display the cursor over the mode name of the selection candidate.

When the operation button 32 is double-clicked, the main control unit 11 determines that a determination instruction has been issued. Then, at this time, the main control unit 11 shifts to the processing routine of the operation mode that is currently the selection candidate. Note that the main control unit 11 can also issue a change instruction and a determination instruction of a selection candidate based on an input state on the touch panel 30. Hereinafter, the operation in each operation mode will be described by the main control unit 11.
Will be described one by one according to the processing procedure. [Telephone Mode] In this operation mode, it is possible to make a voice call as a PHS terminal. In this operation mode, the main controller 11 controls the demultiplexer 17 and the audio codec 23
Is set to voice call mode. Then, the main controller 11 controls the text LCD 16 to display a predetermined telephone number designation method menu screen on the text LCD 16.
The CD control circuit 15 is controlled. The phone number designation method includes “number input”, “kana search”, “alphabet search”, and “number search”. "Enter number" displays a list of numbers on the text LCD 16 and the scroll dial 31.
Is operated and selected, and the telephone number is input by clicking the operation button 32. “Kana search”, “English search”, and “Numeric search” display an input screen showing a list of Kana characters, English characters, and numbers on the LCD 16 for text, and click the operation of the scroll dial 31 and the operation button 32. The desired destination name. Then, based on the telephone directory information (a plurality of destination information in which a telephone number and a name are associated with each other are stored) stored in a RAM (not shown) in the main control unit 11, the input character string is The candidate destination information is displayed, and desired destination information is selected from the displayed destination information.

When the outgoing telephone number is determined as described above, the main control section 11 controls the PHS line IF section 18 so as to perform outgoing call processing using the outgoing telephone number. If a communication path with the destination terminal (the partner information terminal device) is formed by the PHS network in response to this transmission process,
The user can make a call using the earphone microphone 3. That is, the encoded voice data (ADPCM system) transmitted from the partner information terminal device via the PHS network is received by the antenna 19 and the PHS line IF unit 18, and is received via the demultiplexing unit 17 and the synchronization bus 38. This is provided to the audio codec 23. Then, the encoded audio data is decoded by the audio codec 23 into an analog signal, converted to an analog signal, and reproduced as an audio signal. Then, when given to the earphone microphone 3, it is output from the earphone microphone 3 as a received voice.
On the other hand, the transmitted voice transmitted by the user is converted into a voice signal by the earphone microphone 3 and provided to the voice codec 23. The audio signal is digitized and ADPCM-encoded by the audio codec 23 to be coded data, and then supplied to the PHS line IF unit 18 via the synchronization bus 38 and the multiplex / demultiplex unit 17, and the PHS line I
The information is transmitted by the F unit 18 to the partner information terminal device via the antenna 19 and the PHS network.

[Video Phone Mode] In this operation mode, video transmission and reception can be performed while a voice call is being made. In this operation mode, the main control unit 11 sets the operation mode of the demultiplexing unit 17 and the audio codec 23 to the voice communication mode in an initial state. Then, the main control unit 11 performs the designated reception of the outgoing telephone number and the outgoing call processing in the same manner as in the above-described telephone mode. Then, if a communication path with the destination terminal (the partner information terminal device) is formed by the PHS network, the main control unit 11 performs a predetermined procedure (for example,
ITU-T Recommendation H. 245), a negotiation is performed with the partner information terminal, and the partner information terminal
It is confirmed whether audio multiplex communication can be performed and whether implementation of video / audio multiplex communication is permitted. If the information terminal of the other party cannot perform the video / audio multiplex communication or refuses to perform the video / audio multiplex communication, the main control unit 11 shifts to the telephone mode thereafter, and Only calls can be made. On the other hand, when the partner information terminal device can perform the video / audio multiplex communication and permits the execution of the video / audio multiplex communication, the main control unit 11 The operation mode is switched to the multimedia communication mode.

As described above, the encoded audio data obtained by encoding the audio signal generated by the earphone microphone 3 by the audio codec 23 using the low-rate audio encoding method, and the video signal generated by the camera unit 4 Alternatively, the encoded video data obtained by encoding the video data sent from the region extracting unit 21 by the video encoder 27 and other data output from the main control unit 11 are separated by the demultiplexing unit 17.
Are multiplexed. Then, the transmission data thus obtained is sent to the partner information terminal device. In addition, coded audio data, coded video data, and other data are separated by the demultiplexing unit 17 from the transmission data transmitted from the partner information terminal device. The encoded audio data is returned to the audio signal by the audio codec 23 and output from the earphone microphone 3 as audio. The coded video data is returned to video data by the video decoder 12, and video is displayed on the video LCD 14 under the control of the video LCD control circuit 13 based on the video data.
Thereby, while making a call using the earphone microphone 3, the video taken by the camera unit 4 is transmitted to the other party,
Also, the image sent from the other party can be displayed on the image LCD 14 and viewed.

In the videophone mode, the main control unit 1
1 is a state in which the detection signal from the camera orientation sensor unit 28 is checked, and the camera unit 4 is mounted on the terminal main body 1 and is shooting in the same direction as the side on which the display surfaces of the video LCD 14 and the text LCD 16 are provided. If so, the following processing is performed. The camera unit 4 is in a state where the subject of the user holding the terminal body 1 is being photographed, and the subject of the other party is displayed on the video LCD 14. The camera unit 4 is shown in FIG.
2, the image of the subject is cut out as shown in FIG. 4 by the area extraction unit 21 through the camera IF unit 25, and the encoded image obtained by encoding by the image encoder 27 is demultiplexed. The data is multiplexed by the unit 17 and sent to the information terminal device on the other side. Various methods of extracting an image of a subject have been conventionally proposed. For example, a face shape can be cut out by the method disclosed in Japanese Patent Application Laid-Open No. 6-30318. This method extracts a skin color region from a hue signal and a hair region from a luminance signal for an image displayed on a screen, and integrates the two to obtain a face region suitable for a human face model. In addition, since the above-mentioned method can extract only a face image, it is possible to extract even when the subject's head has no hair, that is, a bald head. Further, the present invention can be applied to a black face, white hair, and the like by expanding the extraction range of hue information and luminance information as target ranges to be cut out.

Here, conventionally, there is a method of extracting a subject portion by identifying a moving portion from a still background video and a shaking subject video. However, since the user holds the portable information terminal device with his / her hand, when the camera shake occurs, both the subject video and the back video fluctuate, so it is difficult to distinguish the video of the subject portion. . However, the above method has a feature that a face region can be extracted even if camera shake occurs because a human face model which is not affected by a change in coordinates on a screen is used without using a face motion vector. Next, an embodiment of extracting a region of a subject by improving a conventional method of extracting a motion vector of a subject will be described with reference to FIG. FIG. 3 shows a azimuth sensor 41, a conversion unit 42 instead of the region extraction unit 21 in FIG.
And a person recognition unit 43. The direction sensor 41 is
It is a sensor that detects a rotational angular velocity when the terminal body 1 moves in the vertical and horizontal directions and outputs an electric signal including information on the detected rotational angular velocity. Direction sensor 41
For example, a piezoelectric vibrating gyroscope is applied. In that case,
The azimuth sensor 41 includes, for example, an oscillation circuit that generates a pulse signal for vibrating the piezoelectric vibrating gyroscope, and a drive circuit such as an amplifier circuit that amplifies a signal indicating a transition state of the vibration. The direction sensor 41 outputs the converted electric signal to the conversion unit 42.

The conversion section 42 is a circuit for converting the electric signal output from the direction sensor 41 into a shake signal corresponding to the shake state of the terminal body 1. Specifically, the conversion unit 42 receives an electric signal supplied from the direction sensor 41, and, based on the inputted electric signal, moving amount data representing an angular velocity when the terminal main body 1 rotates, and moving amount data in the vertical and horizontal directions. A blur signal including direction data representing each moving direction is generated. On the other hand, the subject image captured by the camera unit 4 is input to the person recognizing unit 43 via the camera IF unit 25. The blur component is removed by moving the subject image in a direction that cancels the vector with respect to the vector generated by the blur direction detected by the conversion unit 42 and the movement amount in that direction.
Then, the subject image from which the blur component has been deleted is specified and cut out by the person recognizing unit 43 to specify the subject area, and is input to the video encoder 27 and encoded. As a method of extracting a region of a subject, for example, it can be realized by the method disclosed in Japanese Patent Application Laid-Open No. 7-73298. This method detects a motion of a subject and determines whether the detected motion region is present in a preset face area, and determines whether the detected motion region has a facial feature by detecting an image edge in the area. It is characterized by extracting a region.

Further, since the present method is a method for extracting a motion vector of a subject, there is no limitation on a color application range when extracting a face color as in the above-described region extraction method using a human face model. Can extract not only the face region but also the upper body of the subject. After the face region is extracted, the face image corresponding to the region is cut out from the entire image, and the face image 10 from which the background image is deleted as shown in FIG.
1 is obtained. Here, in FIG. 4, the face 102 is surrounded by a rectangle 102 that is in contact with the outline of the face video 101, and the center 103 of the rectangle is the face portion, that is, the center of the face video 101. The face image of the other party sent from the other party and displayed on the video LCD 14 may be a video including a background as shown in FIG. 83, or may be the same as the portable information terminal device as shown in FIG. The information terminal device on the side may extract the face video 105 of the other party, send only the face video 105 of the other party, and display it. Here, as a result of photographing by the camera unit 4 and extracting a face image by the area extracting unit 21, the face image 106 to be sent to the other side is shifted from the center of the display frame 111 of the screen due to camera shake as shown in FIG. It may come to the edge. Therefore, the main control unit 11
The region extracting unit 21 detects the direction and the moving distance of the center 104 of the face image 106 from the center 103 of the rectangle located at the center of the display frame 111, and displays the face image 106 using the detection result. It is controlled to move to the center of the frame 111, that is, to the position of the face image 101. The face image 106 returned to the center of the display frame 111 is encoded by the image encoder 27, and the encoded image data is sent to the partner information terminal device and displayed on the partner information terminal device.

The camera unit 4 captures an image of a subject, the area extraction unit 21 extracts a face image 106, moves the face image 106 to its original position, and transmits the image to the partner information terminal device. Since it is a series, it is necessary to complete the cycle of shooting an image by the camera unit 4, that is, the time within one frame. When the face image 106 is sent to the other party's terminal, the face image 106 is displayed over a single color background instead of the background scenery. Alternatively, a pattern such as wallpaper may be displayed as a background, and the face image 106 may be displayed over the pattern. As described above, conventionally, when a camera shake is corrected for a raw image as shown in FIG. 84, a portion 123 where an image is missing due to a lack of a background edge occurs as shown in FIG. 87. However, according to this embodiment, the background image is not sent to the other party, only the face image is targeted, and even if camera shake occurs on the sending side, the face image is moved to the center of the screen and then sent to the other party. , The other party can see a stable image. Since the image to be sent to the other party is only a face image, there is an advantage that the amount of data transmission is smaller than in the method of sending an image including a back. Here, in order to send the face video to the other party, the face video may be decomposed into 16-bit and 16-bit blocks, for example, and sent to the other party in block units.
At this time, each block is added with the address where the block is located in the display frame 111 of the screen and sent, and the other party displays the contents of the block at the specified address, so that the entire face is displayed. Display video. As described above, when the face area is extracted, since the already extracted area is divided into blocks, these blocks may be transmitted to the other party in order.

In this embodiment, the communication line between the portable information terminal device on the user side and the information terminal device on the partner side is partially connected by a wireless line. In this embodiment, the embodiment of the communication line is a wired line. However, it can be realized without distinction even by wireless. (Embodiment 2) Next, Embodiment 2 of the portable information terminal device according to the present invention will be described. FIG. 7 is a block diagram illustrating a configuration of the portable information terminal device according to the second embodiment. The configuration shown in FIG. 7 is different from the configuration of the first embodiment shown in FIG.
6. Zooming button 47 and zooming drive unit 48
Is different. The image memory 45 stores the camera I
The video data given from the F unit 25 is frame-by-frame
It is a memory for storing. The shape extraction unit 20 stores the image memory 4
The face shape is extracted in frame units from the video data given from 5. The face video memory unit 46 is a memory that stores the face video generated by the area extraction unit 51 including the shape extraction unit 20 and the video encoder 27 at a predetermined cycle. In the second embodiment, the camera IF unit 25 takes in a video signal output from the camera unit 4 connected via the camera terminal 26, digitizes the video signal to obtain video data, and converts the video data into frame units. To the image memory 45. The video data stored in the image memory 45 is provided to the video encoder 27 and the video LCD control circuit unit 13.

On the other hand, the luminance signal and the hue signal 50 are given from the image memory 45 to the shape extracting section 20. The shape extraction unit 20 extracts the shape of the face portion from the human face model using the input luminance signal and hue signal 50. The video encoder 27 receives the digital video 5 supplied from the image memory 45.
4 and the face shape 112 given from the shape extraction unit 20 to obtain object-encoded video data according to the MPEG4 system. The video encoder 27 supplies the coded video data to the video decoder 12 and the demultiplexer 17. The face image generated by the area extracting unit 51 is stored in the face image memory unit 46 while being updated at a predetermined cycle. In the second embodiment, a zooming button 47, a touch panel 30, a scroll dial 31, an operation button 32, and a power button 34 are connected to the operation input control circuit 29, respectively. The operation input control circuit unit 29 receives a user's instruction operation on the zooming button 47, the touch panel 30, the scroll dial 31, the operation button 32, and the power button 34, and notifies the main control unit 11 of the content of the instruction operation. The zooming button 47 receives a user's instruction operation, and controls a zooming drive unit 48 to zoom the lens portion of the camera unit 4.

FIG. 8 shows the appearance of the terminal body 1 and the camera unit 4 in the second embodiment. 8 differs from that shown in FIG. 2 in that a zooming button 47 is provided. In the second embodiment as well, as in the first embodiment, there are two operation modes, a telephone mode and a videophone mode. 28, the camera unit 4 is attached to the terminal body 1, and the image LCD 14 and the text L
If the same direction as the side on which the display surface of the CD 16 is provided is photographed, the following processing is performed. The camera section 4 is in a state where the face of the user holding the terminal body 1 is being photographed, and the face of the other party is displayed on the video LCD 14.
The camera unit 4 captures an image as shown in FIG. 82 and accumulates the image in the image memory 45, and then the shape extraction unit 20 extracts a face shape 112 as shown in FIG. Using the face shape 112 and the digital video 54, the video encoder 27 generates MPEG4 object encoded data, that is, a face video 121 as shown in FIG. Here, various methods for extracting a face shape have been conventionally proposed. For example, a face shape can be cut out by the method described in Japanese Patent Laid-Open No. 6-30318 as described in the first embodiment.

In the second embodiment, as in the first embodiment, the area of the subject can be extracted by improving the conventional method of extracting the motion vector of the subject. FIG. 11 shows the configuration in this case. That is, FIG. 11 shows the azimuth sensor 4 instead of the shape extraction unit 20 of FIG.
1, a converting unit 42 and a person recognizing unit 43 are provided, but the operation of extracting a subject area is the same as that of the first embodiment. Here, in FIG. 9, the image is surrounded by a rectangle 113 that is in contact with the contour of the face shape 112, and the center of the rectangle is the center 114 of the face image 112. Also, sent from the other party,
The face image of the other party displayed on the image LCD 14 is shown in FIG.
3, the face video 105 may be extracted by the information terminal of the other party, and only the face video 105 of the other party may be sent and displayed as shown in FIG. Here, as a result of the photographing by the camera unit 4 and the extraction of the face image by the area extracting unit 51, the face image to be sent to the other party is displayed on the screen by the camera shake or the shaking of the user's body as shown in FIG. In some cases, the image may be shifted from the center of the image 111 to an end, so that the image looks like a face image 122. Therefore, the main control unit 11 causes the area extracting unit 51 to move the face image 122 to the center of the display frame 111, that is, to the position of the face image 121. That is, since the center of the face image 122 is 117 and the center of the face image 121 is 114, the face image 122 is shifted from the center 117 to the center 1
That is, it has moved in the direction of the vector drawn by.
The face image 121 is sent to the partner terminal and displayed on the partner terminal.

As described above, if a raw image as shown in FIG. 84 is captured only in the range of the display frame 111, if the face image is not located at the center of the screen, the face image is displayed. 87, the edge of the background always drops off as shown in FIG. However, according to this method, even when the shooting range is equal to the display frame 111, the face image 122 is in the range 122-1 that contacts the inside of the display frame 111 as shown in FIG.
The face image 122 will not be lost even if the movement is performed at -122-6. That is, inside the display frame 111, the face image 122
Can be freely moved. In other words, it can be said that the captured image is more resistant to fluctuation than the conventional method of adding a background image. Here, this method has an advantage that the amount of data transmission is smaller than the method of transmitting the image including the background, since the image to be transmitted to the other party is only the face image. The transmission rate allocated to the background video can be transferred to the transmission of the face video, and the video encoder 27 can encode the face video with a finer video. Further, if the transmission rate allocated as the background is allocated to increasing the transmission rate of the audio codec 23, it is possible to encode with higher quality audio. In addition, since there is no need to send the background to the other party, if the transmission capacity has room, as shown in FIG.
2 may be sent to the other party and displayed on the other party's information terminal device. In this case, after the face image 122 is generated by the area extracting unit 51, the face image 122 is moved to a predetermined position (the right side in FIG. 14) instead of the center of the screen, and the text text 130
May be displayed.

Since the vacant area has no background, the text text 130 is clearly displayed without being affected by the background image. Although FIG. 14 displays the text 130, any graphic, icon, or advertisement of an infrastructure company operating this system may be displayed. In addition, when the user uses the portable information terminal device in a place related to privacy, he or she may not want to send a background image to the other party. In the method of this embodiment, since the background image is deleted, it is not necessary to worry about where the portable information terminal device is used, and it can be used anywhere. Also, as in the conventional method, if the face image and the background are sent to the other party together in places with heavy traffic, even if the other party tries to check the face image, the face image will be difficult to see because of the strong background movement. Sometimes. However, in this method, since the background is not sent to the other party, only the face image can be clearly checked. When the zoom button 47 is pressed, a face image 121 is displayed on the image LCD 14 as shown in FIG. Then, when the zooming button 47 is further pressed, the zooming drive unit 23 operates to zoom the lens 70 of the camera unit 4 and the face image 121 is reduced as shown in FIG. In addition, the image can be enlarged as shown in FIG. 10C, and the face image 121 can be set to a size desired by the user. Then, when the zooming operation is completed, the set face image 121 is sent to the other party and the image LC 121
The face image 105 of the other party is displayed on D14. Further, while no communication with the other party is being performed, the face LCD 121 of the user may be constantly displayed on the video LCD 14 so that the zooming operation can be performed as described above.

Embodiment 3 Next, Embodiment 3 of the portable information terminal device according to the present invention will be described. FIG.
FIG. 13 is a block diagram illustrating a configuration of a portable information terminal device according to a third embodiment. The configuration shown in FIG. 15 is different from the configuration of the second embodiment shown in FIG.
0 and a face image memory unit 46, a trimming shape storage unit 55 and a trimming image memory unit 56 are provided.
57 and a pupil extraction unit 58 are provided, and the infrared camera 5 is connected via the infrared camera terminal 22. The trimming shape storage unit 55 stores a plurality of trimming shapes. A predetermined shape is selected from the trimming shapes, read out, and applied to the video encoder 27. The trimming video memory unit 56 is a memory for storing the trimming video generated by the area extracting unit 51 at a predetermined cycle. In the third embodiment, the camera IF unit 25 takes in a video signal output from the camera unit 4 connected via the camera terminal 26, digitizes the video signal to obtain video data, and converts the video data into frame units. To the image memory 45. Digital image 54 stored in image memory 45
Is supplied to the video encoder 27 and the video LCD control circuit 13. The video encoder 27 uses the digital video 54 supplied from the image memory 45 and the trimming shape 200 supplied from the trimming shape storage unit 55 to generate an MP
Obtain object encoded video data according to the EG4 method.

The trimming shape storage unit 55 and the video encoder 27 are referred to as an area extraction unit 51. The video encoder 27 supplies the coded video data to the video decoder 12 and the demultiplexer 17. The trimmed video generated by the area extracting unit 51 is stored in the trimmed video memory unit 56 while being updated at a predetermined cycle. The infrared LED 57 is an LED for irradiating infrared rays to the user's face. The infrared camera 5 is a camera for inputting infrared light reflected from the user's face. The pupil extraction unit 58 includes the infrared camera 5
The pupil portion is extracted from the user's face photographed in step (1). In the third embodiment, the zooming drive unit 48 is a mechanism for simultaneously zooming the camera unit 4 and the infrared camera 5 under the control of the main control unit 11. Further, the zooming button 47 is for manually receiving a zooming instruction for the camera unit 4 and the infrared camera 5 at the same time. Terminal body 1 and camera unit 4 in the third embodiment
FIG. 16 shows the appearance of the infrared camera 5. This FIG.
The one shown in FIG. 8 is the same as that shown in FIG.
57 and an infrared camera 5 are provided.
As shown in FIG. 16, the infrared camera 5 is mounted on the same side of the housing as the camera unit 4, on which the operation buttons 32, the power button 34, and the zooming button 47 are provided. The infrared LED 57 is attached to the upper part of the video LCD 14 on the front of the housing.

Next, this operation will be described. The camera unit 4 is in a state in which the face of the person holding the terminal body 1 is being photographed, and the face of the other party is displayed on the video LCD 14. The camera unit 4 captures an image including the user's face and background in the display frame 111 as shown in FIG.
The image memory 45 outputs a digital video 54. In FIG. 17A, the face of the user is shifted to the lower left from the center. Here, feature points are extracted from the face of the user. In the third embodiment, a user's eyeball portion is extracted as a feature point (details will be described later). Then, the trimming shape 200 is read from the trimming shape storage unit 55. The trimming shape 200 is arranged on the digital image 54 as shown in FIG. 17B so that the face portion does not protrude using the feature point as a clue. Then, the video encoder 27 uses the trimming shape 200 and the digital video 54,
Object encoded data by PEG4, that is, FIG.
8 (a) is generated. And
Further, the video encoder 27 moves the trimmed video 201 to the center of the display frame 111 as shown in FIG. 18B, and transmits only the trimmed video 203 moved to the center to the other party.

When the trimming video 203 is sent to the other party's terminal, the trimming video 203 is displayed over the single color background instead of the background scene. Alternatively, a pattern such as wallpaper may be displayed as a background, and the trimmed image 203 may be displayed over the pattern. Alternatively, the background image separately stored in the portable information terminal device may be sent to the other party, and the face image and the background image sent by the other party may be displayed in a superimposed manner. As the trimming shape 200 stored in the trimming shape storage unit 55, a plurality of shapes such as a rectangle as shown in FIG. 19A and a rhombus as shown in FIG. 19B are stored. Then, a plurality of trimming shapes 200 are displayed on the video LCD 14 in advance, and the touch panel 3 is displayed.
0 operation, the desired trimming shape 20
What is necessary is just to select 0. FIG. 20 is a diagram for explaining a method of detecting the position of the user's eyeball and determining the trimming shape 200 from the detected position. An infrared ray is emitted from the infrared LED 57 to the user's face. The infrared rays are reflected from the user's face and are taken in by the infrared camera 5. Since the infrared ray absorbs visible light and reflects the infrared ray in the retina at the fundus of the user, only the pupil portions 204 and 205 of the video signal 210 of the infrared camera 5 are imaged with very high brightness.

The luminance of the pupil portions 204 and 205 is so large as to be incomparable with the amount of reflected light from the skin portion of the surrounding face.
205 can be binarized and cut out. In FIG. 20, a space between g and h, that is, a is a distance between both eyes of the user. The middle 209 between g and h corresponds to the position of the nose. The trimming width is set to b right and left from the middle 209 by b. b is a statistically determined average value for a human face with a margin. Also, the position 208 of the height of both eyes
From above, a statistical value d (value beyond the vertex) and a downward value c (value beyond the chin) are set, and the trimming shape 20 is set.
0 is determined. The trimmed shape 200 will include the user's face along with the surrounding background. Then, the trimming shape 200 (including the internal image)
By moving the center of gravity 206 to the center 207 of the display frame 111, the trimmed image 203 has moved to the center of the display frame 111. FIG. 21 is a diagram for explaining a method of detecting the position of the eyeball when two persons are arranged. As described above, the position of the eyeball is determined by binarizing the infrared light reflected from the eyeball with a threshold value. g, h, i, and j
Is the position of the eyeball. Then, a statistical set value a is taken leftward from the position g of the leftmost eyeball. Similarly, a statistical set value a is taken rightward from the rightmost eyeball position j. The trimming shape 200 is determined by taking c upward from the upper eyeballs i and j and e downward from the lower eyeballs g and h.

The trimming shape 200 includes the face portions of two persons. Then, by moving the center of gravity 206 of the trimmed shape 200 to the center 207 of the display frame 111, the trimmed image 203 is displayed on the display frame 11
This means that it has moved to the center of 1. Here, even with the above method, the user may be able to detect only one eye position by turning sideways or blinking. Further, the positions of both eyes may not be detected. Therefore, regarding the detection of the position of the eyeball, it is necessary to delete the detection result of the eyeball position when the detection position of only one eye or the detection position suddenly deviates extremely during the passage of time. . For example, when the camera is turned halfway, the value of “a” in FIG. 20 is suddenly reduced. If the eyes are completely turned, both eyes are not detected. In the case of blinking, one or both eyes are not detected instantaneously. In the method described with reference to FIG. 20, the trimming shape 200 is uniquely determined by detecting the distance a between both eyes.
Therefore, it is necessary to read out the trimming shape 200 from the trimming shape storage unit 55 and enlarge or reduce the trimming shape 200 in proportion to the read trimming shape 200 from the distance a between the eyes. That is, the trimming shape storage unit 55
The original trimming shape to be trimmed is stored in the trimming shape 200 which is to be trimmed from now on.
The size may be changed in proportion to 0.

As described above, the size of the user's face has been estimated by detecting the distance between both eyes in order to determine the size of the trimming shape 200. However, the trimming shape 200 may be determined by a conventional method of extracting a face region and obtaining a face outline. According to FIG. 22, the face shape 112 of the user may be determined by a conventional method, and a trimming shape 200 including the face shape 112 may be set. in this case,
Center of gravity of face shape 112 and center of gravity 20 of trimmed shape 200
6 should be matched. The present method has an advantage in that a rough face shape is obtained without trimming the face shape accurately, and the trimming shape 200 that embraces the face shape is overlapped. Here, as a method of detecting the face shape, for example, as described in the first and second embodiments,
The face shape can be cut out by the method of 30318. This method extracts a skin color region from a hue signal and a hair region from a luminance signal for an image displayed on a screen, and integrates the two to obtain a face region suitable for a human face model. Conventionally, there has been a method of extracting a face portion by identifying a moving portion from a still background image and a shaking face image. However, since the portable information terminal device is held by hand, and when a camera shake occurs, both the face image and the background image fluctuate, it is difficult to distinguish the image of the face portion by the conventional method. there were. However, this method does not use a face motion vector and uses a human face model that is not affected by changes in coordinates on the screen. Is extracted.

In FIG. 20, the operation using a rectangle as the trimming shape 200 has been described. However, the trimming shape 200 may be an ellipse as shown in FIG. In short, any shape is acceptable as long as the user's face is hugged. In the case of the ellipse shown in FIG. 23, the length d + c in the vertical direction determined in FIG. 20 may be set to the long diameter, and the length 2b in the horizontal direction may be set to the short diameter. Also, when displaying the trimmed video 203 on the information terminal device of the other party by blurring the outline of the trimmed video 203 with a color similar to the display color of the background, the other party can display the trimmed video 203.
Less fatigue even if you watch for a long time. As described above, in the method according to the third embodiment, the distance between the user's eyes is measured, and the size of the user's face is estimated from the measured value. However, as described above, when the user is having a conversation with the other party, if the user's face turns sideways, the distance between the eyes changes. Further, even if the user moves the portable information terminal device closer to or away from the face, the distance between the eyes changes, and it is not possible to distinguish which operation has changed the distance. Therefore, the pupil extraction unit 5
From the positions of both eyes detected in step 8, the center of the position is set as the center of gravity of the user's face, and the center of gravity is used as reference coordinates for moving the trimmed image 201 on the display screen.
Then, the distance between the user and the mobile information terminal device, which changes as the mobile information terminal device moves away or approaches, is measured by another method, and the trimming image 20 is determined in accordance with the distance.
By changing the size of 3, the method becomes a stable method that is more resistant to camera shake.

FIG. 24 is a configuration diagram showing only the necessary components extracted from FIG. 15 with respect to the above-mentioned one including the function of measuring the distance between the portable information terminal device and the user. That is, in the configuration shown in FIG. 24, an ultrasonic transmission unit 62, an ultrasonic reception unit 63, and a trimming shape variable button 64 are added to the components shown in FIG. First, before communicating with the other party, the user's face is photographed by the camera unit 4, the region is extracted by the region extracting unit 51, and the trimmed video 203 is displayed on the video LCD 14. Then, the user operates the zooming button 47 to set the user's face inside the trimming shape 200 to a desired size. At the same time, the trimming shape 20 read from the trimming shape storage unit 55 by the trimming shape variable button 64
The value of 0 is changed in proportion to the desired size. At this time, depending on the size of the trimming shape 200, the user's face is embraced, and the area of the surrounding background is increased or decreased. At that time, the size of the trimming shape 200 is adjusted according to the communication partner. Can be changed.
Next, while the zooming button 47 and the trimming shape variable button 64 are being operated as described above, the ultrasonic wave is emitted from the ultrasonic wave transmitting unit 62 to the user's face, and the reflected ultrasonic wave is transmitted to the ultrasonic wave receiving unit 63. Receive. FIG. 25 illustrates an ultrasonic wave received by the ultrasonic wave receiving unit 63. a is the earliest reflected wave, which is the reflected wave from the user's face closest to the portable information terminal device. Therefore, the distance from the portable information terminal device to the user can be determined from the time t1 from transmission to reception.

The ultrasonic waves below b and c are ultrasonic waves reflected by the background and are received later than a. Hereinafter, the distance to the user's face is constantly measured by the above-described method, and by driving the zooming drive unit 48 according to the distance, the size of the user's face captured by the camera unit 4 is adjusted within the trimming shape 200. , So as to be equal. That is, the trimmed video 203 displayed on the video LCD 14 is maintained at a constant size even if the distance between the user's face and the portable information terminal device changes. Also, by changing the trimming shape 200 and the internal face image simultaneously in proportion to each other, the trimming image 203 can be reduced when the portable information terminal device approaches the user's face, and can be increased when the mobile information terminal device moves away from the user. The trimmed image 203 can be distinguished satisfactorily regardless of the distance. In the above description, the zooming is performed by controlling the zooming drive unit 48. However, an operation equivalent to zooming is performed by enlarging or reducing digital data captured by the camera unit 4 and stored in the image memory 45. May be realized. Note that the trimming shape 200 may be changed depending on the communication partner. For example, when starting communication, a trimming image 203 selected from the trimming shape storage unit 55 according to the telephone number of the other party and cut out by the trimming shape 200 may be sent to the other party.
If the other party is close, the trimming shape 200 may be slightly larger to show the background to the other party. However, I don't want to know the privacy of occasional callers. Therefore, it is possible to make the trimming shape 200 just before the user's face part so that the background part is not shown to the other party as much as possible. In some cases, the trimmed shape 200 may be smaller than the user's face area to completely hide the background.

As described above, if the image of the raw image as shown in FIG. 84 is captured only in the range of the display frame 111, if the face image is not located at the center of the screen, the face image is displayed. 87, the edge of the background always drops off as shown in FIG. However, according to the method of the third embodiment, even when the photographing range is equal to the display frame 111, FIG.
As shown in FIG. 6, even if the trimming shape 200 moves in a range 200-1 to 200-6 that is in contact with the inside of the display frame 111, the trimming image 201 does not drop out. That is, it can be said that the trimmed image 201 can be freely moved inside the display frame 111. In other words, it can be said that the captured image is more resistant to fluctuation than the conventional method of adding a background image. Here, in this method, the video to be sent to the other party is a trimming video 20.
Since only 3 is used, there is an advantage that the amount of data transmission is smaller than in the method of transmitting the video including the background. The transmission rate allocated to the background video can be transferred to the transmission of the trimmed video 203.
03 can be encoded with finer video.
Further, if the transmission rate allocated as the background is allocated to increasing the transmission rate of the audio codec 23, it is possible to encode with higher quality audio.

If the transmission capacity has room because the background need not be sent to the other party, as shown in FIG.
3 and may be sent to the other party and displayed on the information terminal device of the other party. In this case, after the trimming video 203 is generated by the area extracting unit 51, the trimming video 203 is not at the center of the screen but at a predetermined position (FIG. 2).
7, the text 130 may be displayed in an empty area on the left. The empty area has no background and is not affected by the background image.
30 is clearly displayed. In FIG. 27, the text 130 is displayed. However, an arbitrary figure, an icon, or an advertisement of an infrastructure company that operates the system may be displayed. (Embodiment 4) Next, Embodiment 4 will be described. According to the above-described first to third embodiments, only a face image is cut out from an image in which a camera shake or a user's body is shaken, and sent to the other party. be able to. However, in the first embodiment or the second embodiment, on the user side, the image captured by the camera unit 4 fluctuates greatly, and the user's face is
29, or may exceed the range of the display frame 111 as shown in FIG.

In the case of FIG. 28, the face image 122 is moved to the center and sent to the other party as the face image 121, so that no problem occurs. However, in the case of FIG. 29, when the face image 122 is cut out, a part of the face image 122 is missing or the image captured by the camera unit 4 fluctuates greatly, and the display frame 11 is completely removed.
Sometimes it disappears from one. Then, the missing face image 122 moves to the center of the display frame 111 as it is and is sent to the partner terminal as the face image 121. However, the user does not know that his / her own face that has been photographed protrudes from the edge of the screen because he / she is watching the other party's face image on the image LCD 14. Therefore, in this embodiment, when the cut out face image 122 touches the edge of the display frame 111 as shown in FIG. 28 or protrudes from the edge as shown in FIG. 29, the other person's face image 105 (for example, Instead of FIG. 5), a raw video image as shown in FIG. 30 or FIG. 31 captured by the camera unit 4 at that time is displayed. By directly looking at the screen on which the user's face is displayed, the user knows that the shooting direction of the camera unit 4 is out of order, and can move the hand to immediately change the direction of the camera unit 4. When the user's face image 122 moves away from the edge of the screen again, the other person's face image 105 is displayed on the image LCD 14 again.
(For example, FIG. 5) is displayed.

Here, in order to check the state where the face image 122 is in contact with the display frame 111, as shown in FIG.
One of the four sides 18 may be checked by contacting one of the sides with the display frame 111. In addition, in order to check the state in which the face image 122 has exceeded the display frame 111, the face image 122 is displayed in a state where two sides (two horizontal sides, that is, the length a in FIG. 29) of the four sides of the rectangle 118 are shortened. What is necessary is just to exceed the display frame 111. Further, the state in which the rectangular area a × b is reduced may be a state in which the face image 122 exceeds the display frame 111. Here, when the face image 105 (for example, FIG. 5) of the other party is switched to a live image of the user (for example, FIG. 30 or FIG. 31), when the color of the user image is changed or displayed faintly, Video or a user's video and it is easy to understand. Further, as shown in FIG. 28 or FIG. 29, when the face image 122 touches or protrudes from the edge of the display frame 111, instead of displaying the user's raw image on the image LCD 14 immediately, Since the face image 122 may momentarily protrude from the edge of the display frame 111 due to shaking of the body, the screen may be switched when the face video 122 protrudes from the edge of the display frame 111 for a predetermined time, for example, about one second. Also, in consideration of the fact that the face image 122 protrudes from the edge of the display frame 111 a plurality of times due to camera shake or shaking of the user's body, the face image 122 is displayed on the display frame 111.
If it protrudes from the end a predetermined number of times, for example, three times, this may be detected and the display of the screen may be switched.

Also, in the third embodiment, similarly, on the user side, the image taken by the camera unit 4 fluctuates greatly, and the trimmed image 217 is displayed on the display frame 11 as shown in FIG.
Also, when the user touches the display frame 111 or exceeds the range of the display frame 111, the trimming image 217 is cut out and the trimmed image 217 is cut out.
, The missing portion 219 may occur in the image inside the trimmed image 218. Also in this case, the trimmed video 218 that is partially missing is sent to the partner terminal as it is. However, the user does not know that his / her own face that has been photographed protrudes from the edge of the screen because he / she is watching the other party's face image on the image LCD 14. Therefore, when the trimmed image 217 cut out as shown in FIG. 32 touches or protrudes from the edge of the display frame 111, the face image 10 10
Instead of 5 (for example, FIG. 5), a live image as shown in FIG. 30 or 31 captured by the camera unit 4 at that time is displayed. By directly looking at the screen on which the user's face is displayed, the user knows that the shooting direction of the camera unit 4 is out of order, and can move the hand to immediately change the direction of the camera unit 4. Here, the trimming image 217 is displayed on the display frame 111.
In order to check the state of contact with the image, the area of the trimming image 217 is checked.
7 may be beyond the display frame 111.

Here, when switching from the other party's face image 105 to the user's live image, if the color of the user's image is changed or displayed faintly, the other party's image will be displayed.
It is easy to understand because it is distinguished from the user's video. FIG. 32
When the trimmed image 217 protrudes from the edge of the display frame 111 as shown in FIG. 7, the user's image is not displayed on the image LCD 14 immediately, but the trimmed image 217 is instantaneously displayed by the camera shake or the shaking of the user's body.
Since the image may protrude from the end of the display frame 111 for a predetermined time, for example, about one second, the screen may be switched. Also, considering that the trimmed image 217 protrudes from the edge of the display frame 111 a plurality of times due to camera shake or shaking of the user's body, the screen is displayed when the trimmed image 217 protrudes from the edge of the display frame 111 a predetermined number of times, for example, three times. May be switched.
Further, in this embodiment, when the display of the face video 105 of the other party is switched to the display of the raw video of the user, the face video 105 of the other party and the raw video of the user may be displayed together in a multi-window. Good. (Fifth Embodiment) Next, a fifth embodiment will be described.

In the fourth embodiment, the cut face image 122 or the trimmed image 217 is displayed on the display frame 11.
LCD for video if it touches the edge of 1 or protrudes from the edge
In place of the face image 105 of the other party, the live image captured by the camera unit 4 at that time is displayed on the display 14. In the fifth embodiment, the face image 122 or the trimming image 217 is displayed. An object is to prevent the unsightly image from being sent to the other party by sending another screen to the other party while the image is outside the frame 111.
That is, in the fifth embodiment, when the cut out face image 121 is transmitted as in the second embodiment, the face image 121 is normally extracted by the region extracting unit 51 and the face image 1 is sent to the other party.
21 and at the same time, the face image 1 is stored in the face image memory 46.
21 are stored. The face image 121 stored in the face image memory unit 46 is updated for each frame or at a predetermined frame cycle. When the face image 121 protrudes from the display frame 111 as in the fourth embodiment, the accumulation of the face image 121 in the face image memory unit 46 is stopped.
Then, as in the fourth embodiment, the video (FIG. 31) of the user is displayed on the video LCD 14.

Face image 1 stored in face image memory section 46
21 is sent to the other party only once as a still image, and is displayed on the display unit of the other party's terminal device. The display of the still image is maintained as it is on the display unit of the partner terminal device. Here, the still image is photographed by the camera unit 4, the shaking becomes severe, and is a normal face image 121 without any missing portion just before protruding from the display frame 111. Then, the user switches the portable information terminal device, and the face of the user fits in the display frame 111, and the face image 1
When the area extraction unit 51 can extract the image normally without any missing parts in the area 21, the transmission of the face image 121 is restarted as in the second embodiment. Also, when the trimming video is transmitted as in the third embodiment, the region extracting unit 51 normally extracts the trimming video 203 and sends the trimming video 203 to the other party.
At the same time as sending No. 3, the trimming video 203 is stored in the trimming video memory unit 56. Then, the trimmed video 203 stored in the trimmed video memory unit 56
Are updated every frame or at a predetermined frame cycle. Then, as in the fourth embodiment, the trimming video 21
When the number 7 protrudes from the display frame 111, the accumulation of the trimmed video 203 in the trimmed video memory unit 56 is stopped. Then, as in the fourth embodiment, the video (FIG. 31) of the user is displayed on the video LCD 14.

The trimmed video 203 stored in the trimmed video memory 56 is sent to the other party only once as a still image, and displayed on the display unit of the other party's terminal. The display of the still image is maintained as it is on the display unit of the other party. here,
The still image is a normal trimmed image 203 which is shot by the camera unit 4, has severe shaking, and has no missing portions just before protruding from the display frame 111. Then, when the user switches the portable information terminal device and the user's face fits in the display frame 111, and the trimming image 203 can be normally extracted by the region extracting unit 51 without any missing portion, as in the third embodiment, The transmission of 203 is restarted. (Embodiment 6) Next, Embodiment 6 will be described. This embodiment is different from the second embodiment or the third embodiment in that the cut-out face video 122 or the trimmed video 217 is displayed on the video LCD 14 and a zooming operation is performed. Is enlarged, and when it protrudes from the display frame 111, a raw video taken by the camera unit 4 is displayed, and the video is sent to the other party. That is, in the second embodiment, when the zooming button 47 is pressed to perform the zooming operation, the face image 121 is displayed on the image LCD 14.
Is displayed, but the face image 12 as shown in FIG.
As the zooming button 47 is pressed down on the image LCD 14 displaying 1, the face image 121 is enlarged as shown in FIG. Then, the face image 121 may protrude from the display frame 111 as shown in FIG. Therefore, in such a case, a raw video taken by the camera unit 4 is displayed on the video LCD 14 as shown in FIG. The feature is that this video is sent to the other party as it is.

That is, when the face image 121 becomes large,
Even if the area of the user's face is extracted by the area extraction unit 51, the background portion is small, so even if a video including the background is sent to the other party, the privacy of the user is not known to the other party. No need to delete. Then, the processing in the area extracting unit 51 is simplified, and the processing for transmitting a normal rectangular image can be completed, which leads to a reduction in power consumption. When the face image 121 protrudes from the display frame 111 as in the fifth embodiment, the face image 121 stored in the face image memory unit 46 (the face image 121 not protruding from the display frame 111) is sent to the other party as a still image. You may send it. Also, the trimming can be performed in the same manner as in the third embodiment. That is, FIG.
As shown in (a), the image inside the display frame 111 is cut out by the trimming shape 202, and the region extraction unit 51
A trimming video 203 is generated as shown in FIG. 3B, but the user looks at the video LCD 14 on which the trimming video 203 is displayed, and
Is pressed, the trimming image 2 as shown in FIG.
03 is expanding. Then, the trimmed video 203 may run off the display frame 111 as shown in FIG. Therefore, in such a case, a raw video taken by the camera unit 4 is displayed on the video LCD 14 as shown in FIG. The feature is that this video is sent to the other party as it is.

That is, when the trimmed image 203 becomes large, even if the area of the user's face is extracted by the area extracting unit 51, the background part is small, and the half-cut parts 215 and 21 are incomplete.
6 is less likely to occur. Therefore, even if the image including the background is transmitted to the other party as shown in FIG. 11E, the privacy of the user is not known to the other party, and there is no need to delete the background. Then, the region extraction unit 51
, The processing can be simplified, and the processing of sending a normal rectangular image can be completed, which leads to a reduction in power consumption. Also, during the period in which the trimmed video 203 protrudes from the display frame 111 as in the fifth embodiment, the trimmed video memory unit 5
6 (the display frame 11)
The trimmed video 203 that does not protrude from 1 may be sent to the other party as a still image. (Embodiment 7) Next, Embodiment 7 will be described. In the seventh embodiment, a region larger than the display frame is imaged as a shooting frame, the image is stored in the image memory 45, and the processing as in the second or third embodiment is performed. . First, a method of capturing a region larger than the display frame as a shooting frame when a face region is cut out and a face image is extracted as in the second embodiment will be described.

FIG. 35 is a configuration diagram in the case of applying the method of taking an image of a region larger than the display frame as a photographing frame to the method of FIG. 86. FIG. 35 shows only necessary components for the movement operation of the face image. The address control unit 72, which was required in the method of FIG. 86, is not required in this embodiment, and has been deleted. 35 differs from the one shown in FIG. 7 in that the image memory 45 has a memory not in the range of the display frame 111 but in the range of the photographing frame.
Then, in the method of this embodiment, as shown in FIG. 37, the image of the face shape 112 moving within the range of the shooting frame 120 may be extracted by the region extracting unit 51. Accordingly, the present method can sometimes extract an image of the face shape 112 deviating from the display frame 111, and is more resistant to shaking of the captured image than when the imaging frame 120 is equal to the display frame 111. FIG. 36 is a view for explaining the movable area of the face shape 112 in the conventional method. That is, in the conventional method, the display frame 111 is moved inside the shooting frame 120 so that the background and the face region are the same, and the face shape 112 is at the center of the display frame 111. The video inside is cut out. Under the condition that the face shape 112 is located at the center of the display frame 111, the movable range of the face shape 112 is drawn in the range of 112-1 to 112-6. That is, if the face shape 112 is moved inside the rectangle 119, the face shape 112 can be moved to the center of the display frame 111.

On the other hand, in this method, since the background is deleted, it is sufficient that the image of the face shape 112 can be cut out from the photographing frame 120. That is, as shown in FIG.
It can be moved to the range of -1 to 112-8. That is, it can be moved inside the shooting frame 120. 36 and 37
When this method is compared with this method, the movable area of the face shape 112 is expanded from the rectangle 119 to the shooting frame 120,
This indicates that the shot image has become more resistant to shaking. Next, when the face shape 112 protrudes from the photographing frame 120 as shown in FIG. 38, the image inside the display frame 111 is displayed as it is on the image LCD 14 (or the other party's face image 105 and the display frame 111 and the image inside
While the face shape 112 protrudes from the display frame 111 while the face shape 112 protrudes from the display frame 111, the normal and normal face image 121 stored in the face image memory unit 46 may be sent to the other party. . Next, as to the method of FIG. 85, a case will be described in which a method of imaging a region larger than the display frame as a shooting frame is applied. FIG.
FIG. 86 is a configuration diagram illustrating only necessary components when a method of imaging a region larger than the display frame 111 as a shooting frame 120 is applied to the system of FIG. 85.

The position information required for the panning / tilting drive unit 71 by the area extraction unit 51 is supplied to the panning / tilting drive unit 71 via the main bus 37. FIG. 40 is a diagram illustrating a range in which the face shape 112 can be moved in order to move the face shape 112 to the center of the display frame 111 in the conventional method, that is, for an image in which the face image and the background are combined. Since the panning / tilting drive unit 71 is a mechanical drive unit, the range in which panning and tilting can be performed is determined. In FIG. 40, a is a range in which panning is possible, and b is a range in which tilting is possible. In other words, it is possible to perform imaging by moving the optical axis of the lens 70 only within the range of the imaging frame 120. Then, under the condition that the panning / tilting driving unit 71 is controlled so that the face shape 112 is positioned at the center of the display frame 111 within the range of the shooting frame 120, the range in which the face shape 112 can move is 112-1.
１１２112-6, that is, the range of the rectangle 119. On the other hand, in this method, the panning is roughly performed so that the face shape 112 is included in the video data stored in the image memory 45.
What is necessary is just to control the tilting drive part 71. That is,
This method has an advantage that the panning / tilting drive unit 71 does not need to be precisely controlled. Then, the face shape 112 is extracted by the shape extraction unit 20, and the face image 1 is extracted by the region extraction unit 51 using the face shape 112 and the digital image 54.
22 is generated, the face image 122 is moved to the center of the display frame 111, and transmitted to the other party.

FIG. 41 is a diagram showing a movable area of the face shape 112. That is, since there is no background, the face shape 112 can move until it comes into contact with the shooting frame 120. That is, 112-1 to 112-8 are movable ranges. The movable range 112-1 to 112-8 is equivalent to the range of the shooting frame 120. Comparing FIG. 40 with FIG. 41, the movable range of the face shape 112 in the conventional method is a rectangle 119.
Therefore, the movable range of the present method is the shooting frame 120, and the movable range of the present method is wider, and it can be said that the movable range is stronger against the fluctuation of the captured image. Next, as shown in FIG.
When the image 12 extends outside the shooting frame 120, the image inside the display frame 111 is displayed as it is on the image LCD 14, and the face image memory What is necessary is just to send the normal face image 112 that is stored in 46 to the other party.
In addition to the method configured as shown in FIG. 35 or FIG. 39 as described above, a text sentence 130 or the like can be displayed together with the face image 122 as shown in FIG. In this case as well, the face image 122 may be shifted to the right side and the display area of the text 130 may be secured on the left side in the same manner as described above.

For example, in the conventional method of displaying the background together in FIG. 86, the face shape 112 is shifted to the right of the display frame 111 and the display area of the text text 130 is secured on the left as shown in FIG. Face shape 112 and display frame 111
Is moved inside the shooting frame 120. And FIG.
Even when the face shape 112 approaches the right side as in FIG. 4, the text sentence 130 can be secured in the left area. However, when the face shape 112 is shifted to the left as shown in FIG. 45, an area for displaying the text 130 can be secured only on the right. That is, the text sentence 13
When trying to display 0, the display area of the text 130 changes depending on the position of the face shape 112, and the display screen displayed on the other party becomes very difficult to see.
Further, since the text 130 is superimposed on the background, there is a disadvantage that the contents of the text 130 are difficult to see depending on the video of the background. However, this method uses face shape 11
As long as 2 moves inside the shooting frame 120, the face video 122 and the text 130 are stably displayed at predetermined positions inside the display frame 111 as shown in FIG. Since the background of the text sentence 130 has no background, the text sentence 130 is clear and easy to see.

Next, a description will be given of a method of taking a large area beyond the display frame as a shooting frame when a face image is extracted by trimming as in the third embodiment. FIG.
FIG. 6 is a configuration diagram in which components related to an operation in a case where a system in which a region larger than the display frame is captured as a shooting frame is applied to the system of FIG. 86 are extracted. In this case, the operation of the address control unit 72 has nothing to do with this embodiment. The trimming shape 200 moving within the range of the shooting frame 120 may be extracted by the area extracting unit 51. Display frame 1
In some cases, the trimming shape 200 deviating from 11 can be extracted, and as described above, the image is more resistant to shaking of the captured image than when the imaging frame 120 is equal to the display frame 111. FIG. 47 is a diagram illustrating the movable area of the face shape 112 in the conventional method. That is, in the conventional method, the display frame 111 is set to the shooting frame 12 such that the background and the face region are the same, and the face shape 112 is located at the center of the display frame 111.
By moving the inside of 0, an image inside the display frame 111 is cut out. Under the condition that the face shape 112 is located at the center of the display frame 111, the movable range of the face shape 112 is drawn in the range of 112-1 to 112-6. That is, if the face shape 112 is moved inside the rectangle 119, the face shape 112 can be moved to the center of the display frame 111.

On the other hand, in this method, since the background is deleted, the trimming shape 200 may be cut out from the photographing frame 120. That is, as shown in FIG. 48, the trimming shape 200 can move to a range of 200-1 to 200-8. That is, it can be moved inside the shooting frame 120. 47 and FIG. 48, it is found that this method has a trimming shape 200
Indicates that the movable area has expanded from the rectangle 119 to the shooting frame 120, which indicates that the shot image is more resistant to shaking. Next, as shown in FIG.
When the number 7 protrudes from the shooting frame 120, as described above,
The image inside the display frame 111 is displayed as it is on the image LCD 14, and while the trimmed image 217 protrudes from the display frame 111, the normal trimmed image 203 stored in the trimmed image memory unit 56 without missing is not missing.
Can be sent to the other party. Next, a case will be described in which a method of imaging a region larger than the display frame as a shooting frame is applied to the method of FIG. 85. FIG. 50 is a configuration diagram in which components necessary for applying this method to the method of FIG. 85 are extracted and shown. FIG. 51 shows a conventional method.
That is, it is a diagram illustrating a range in which the face shape 112 can move in order to move the face shape 112 to the center of the display frame 111 in the image in which the face image and the background are combined. Since the panning / tilting drive unit 71 is a mechanical drive unit, the range in which panning and tilting can be performed is determined. In FIG. 51, a is a range in which panning is possible, and b is a range in which tilting is possible. In other words, it is possible to perform imaging by moving the optical axis of the lens 70 only within the range of the imaging frame 120. Then, under the condition that the panning / tilting driving unit 71 is controlled so that the face shape 112 is positioned at the center of the display frame 111 within the range of the shooting frame 120, the face shape 112 is controlled.
Is movable in the range of 112-1 to 112-6, that is, the range of the rectangle 119.

On the other hand, in the method shown in FIG. 50, the panning / tilting drive unit 7 is roughly set so that the trimming shape 200 is included in the video data stored in the image memory 45.
1 may be controlled. In other words, this method uses panning
There is an advantage that the tilting drive unit 71 does not need to be precisely controlled. Then, the trimming shape 200 is extracted by the region extracting unit 51, and the trimming image 201 is generated by the region extracting unit 51 using the trimming shape 200 and the digital image 54, and the trimming image 201 is displayed on the display frame 111.
And send it to the other party. FIG. 52 is a diagram showing a movable area of the trimming shape 200. That is, since there is no background, the trimming shape 200 can move until it comes into contact with the photographing frame 120. That is, the trimming shapes 200-1 to 200-8 are movable ranges. The movable range is equivalent to the range of the shooting frame 120.
51 and 52, since the movable range of the face shape 112 in the conventional method is a rectangle 119 and the movable range of the present method is the shooting frame 120, the movable range is wider in the present method. Therefore, it can be said that it is strong against shaking of the shot image. Next, as shown in FIG. 53, when the trimmed image 217 protrudes from the shooting frame 120, the image inside the display frame 111 is displayed as it is on the image LCD 14, and the trimmed image 217 is
During the protruding operation, the trimming video memory unit 56
Normal cropped video 2 not missing from the camera
03 can be sent to the other party.

In addition, in contrast to the method configured as shown in FIG. 46 or FIG. 50 as described above, a text 130 or the like can be displayed together with the trimmed image 203 as shown in FIG. In this case as well, the trimming video 203 may be shifted to the right and a display area for the text 130 may be secured on the left in the same manner as described above. For example, in the conventional method of displaying the background together with the method of FIG. 86, the area of the face shape 112 is secured on the right side of the display frame 111 and the display area of the text sentence 130 is secured on the left side as shown in FIG. For this purpose, the face shape 112 and the display frame 111 are moved inside the shooting frame 120. Then, as shown in FIG.
, The text sentence 130 can be secured in the left area. However, when the face shape 112 is shifted to the left as shown in FIG. 56, an area for displaying the text 130 can be secured only on the right. That is, when the text text 130 is displayed by the conventional method, the display area of the text text 130 changes depending on the position of the face shape 112, and the display screen displayed on the other party becomes very difficult to see. . Further, since the text 130 is superimposed on the background, there is a disadvantage that the contents of the text 130 are difficult to see depending on the video of the background.

However, in this method, as long as the trimming shape 200 moves inside the photographing frame 120, as shown in FIG.
0 is stably displayed at a predetermined position. Since the background of the text sentence 130 has no background, the text sentence 13
0 is clear and easy to see. Embodiment 8 Next, Embodiment 8 will be described. In the process of having the other party's subject image displayed on the user's portable information terminal device and having a conversation with each other, from the user side,
The camera unit 4 may be used to photograph the surrounding scenery or a specific object, and transmit the photographed image to the other party. In this case, it is a common usage that both the user and the other party have a conversation with each other while watching the captured video. The user needs to photograph the object and send the scenery as shown in FIG. 57 to the other side, and at the same time, monitor the image on the LCD 14 and adjust the camera angle and the like so as to be in an optimum state. Therefore, in this embodiment, the output of the camera direction sensor unit 28 is set to a state where the camera unit 4 is detached from the terminal main body 1 and the camera unit 4 can be freely turned in various directions. When detecting that the state has been switched to the opposite direction with respect to the display surfaces of the LCD 14 and the text LCD 16, the main control unit 11 automatically recognizes a state in which the user side captures an object and imitates the image. Face image 105 of the other party displayed on LCD 14
Then, the image is switched to the image captured by the camera unit 4, that is, the scenery shown in FIG.

Then, the output of the camera orientation sensor unit 28 detects a state in which the camera unit 4 is mounted on the terminal body 1 and the camera unit 4 is switched to the direction of the display surface of the video LCD 14 and the text LCD 16. In this case, the face image 105 of the other party is displayed on the image LCD 14 again.
As described above, in the present embodiment, the user side receives the output of the camera direction sensor unit 28 that detects the mounting state of the camera,
It detects whether it is taking a picture of the surrounding scenery or a specific object, or taking a picture of the user's own face. The purpose is to automatically switch the display image of the video LCD 14 to either a video image of the object itself or the face video 105 of the other party, and to switch the screen content of the video LCD 14. The advantage is that no switches are required. (Embodiment 9) Next, Embodiment 9 will be described. In this embodiment, an infrared LED and an infrared camera are attached to the terminal body 1 in the second embodiment, and a region of a subject is extracted using an image captured by the infrared camera.
FIG. 58 shows the configuration of the portable information terminal device of the ninth embodiment. In the figure, 65 is a shape control unit, 66 is a shape variable button, and 67 is a video memory unit.

FIG. 59 shows the appearance of the terminal body 1, the camera unit 4, and the infrared camera 5. The shape control unit 65 extracts a subject shape from the image captured by the infrared camera 5 in frame units, enlarges or reduces the shape information, and
Control is performed so that the shape of the subject with respect to 11 is a predetermined ratio. The video encoder 27 obtains object-coded video data according to the MPEG4 system using the digital video 54 given from the image memory 45 and the shape information 230 given from the shape controller 65. The pupil extraction unit 58 extracts a pupil portion from an image captured by the infrared camera 5. The shape variable button 66 changes the ratio of the subject shape to the display frame 111. With the infrared camera 5 attached to the terminal body 1 in the same direction as the shooting direction of the camera unit 4,
The following processing is performed. As described in the third embodiment,
The subject is photographed by the infrared camera 5 and first, the pupil extraction unit 5
At 8, the pupil portion is extracted. Here, the method of extracting the pupil can be extracted by, for example, Japanese Patent Application Laid-Open No. 9-175224. In this method, since the retina at the fundus absorbs visible light and reflects infrared light, only the pupil portions 204 and 205 of the subject are photographed with extremely high brightness as shown in FIG.

The pupil portions 204 and 205 are so large as to be incomparable with the amount of reflected light from the surrounding facial skin portion, and pupil portions g and h can be extracted by setting appropriate thresholds. The pupil parts g and h are moved so that the center of the pupil part becomes the center of the display frame. Next, the shape control unit 65
Extracts the shape information 230 of the subject. The subject is closer to the infrared camera 5 than the background, and the amount of reflected infrared light is larger. Therefore, an appropriate threshold value is set, and the input image is binarized by a binarization circuit. Then, the object shape information 230 can be extracted. Here, the ratio of the object shape to the display frame 111 is calculated from the shape information 230, and assuming a constant value, for example, 40%, the shape information 23 is set so that the calculated ratio is always 40%.
Control 0. When there are a plurality of subjects, the pupil parts g and h and the pupil parts g1 and h1 are extracted by the pupil extraction unit 58 as shown in FIG. In this case, the pupil parts g and h
Is extracted from the center i of the pupils g1 and h1 and the center i1 of the pupil parts g1 and h1, and is moved so that j is in the middle of the display frame. In this case, the ratio of the shape of the subject to the display frame may be set as large as, for example, 50%.

When the camera unit 4 is capturing an image as shown in FIG. 82, the shape control unit 65 extracts the object shape as shown in FIG. Since the area of the shape is calculated to be 30% from FIG. 61A, it is enlarged to 40% as shown in FIG. FIG.
In the case of (c), since the shape area is 45%, the shape area is reduced to form shape information 230. This ratio can be changed using the shape variable button 66. FIG. 62
In the case of (a), only one pupil portion is extracted by the pupil extraction unit 58, so that it is possible to detect that the subject is out of the display frame 111. In the case of FIGS. 62 (b) and 62 (c), the pupil portion can be extracted, but the subject is out of the display frame 111. In such a case, the state in which the subject image is in contact with the display frame 111 is checked,
It can be detected when one of the four sides of the trimming shape 200 that can be estimated from the pupil portions g and h in FIG. In such a case, when the subject image is normally extracted by the region extracting unit 51 as in the third embodiment, the subject image stored in the video memory unit 67 at the same time as the subject image is sent to the other party. As a still image.

The subject image stored in the video memory unit 67 is updated every frame or at a predetermined frame cycle. To stop. (Embodiment 10) In the ninth embodiment, the region of the subject is extracted using the infrared camera 5, but in the tenth embodiment, an ultrasonic wave is used instead of the infrared ray to extract. FIG. 63 shows a configuration in which only the components necessary for the description of this embodiment are extracted. When the ultrasonic pulse is transmitted from the ultrasonic transmitting unit 62, the ultrasonic pulse is reflected by the subject and the background. Therefore, the ultrasonic receiving unit 63 transmits the ultrasonic pulse as described with reference to FIG. Response time can be measured. Since the subject is closest to the terminal body 1, the shape of the subject can be extracted in the same manner as in the ninth embodiment by examining the shape of the point at which the response time is the shortest at t1. (Embodiment 11) Next, Embodiment 11 will be described. FIG. 64 is a block diagram illustrating a configuration of the portable information terminal device according to the eleventh embodiment. In this figure, 33 in the terminal body 1 is a screen switching button, and 39 is a hand shake sensor. Reference numeral 68 in the camera unit 4 denotes a CCD, and 69 denotes a signal processor.

The screen switching button 33 is used to switch between displaying the video sent from the other party on the video LCD 14 and displaying the upper body video of the user captured by the camera unit 4. is there. The camera unit 4
The CCD 68 captures light incident through the lens 70, converts the light into digital video data by the signal processor 69, and applies the digital video data to the camera IF unit 25. FIG.
FIG. 2 is a diagram illustrating an appearance of a terminal body 1 and a camera unit 4 according to the embodiment. In addition, the scroll dial 31, the operation button 32, and the screen switching button 33 take the size of a human hand into consideration,
While operating the scroll dial 31 with the thumb of the same hand,
Operation button 32 and screen switching button 3 with the remaining fingers of the same hand
The relative positions are determined so as to be able to operate. This operation will be described. As shown in FIG.
While the user is talking with the other party, the upper body image of the other party is displayed on the image LCD 14 of the terminal body 1 held by the user.
As shown in FIG. 2, the upper body video taken by the camera unit 4 is encoded by the video encoder 27 and sent to the other party,
It is assumed that the information is displayed on the display unit of the portable information terminal device of the other party.

Here, if the hand shake occurs while the user is holding the terminal body 1 with his / her hand, the hand shake sensor 39 detects the magnitude of the hand shake and the main control unit 11 knows the magnitude. Here, as the hand shake sensor 39, a gyro sensor, an acceleration sensor, a rotational angular velocity sensor, or the like is used. The shake detection output obtained from the hand shake sensor 39 is output separately for an X-axis (horizontal axis) component and a Y-axis component (vertical axis). The main controller 11 calculates the absolute value (swing displacement) of the vector synthesized by the X-axis component and the Y-axis component. Then, as shown in FIG. 66, the main control unit 11 sets the upper body of the user to the display frame 111 between a time a when the swing displacement exceeds a predetermined threshold value e and a predetermined time b as shown in FIG. The image approaching the end is displayed on the image LCD 14. Then, the user sees such an image and finds that the hand shaking has become severe, and within the period b in which the image of the user is displayed, the user switches the terminal body 1 and as shown in FIG. Display frame 1 on the upper body
Modify it to be at the center of 11. Here, the terminal body 1
In some cases, the terminal body 1 is fixedly placed on a desk or the like in front of the user. In this case, the upper body image of the user sent to the other party is stable because hand shaking does not occur, but the direction of the camera unit 4 is not necessarily in the direction of the user.
It may not fit in. Therefore, the direction of the camera unit 4 is operated with a finger, and the upper body image of the user is displayed on the image LCD 14.
May be adjusted so as to fit in the display frame 111.

Therefore, as shown in FIG. 67, the hand shake sensor 39 may be mounted inside the camera unit 4 and only the movement of the camera unit 4 may be detected by the hand shake sensor 39. If the hand shake sensor 39 is attached to the camera unit 4, the camera unit 4 also shakes when the terminal body 1 shakes, so that there is an advantage that the shake of the terminal body 1 can be detected. Embodiment 12 Next, Embodiment 12 will be described. In the eleventh embodiment, when the hand shake becomes severe and the displacement of the hand shake exceeds a predetermined size when the user holds the terminal body 1 with his / her hand, the image LCD 14 is displayed instead of the upper body image of the other party. The upper body image was displayed. However, there is a disadvantage that the upper body of the opponent is not displayed on the video LCD 14 even for a short period, and the opponent cannot be observed during that period. Therefore, this embodiment is characterized in that when the hand-shake displacement exceeds a predetermined threshold value e as shown in FIG. 68A, the user's upper body image 140 is displayed in a multi-window together with the partner's upper body image 141. I have. In this embodiment, the upper body image 140 of the user overlaps with the upper body image 141 of the other party, making it difficult to see. However, the image to be focused on now depends on how the upper body image 140 of the user is reflected. And the user does not need to take a short time to take a correct pose, so that no problem occurs. Further, when the displacement of the hand shake exceeds a predetermined size as shown in FIG. 68 (b), the upper body image 140 of the user is made larger, and the upper body image 141 of the other party is made smaller by overlapping the upper body image 140 of the user. They may be displayed in an overlapping manner.

(Thirteenth Embodiment) Next, a thirteenth embodiment will be described. In the twelfth embodiment, the image LCD 1
4 shows the upper body image 141 of the opponent and the upper body image 14 of the user
When displaying 0 in a multi-window, the size of the user's upper body image 140 is a fixed size. However, in this embodiment, the user's upper body image 140 is made variable according to the magnitude of the shaking displacement. It is characterized by. FIG.
As described above, the threshold value of the swing displacement is divided into a plurality of grades and set in three stages, e1, e2 and e3, for example. Then, it is checked whether or not the rising peak of the shaking displacement, that is, the point a, is within the threshold value. If the peak value is between e1 and e2, the displacement is considered to be relatively small, and FIG.
An upper body image 140 of a smaller user as shown in FIG. The peak value is e
If it is between 2 and e3, the middle user's upper body image 140 is displayed as a medium user's upper body image 141 as shown in FIG.
Layer on. When the peak value is e3 or more, FIG.
1, the user's upper body image 140 is completely replaced with the user's upper body image 140 instead of the partner's upper body image 141. That is, in this embodiment, when the shaking displacement is small, the relatively small upper body image 140 of the user is placed on the upper body image 141 of the opponent.
, It is possible to reduce the concealed display that occurs when the upper body image 141 of the opponent is overlapped by the multi-window. Further, it is another object of the present invention to make it easier for the user to adjust the direction of the terminal main body 1 by displaying the upper body image 140 of the user larger when the displacement becomes larger.

Embodiment 14 Next, Embodiment 14 will be described. In the eleventh embodiment to the thirteenth embodiment, when checking the swing displacement, as shown in FIG. 66 (or FIG. 69), the user's movement is performed for a predetermined period b after the swing displacement becomes larger than the predetermined threshold value e (or e1). Upper body image 140 for image LC
D14 is displayed. Then, the user operates the upper body image 14 displayed on the image LCD 14 during this period.
For example, the terminal body 1 is held differently so that “0” falls within the display frame 111 of the video LCD 14. However, if the period b is short, the period b may pass while the terminal body 1 is moving. Also, if the period is too long, the user moves the terminal body 1 and the user's upper body image 140 can be fit into the display frame 111 of the video LCD 14 before switching to the other person's upper body image 141. There is. Therefore, the present embodiment aims at displaying the upper body image 140 of the user on the image LCD 14 only during the period when the user is moving the terminal body 1 during the switching period. As shown in FIG. 72, the envelope 13 connecting the peaks of the shaking displacement 131
Draw 2. After a lapse of time a from the time when the envelope 132 becomes larger than the threshold value e, the video is switched to the upper body video 140 of the user. Time a is the envelope 1
When 32 falls below the threshold value e, it is ignored because it is regarded as the peak value of the temporary fluctuation.

When the envelope 132 has fallen below the threshold value e and is still lower than time a after a lapse of time a, the video is switched to the upper body image 141 of the other party. Here, if the envelope 132 has exceeded the threshold value e within a time a after the envelope 132 has fallen below the threshold value e, the envelope 132 has been ignored because it has caused a temporary valley. In this embodiment, when the display of the upper body image 141 of the other party is switched to the display of the upper body image 140 of the user, the embodiment 12
Also, as in the thirteenth embodiment, the upper body image 141 of the other party and the upper body image 140 of the user may be displayed together in a multi-window. (Embodiment 15) Next, Embodiment 15 will be described. This embodiment is characterized in that the user's upper body image 140 and the partner's upper body image 141 are periodically switched. That is, as shown in FIG. 73, a process in which the display time of the user's upper body image 140 is set to a and the display interval b, and the other party's upper body image 141 is displayed periodically except for the display time of the user's upper body image 140, and is periodically repeated. I do. In this embodiment, since the upper body image 140 of the user is displayed periodically without checking the swing displacement 131 or the like, there is an advantage that the processing is simplified.

In the above description, as shown in FIG.
The time for displaying the upper body image 140 of the user is inserted into the period in which the upper body image 141 of the other party is displayed on the CD 14. However, most of the time, as shown in FIG.
The upper body image 140 of the user may be displayed on D14, and the other person's upper body image 141 may be inserted in between. In particular, when the opponent is a superior person, the upper body image 140 of the user can always be confirmed when the user's upper body image 140 that fits in the display frame 111 is sent. Further, the display time a of the user's upper body image 140 and the display time b of the partner's upper body image 141 may be arbitrarily set via the touch panel 30. In this embodiment, when the display of the partner's upper body image 141 is switched from the display of the partner's upper body image 141 to the display of the user's upper body image 140, as in Embodiments 12 and 13, the partner's upper body image 141 and the user's upper body image 140 May be displayed together in a multi-window. (Embodiment 16) Next, Embodiment 16 will be described. In the fifteenth embodiment, a period in which the upper body image 140 of the user is periodically switched while the upper body image 141 of the other party is displayed is inserted.

In this embodiment, when a line is connected to a partner and a telephone call is started, first, a user's video LCD 14
A user's upper body image 140 is displayed for a predetermined period of time. In other words, call the other party,
When a call is received from the other party, it is necessary to perform a key operation of the terminal body 1 or the like. When a key operation is performed, the terminal body 1 may move, and it may not be possible to determine whether the camera unit 4 is correctly facing the user. Then, in this state, the upper body image 141 of the other party is received, and the upper body image 140 of the user is sent to the other party without confirming by himself, and there is a possibility that the upper body image outside the display frame 111 is sent. Therefore, in this embodiment, when the communication is started, the upper body image 140 of the user is always displayed on the display frame 1.
The purpose is to make a call with the other party after confirming that the number is within 11. In other words, the user can take a pause at the beginning of the communication and make a pause so that a normal upper body image is obtained, and then make a call. In this embodiment, when displaying the upper body image 140 of the user, as in the twelfth and thirteenth embodiments, the upper body image 141 of the other party and the upper body image 140 of the user may be displayed together in a multi-window. Good.

(Embodiment 17) Next, Embodiment 17 will be described. In the fifteenth embodiment, the cycle of displaying the upper body image of the user is fixed. However, the cycle may be changed in accordance with the swing displacement by always checking the swing displacement. FIG. 75 is an example in which an envelope 136 is drawn for the peak value of the swing displacement 135. Then, from the graph of FIG.
The display interval is determined according to 6. When the displacement is between the first value 139 and the second value 137, as the magnitude of the displacement increases, the display interval is shortened as shown by a straight line 138. That is, when the displacement is large, the shaking is severe, so that the display interval is also shortened so that the user's upper body image 140 is frequently shown to the user. In the state where the displacement is small, the display interval is lengthened because the swing is small. And the displacement is the first value 13
In the case of 9 or less, the display interval is fixed to a constant value. Also, when the displacement is equal to or greater than the second value 137, the display interval is fixed at a constant value. FIG. 77 is a diagram showing a screen switching state. The display interval, that is, the interval of the user's upper body image display period 144 changes with time. In addition, as shown in FIG.
When the envelope 143 of 42 is equal to or smaller than the threshold e, the period during which the upper body image 140 of the user is displayed is fixed as a.
However, as shown in FIG.
After the period b during which 0 is displayed, the user notices that the upper body image 140 of the user is out of the display frame 111, and shifts to the operation of changing the direction of the terminal body 1. The hand movement of the terminal body 1 becomes large, and the envelope 143 exceeds the threshold value e at the time point g.

Then, during the period c, the terminal main body 1 is moved. Then, at time h, the envelope 143 of the shaking displacement 142 falls below the threshold value e. If the shaking displacement 142 does not exceed the threshold value e again during the period d, the display is switched to the upper body image 141 of the partner at the time point i. That is,
This embodiment aims at displaying the user's upper body image 140 while the user is moving the terminal body 1 by hand while the user's upper body image 140 is being displayed. That is, in the case of FIG. 78 (a), the period during which the upper body image 140 of the user is displayed is fixed.
In the case of FIG. 8 (b), the user confirms the degree of shaking for a certain period of time b, and after the user adaptively takes the time required to correct the hand shaking, the upper body image 141 of the other party. The display is switched. This embodiment, which switches as shown in FIG. 78 (b), can also be applied to the fifteenth embodiment. In other words, even if the display interval b is constant, the display time a may be switched to the display of the other party after the confirmation time of the user is secured and the display time a is adaptively extended in accordance with the operation time of the user. Similarly, this embodiment switched as shown in FIG. 78B can be applied to the sixteenth embodiment. That is, after starting communication with the other party, the upper body image 140 of the user is displayed on the image LCD 14, and after confirming the user's confirmation time, the user is adaptively changed while holding the terminal body 1. The timing of switching the display to the upper body image 140 may be extended.

In this embodiment, when the display of the upper body image 141 of the other party is switched from the display of the upper body image 141 of the other party to the display of the upper body image 140 of the user, the upper body image 141 of the other party and the upper body image 140 of the user are changed as in the twelfth and thirteenth embodiments. May be displayed together in a multi-window. (Embodiment 18) Next, Embodiment 18 will be described. In a state in which the user or the other party is uttering voice and has been silent for a predetermined period or more, the conversation has been settled down and a break has taken place, and time has been secured. The upper body image 140 of the user may be displayed on the image LCD 14 during the pause period exceeding the predetermined period. FIG. 79 shows the result of examining the power 150 of the voice of the user or the other party.
If 0 does not exceed the threshold value e, the display of the image LCD 14 is switched to the upper body image 140 of the user. If the power 150 does not fall below the threshold e even after the time a has elapsed since the voice of the user or the partner has exceeded the threshold e again, the display on the video LCD 14 is displayed on the upper body image 14 of the partner.
Switch to 1. In this embodiment, when the display of the upper body image 141 of the partner is switched from the display of the upper body image 141 of the user to the display of the upper body image 140 of the user, the upper body image 141 of the partner and the upper body image 140 of the user are multi-windowed as in the twelfth and thirteenth embodiments. They may be displayed together.

(Nineteenth Embodiment) Next, a nineteenth embodiment will be described. In the process in which the user is talking with the other party, the uttering period during which the user is uttering is determined by the upper body image 14 of the user.
1 is displayed on the video LCD 14, and the other person's upper body image 140
May be displayed. That is, when the user is uttering, by taking a pause, the user is less likely to feel uncomfortable. FIG. 80 is a diagram showing a time relationship between the utterance of the user and the utterance state of the partner. When the user starts uttering, the audio power 151 exceeds a predetermined threshold value e, and within a predetermined time c, the audio power 151 becomes equal to the threshold value e.
If not, the display on the video LCD 14 is switched to the upper body video 140 of the user. Then, even when the utterance of the user ends and a silent section is reached (period f), the display of the upper body 140 of the user is displayed as it is. Then, when the other party utters the voice in the middle of the silent section, the power 152 of the other party's voice exceeds the threshold value e and the predetermined time c
If the audio power 152 does not fall below the threshold value e, the display of the image LCD 14 is switched to the upper body image 141 of the other party. Then, when the user utters while the other party is uttering and the voice of the other party and the user's voice overlap, the user's utterance is prioritized, and after a lapse of time c from the start of utterance in the same manner as described above, the image LCD 14 is displayed. Display the upper body image of the user 14
The display is switched to 0 (period d).

In the above description, the upper body image 140 of the user is displayed on the image LCD 14 when the user and the other party are in a silent state and when the user and the other party are uttered together. The upper body image 141 of the other party may be displayed on the LCD 14 with priority.
In this embodiment, when switching from the display of the upper body image 141 of the other party to the display 140 of the upper body image of the user, the upper body image 141 of the other party and the upper body image 140 of the user are combined in a multi-window as in the twelfth and thirteenth embodiments. May be displayed. (Embodiment 20) Next, Embodiment 20 will be described. Embodiment 11 to Embodiment 14, and Embodiment 17
Has performed a process of switching from the partner's upper body image 141 to the user's upper body image 140 and displaying it when the shake component detected by the hand shake sensor 39 exceeds a predetermined threshold.
However, the hand shake sensor 39 only detects the shake of the terminal body 1 and shakes the user (mainly the shake of the upper body of the user).
Cannot be detected. That is, even if the hand shake is small, if the user's upper body shake is large, the user's upper body image 140 sent to the other party will protrude from the display frame 111.

Therefore, in this embodiment, as shown in FIG.
The purpose is to extract only the shaking of the upper body of the user by changing the optical axis of the lens 70 according to the shaking component and canceling the shaking component. A camera shake sensor 39 is attached to the camera unit 4. The hand shake detection output from the hand shake sensor 39 is output to the hand shake correction signal generation means 7.
3 is applied. A predetermined hand shake correction signal is generated by the hand shake correction signal generation means 73, and this signal is supplied to the hand shake correction means 74. By this loop, even if there is a hand shake, a stable image in which the hand shake component is canceled can be taken. Here, as the hand shake correction means 74, a vertex angle variable prism that changes the optical axis by adjusting the angle of a lens in which a liquid is sealed, or a pair of purely cylindrical lenses (or spherical lenses) is used. An apex angle variable prism or the like, which is arranged so that the cylindrical surface is rotatable, rotates the cylindrical surface by a predetermined amount according to the amount of camera shake, and adjusts the optical axis by changing the apex angle of the prism. Can be used. As described above, the upper body image 140 of the user output from the camera unit 4 is applied to the image encoder 27 via the camera IF unit 25.

The video encoder 27 supplies the motion monitoring unit 75 with a motion vector over a plurality of frames. The motion vector may be a motion vector of all blocks constituting a frame or a part of blocks at a predetermined position in the frame. The motion monitoring unit 75 includes the video encoder 27
And the size of an object (for example, a standard face size) within the field of view of the camera unit 4 based on an evaluation value such as an addition value (and an average value) of absolute values of motion vectors over a plurality of frames supplied from the camera. Determine the frequency. The determination result is compared with a preset threshold value, and when the result is equal to or larger than the threshold value, the result is output to the main control unit 11 as an output of the motion monitoring unit 75. The movement of only the upper body of the user is evaluated from the upper body video from which the hand shake component has been canceled by the above processing. And
The method of detecting the movement of the upper body of the user according to the present embodiment is applied instead of the hand shake detection of the terminal body 1 in the embodiments 11 to 14 and 17, and the detected face movement is determined by a predetermined threshold. If the number exceeds the limit, the processing specified in the embodiments 11 to 14 and 17 may be performed. In this embodiment, when the display of the upper body image 141 of the other party is switched to the display of the upper body image 140 of the user, the upper body image 1 of the other party as in the twelfth and thirteenth embodiments is used.
41 and the upper body image 140 of the user may be displayed together in a multi-window.

(Embodiment 21) Next, Embodiment 21 will be described. In each of the above embodiments, the upper body image 141 of the user is displayed by switching the display of the image LCD 14 as necessary while the received upper body image 141 of the other party is always decoded by the image decoder 12. However, generally, the video decoder 12 has a large processing amount,
Power consumption is high because it is executed by a DSP or the like. Therefore, in this embodiment, the display of the other party's upper body image 141 is switched from the display of the partner's upper body image 141 to the image LCD 14 described in each of the above embodiments, and during the period in which the user's upper body image 140 is displayed, the video decoder 12 is controlled by the power supply unit 36. The purpose is to stop the power supply and save the power consumption of the entire terminal body 1. During the above-mentioned period, the video taken by the camera unit 4 is supplied directly to the video LCD control circuit unit 13 via the camera IF unit 25, and then the video L
Displayed on CD14. Further, in a case where the camera unit 4 is oriented in the direction opposite to the upper body image 140 of the user to shoot a scene as shown in FIG. 57 and the image is sent to the other party, the user monitors the scene with the image LCD 14 while monitoring the scene. Shooting is more convenient. Therefore, the camera orientation sensor unit 2
If the camera unit 4 detects a state in which the camera unit 4 faces in a direction opposite to the user in 8, the power supply of the video decoder 12 may be stopped. When the user operates the screen switching button 33 to forcibly switch the image LCD 14 to display the upper body image 140 of the user, the power supply of the image decoder 12 may be stopped.

Here, the state where the upper body image 141 of the other party is displayed on the image LCD 14 is switched to a state where the upper body image 140 or the scenery of the user is displayed, and then the upper body image 141 of the other party is displayed. Let's consider the state that has been switched to. FIG. 81 is a diagram illustrating a relationship between a received frame and a video switching period. The frame received from the other party continues without interruption. The frame indicated by I in the received frames is an intra picture. First, the video switching period is the upper body image 140 of the user.
Alternatively, the period during which the scenery is displayed, that is, frame a
The decoding process is not performed by the video decoder 12 for the video images of (c) to (c). If the timing of switching to the upper body image 141 of the other party is a frame other than the intra picture, that is, if the picture is a P or B picture, there is a mutual prediction dependency. It may not be possible. Therefore, in this embodiment, it is only checked whether the received frame is an I, P, or B frame, and the decoding is not executed until the next intra picture I arrives, that is, the frames e to d. During the period f, the display of the user's upper body image / scenery is extended, the partner's upper body image 141 is decoded from the frame b which is the intra picture I, and the display of the video LCD 14 is switched to the partner's upper body image 141.

[0100]

As described above, according to the portable information terminal device of the present invention, even when the housing is shaken due to hand shaking or the like, the other party can display the image stably without shaking.
A video with a small transmission amount can be sent. In addition, when the user's shooting angle is not appropriate, the user can be made aware of this, and the user changes the way of holding the housing so that the user's own subject is at a predetermined position on the screen can do. Furthermore, when the direction of the camera is changed to an object such as a landscape, the image displayed on the user side can be automatically switched to the image of the object,
Operability can be improved. In addition, power consumption can be saved by stopping decoding of a video transmitted from the other party while the user's video is being displayed on the user's side.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a portable information terminal device according to a first embodiment.

FIG. 2 is a view showing an appearance of the portable information terminal device according to the first embodiment.

FIG. 3 is a block diagram showing another configuration example of the portable information terminal device according to the first embodiment.

FIG. 4 is a display diagram showing a user's face image displayed on the other side.

FIG. 5 is a display diagram showing a partner's face image displayed on the user side.

FIG. 6 is an explanatory diagram showing an arrangement of a user-side face image displayed on the other side.

FIG. 7 is a block diagram showing a configuration of a portable information terminal device according to a second embodiment.

FIG. 8 is a diagram illustrating an appearance of a portable information terminal device according to a second embodiment.

FIG. 9 is a display diagram showing a face shape on the user side displayed on the other side.

FIG. 10 is a display diagram for explaining a change in a face shape due to a zooming operation.

FIG. 11 is a block diagram showing another configuration example of the portable information terminal device according to the second embodiment.

FIG. 12 is an explanatory diagram showing an arrangement of a user's face image displayed on the other side.

FIG. 13 is a diagram showing a moving range of a face video on the user side displayed on the other side.

FIG. 14 is a diagram showing an example in which a face image and a text sentence are displayed together on a display frame.

FIG. 15 is a block diagram showing a configuration of a portable information terminal device according to a third embodiment.

FIG. 16 is a diagram illustrating an appearance of a portable information terminal device according to a third embodiment.

FIG. 17 is a display diagram for explaining the arrangement of a trimming shape.

FIG. 18 is a display diagram for explaining the arrangement of a trimming video.

FIG. 19 is a diagram showing an example of a trimming shape.

FIG. 20 is an explanatory diagram of a method of determining a trimming shape from the position of the user's eyeball.

FIG. 21 is an explanatory diagram of a method of determining a trimming shape from the positions of two eyes.

FIG. 22 is an explanatory diagram of a method for determining a trimming shape by a conventional face region extraction method.

FIG. 23 is a view for explaining the size and arrangement when the trimming shape is an ellipse.

FIG. 24 is a configuration diagram showing a main part of another example of the portable information terminal device according to the third embodiment.

FIG. 25 is a diagram showing a time relationship between reflected waves of an ultrasonic wave.

FIG. 26 is a diagram showing a moving range of a trimming video.

FIG. 27 is a diagram showing an example in which a trimmed video and a text sentence are displayed together.

FIG. 28 is an explanatory diagram showing a state in which a face image has touched an edge of a display frame.

FIG. 29 is an explanatory diagram showing a state in which a face image protrudes from a display frame.

FIG. 30 is a display diagram showing a raw image when a face image touches an edge of a display frame.

FIG. 31 is a display diagram showing a raw image when a face image protrudes from a display frame.

FIG. 32 is an explanatory diagram showing a state in which a trimming image protrudes from a display frame.

FIG. 33 is a display diagram showing a face image and a raw image that change during a zooming operation.

FIG. 34 is a display diagram showing a trimmed video and a raw video that are changed by a zooming operation.

FIG. 35 is a block diagram showing a main configuration of a portable information terminal device according to a seventh embodiment.

FIG. 36 is a view showing a movement range of a face shape in the conventional method shown in FIG. 86;

FIG. 37 is a view showing a movement range of a face shape in the method shown in FIG. 35;

FIG. 38 is a display diagram showing a raw image when the face shape is out of the shooting frame.

FIG. 39 is a configuration diagram showing a main part of another example of the portable information terminal device according to the seventh embodiment.

FIG. 40 is a view showing a movement range of a face shape in the conventional method shown in FIG. 85;

FIG. 41 is a view showing a moving range of a face shape in the method shown in FIG. 39;

FIG. 42 is a display diagram showing a raw image when the face shape is out of the shooting frame.

FIG. 43 is a view showing an example in which a video and a text sentence are displayed together in a conventional method.

FIG. 44 is a diagram showing another example in which a video and a text are displayed together in a conventional method.

FIG. 45 is a view showing another example in which a video and a text are displayed together in the conventional method.

FIG. 46 is a configuration diagram showing a main part of another example of the portable information terminal device according to the seventh embodiment.

47 is a view showing a movement range of a face shape in the conventional method shown in FIG. 86.

FIG. 48 is a view showing a movement range of a trimming shape in the method shown in FIG. 46;

FIG. 49 is a view showing a raw video when the trimmed video is out of the shooting frame.

FIG. 50 is a block diagram showing another configuration example of the portable information terminal device according to the seventh embodiment.

FIG. 51 is a view showing a movement range of a face shape in the conventional method shown in FIG. 85;

FIG. 52 is a view showing a movement range of a trimming shape in the method shown in FIG. 50;

FIG. 53 is a view showing a raw image when the trimmed image is out of the shooting frame.

FIG. 54 is a view showing an example in which a video and a text are displayed together in a conventional method.

FIG. 55 is a view showing another example in which a video and a text are displayed together in the conventional method.

FIG. 56 is a view showing another example in which a video and a text are displayed together in the conventional method.

FIG. 57 is a display diagram showing a scene captured by changing the direction of a camera in the eighth embodiment.

FIG. 58 is a block diagram showing a configuration of a portable information terminal device according to a ninth embodiment.

FIG. 59 is a diagram showing an appearance of a portable information terminal device according to a ninth embodiment.

FIG. 60 is an explanatory diagram of a method of obtaining shape information of a subject from the positions of two eyes.

FIG. 61 is a display diagram for explaining changes in subject shape information.

FIG. 62 is a display diagram showing shape information when the object is out of the display frame.

FIG. 63 is a block diagram showing a main configuration of a portable information terminal device according to the tenth embodiment.

FIG. 64 is a block diagram showing a configuration of a portable information terminal device according to Embodiment 11.

FIG. 65 is a diagram illustrating an appearance of a portable information terminal device according to an eleventh embodiment.

FIG. 66 is a view showing a relationship between displacement of hand shaking and a video switching period.

FIG. 67 is a configuration diagram showing another example of the portable information terminal device according to the eleventh embodiment.

FIG. 68 is a display diagram in which the video of the other party and the video of the user are displayed in a multi-window.

FIG. 69 is a view showing the relationship between the graded hand shake displacement and the video switching period.

FIG. 70 is a view showing a change in the multi-window display due to the displacement of hand shaking.

FIG. 71 is a display diagram showing the upper body image of the user when the displacement of the hand shake is large.

FIG. 72 is a view showing the relationship between the displacement of hand shaking and the video switching period.

FIG. 73 is a view showing the display time and display interval of the upper body video of the user.

FIG. 74 is a view showing the display time of the upper body image of the user and the other party.

FIG. 75 is a diagram in which the envelope of the peak value of the hand shake displacement is plotted.

FIG. 76 is a view showing the relationship between the shaking displacement and the display interval of the upper body image of the user.

FIG. 77 is an explanatory view showing a state where the display interval of the upper body image of the user is changing.

FIG. 78 is an exemplary view for explaining a video display period according to a displacement of shaking and its extension.

FIG. 79 is a diagram showing a time relationship for displaying the upper body image of the user during the audio pause period.

FIG. 80 is a diagram showing a time relationship in which the upper body image of the user is displayed during the user utterance period.

FIG. 81 is a view showing a time relationship of a pause period of the decoding unit with respect to a received frame.

FIG. 82 is a display diagram showing a user-side subject (center position) displayed on the other side.

FIG. 83 is a display diagram showing a subject on the other side displayed on the user side.

FIG. 84 is a display diagram showing a user-side subject (leftward) displayed on the other side.

FIG. 85 is a configuration diagram illustrating an example of a conventional camera shake prevention mechanism.

FIG. 86 is a configuration diagram illustrating another example of a conventional camera shake prevention mechanism.

FIG. 87 is a display diagram showing deformation of a subject on the user side displayed on the other side.

FIG. 88 is a diagram showing a relationship between a shooting frame and a display frame in a conventional example.

FIG. 89 is a diagram showing a relationship between a photographing frame and a display frame (when the image is protruded) in a conventional example.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Terminal main body 3 ... Earphone microphone 4 ... Camera part 4b ... Support part 4a ... Camera part main body 4c ... Hinge part 5 ... Infrared camera 11 ... Main control part 12 ... Video decoder 13 ... Video LCD control circuit part 14 ... Video LCD 15: Text LCD control circuit 16: Text LCD 17: Demultiplexer 18: PHS line IF (PHS line interface) 19: Antenna 20: Shape extractor 21: Area extractor 22: Terminal for infrared camera 23 audio codec 24 earphone microphone terminal 25 camera IF section (camera interface section) 26 camera terminal 27 video encoder 28 camera direction sensor section 29 operation input control circuit section 30 touch panel 31 scroll dial 32 ... operation buttons 34 ... power button 36 ... power supply section 37 ... main bus 38 Synchronous bus 39 Hand shake sensor 41 Azimuth sensor 42 Conversion unit 43 Person recognition unit 45 Image memory 46 Face image memory unit 47 Zooming button 48 Zooming drive unit 50 Luminance signal and hue signal 51 Area extraction unit 54 Digital image 55 Trimming shape storage unit 56 Trimming image memory unit 57 Infrared LED 58 Pupil extraction unit 60 Housing 61-1, 61-2 Concavity 62 Ultrasonic transmission unit 63 Ultra Sound wave receiving unit 64 ... Trimming shape variable button 65 ... Shape control unit 66 ... Shape variable button 67 ... Video memory unit 68 ... CCD 69 ... Signal processor 70 ... Lens 71 ... Panning / tilting driving unit 73 ... Hand shake correction signal generation Means 74: hand shake correction means 75: motion monitoring unit 100: subject and background 101 of the user 06, 108, 121, 122 ... face images 102, 107, 113, 118, 119 ... rectangle 103 ... center of rectangles 104, 114, 117 ... center of face images 105 ... face image of the other party 111 ... display frames 112, 116 ... Face shape 113,119 ... Rectangle 114,117 ... Center of face image 120 ... Shooting frame 130 ... Text sentence 131,135,142 ... Shake displacement 132,136,143 ... Envelope 137 ... Second value 138 ... Line 139: first value 140: user's upper body image 141 ... partner's upper body image 144 ... user's upper body image display period 151, 152, 153 ... audio power 200, 202 ... trimming shape 201, 203, 217, 218 ... trimming Video g, h, 204, 205: pupil part 206: center of gravity of trimming shape 207: display Center 208 ... position in the height of both eyes 209 ... intermediate 210 ... video signal 215,216,219 ... missing part 230 ... shape information

Claims (22)

[Claims] 1. A photographing means for photographing a subject and a background of a user by a camera while a housing is held by a user, and extracting a region of the subject photographed by the photographing means by predetermined processing, and extracting the region. Area extracting means for obtaining a subject image obtained by the detection, detecting means for detecting a direction and a moving distance of the subject image obtained by the area extracting means from a predetermined position in a predetermined range, and a detection result of the detecting means. Means for moving the subject image to the predetermined position using the communication device, communicating means for communicating with the counterpart information terminal device via a predetermined communication line, and transmitting the subject image moved to the predetermined position to the communication device. And a video transmission control means for transmitting to the other information terminal device by means of the portable information terminal device. 2. The method according to claim 1, wherein the detecting means detects that the subject image contacts an end of the predetermined range or that at least a part of the subject image is out of the predetermined range. And means for displaying the subject and the background.
A portable information terminal device according to claim 1. 3. A means for transmitting an image including the subject and the background to the counterpart information terminal device when the subject image moved to a predetermined position in the predetermined range is enlarged and is out of the predetermined range. The portable information terminal device according to claim 1, further comprising: 4. A means for accumulating the subject image immediately before the subject image protrudes from the predetermined range, and reading out the stored subject image as a still image when the subject image protrudes from the predetermined range, 3. The portable information terminal device according to claim 1, further comprising: a unit that transmits to the counterpart information terminal device via the communication line during a period in which the information terminal device is out of a predetermined range. 5. A photographing means for photographing a subject and a background of the user by a camera in a photographing area wider than a display area while the housing is held by a user, and a photographing means in the photographing area photographed by the photographing means. An area extracting means for extracting an area of the object by predetermined processing to obtain an image of the extracted object; a means for moving the display area so that the image of the object fits in the display area; Detecting means for detecting a direction and a moving distance of the subject image moved from a predetermined position in the display area, and means for moving the subject image to the predetermined position by using a detection result of the detecting means; Communication means for communicating with the information terminal device of the other party through a predetermined communication line; and the subject image moved to the predetermined position by the communication means. Portable information terminal apparatus characterized by comprising a video transmission control means for transmitting to the information terminal device. 6. When the subject image contacts an edge of the photographing area or when it is detected that at least a part of the subject image protrudes from the photographing area, the user's object and background are determined on the user side. The portable information terminal device according to claim 5, further comprising a display unit. 7. A means for accumulating the subject image immediately before the subject image protrudes from the shooting area, and reading out the stored subject image as a still image when the subject image protrudes from the shooting area, 7. The portable information terminal device according to claim 5, further comprising: a unit that transmits to the counterpart information terminal device via the communication line during a period of protruding from an imaging area. 8. A photographing means for photographing a subject and a background of the user by a camera in a state where the housing is held by a user, and extracting a region of the subject photographed by the photographing means by predetermined processing, and extracting the region. Area extracting means for obtaining a subject image obtained, means for detecting the mounting direction of the camera in the housing, and detecting that the mounting direction of the camera deviates from the shooting direction of the subject by this means. In the case, a means for displaying an image captured by the camera on the user side, a communication means for communicating with a counterpart information terminal device via a predetermined communication line, and a mounting direction of the camera in a shooting direction of the subject In some cases, it is detected that the subject image in which the region has been extracted and that the mounting direction of the camera is out of the shooting direction of the subject. Portable information terminal apparatus characterized by comprising a video transmission control means for transmitting the image the camera is photographed on the mating information terminal device by the communication unit if. 9. An infrared radiating means and an infrared camera provided on said housing, and said infrared camera irradiates said infrared camera with a reflection of infrared light to extract shape information of said subject. Means, and a shape control means for controlling the shape information so that a ratio of the subject image to the predetermined range is a predetermined ratio, wherein the region extracting means is configured to detect the object photographed by the photographing means. 2. The portable information terminal device according to claim 1, wherein an area is extracted using the shape information controlled by the shape control means, and the subject image in which the area is extracted is obtained. 10. An ultrasonic transmitter and an ultrasonic receiver provided in the housing, and an ultrasonic pulse transmitted from the ultrasonic transmitter is received by the ultrasonic receiver to form the object. Means for extracting information, and shape control means for controlling the shape information so that a ratio of the subject image to the predetermined range is a predetermined ratio, wherein the area extracting means is photographed by the photographing means. 2. The portable information terminal device according to claim 1, wherein an area of the subject is extracted using the shape information controlled by the shape control unit, and the subject image in which the area is extracted is obtained. . 11. A photographing means for photographing a subject and a background of the user by a camera while the housing is held by the user, a means for detecting a displacement of the shaking when the housing is shaken, and Means for displaying the subject and background of the user for a predetermined period on the user side when the detected displacement of the shaking exceeds a predetermined threshold value, and communicating with the counterpart information terminal device via a predetermined communication line And a video transmission control unit for transmitting a video including the user's subject to the communication partner information terminal device by the communication unit. 12. A photographing means for photographing a subject and a background of the user by a camera while the housing is held by the user, and displaying the subject and the background of the user at a predetermined period and a predetermined period on the user side. Means for performing communication with the other-side information terminal device via a predetermined communication line, and video transmission control means for transmitting a video including the user's subject to the other-side information terminal device by the communication means. A portable information terminal device comprising: 13. A photographing means for photographing a subject and a background of the user by a camera while the housing is held by the user; a communication means for communicating with a counterpart information terminal device via a predetermined communication line; A video transmission control unit for transmitting a video including the user's subject to the counterpart information terminal device by the communication unit, detecting a silent section of the voice uttered by the user, and setting a predetermined length from the silent section to First detecting means for detecting the exceeded pause period, and receiving a voice transmitted from the counterpart information terminal device via the communication means, detecting a silent section of the voice, and determining a predetermined period from the silent section. Second detecting means for detecting a pause period exceeding the length, and displaying the subject and background of the user on the user side during the pause period detected by the first and second detection means. Portable information terminal apparatus characterized by comprising a that means. 14. A photographing means for photographing a subject and a background of the user by a camera in a state where the housing is held by the user, a communication means for communicating with a counterpart information terminal device via a predetermined communication line, A video transmission control unit for transmitting a video including the user's subject to the communication partner information terminal device by the communication unit; and a case where the power of the voice uttered by the user exceeds a predetermined threshold and continues for a predetermined time. A portable information terminal device comprising: means for detecting an utterance period; and means for displaying a subject and a background of the user on the user side during the utterance period. 15. A photographing means for photographing a subject and a background of the user by a camera in a state where the housing is held by the user; a communication means for communicating with a counterpart information terminal device via a predetermined communication line; A video transmission control unit for transmitting a video including the user's subject to the counterpart information terminal device by the communication unit; and a user's subject on the user side when the communication unit starts communication with the counterpart information terminal device. And a means for displaying a background. 16. A photographing means for photographing a subject and a background of the user by a camera while the housing is held by the user, a communication means for communicating with a counterpart information terminal device via a predetermined communication line, Video transmission control means for transmitting an image including the user's subject to the other information terminal device by the communication means, means for detecting a period during which the user's side image is displayed on the user side, and the period Means for stopping the supply of power to decoding means for video information transmitted from said partner information terminal apparatus by said communication means. 17. The system according to claim 17, further comprising: means for displaying, in the user's side, a subject and a background of the user in a multi-window together with a video transmitted from the partner information terminal device. The portable information terminal device according to claim 2, claim 6, or claim 11 to claim 15. 18. A means for displaying a subject and a background of the user on the user side in a multi-window manner together with a video transmitted from the partner information terminal device, and dividing the threshold value into a plurality of classes. 12. The portable information according to claim 11, further comprising: means for applying the displacement of the shaking to the class; and means for changing a size of a display of a subject and a background of the user according to the class. Terminal device. 19. The portable information terminal device according to claim 12, further comprising means for setting said period and period. 20. The apparatus according to claim 1, further comprising: means for detecting a displacement of the shaking when the casing is shaken, and means for changing the period in accordance with the displacement of the shaking detected by the means. 13. The portable information terminal device according to 12. 21. When displaying the subject and the background of the user on the user side, the displacement of the shaking when the housing shakes is determined by a predetermined threshold value because the user performs an operation of correcting hand shaking. 13. The portable information terminal device according to claim 11, further comprising: means for extending a display of a subject and a background of the user during a period exceeding the period. 22. A photographing means for photographing a subject and a background of the user by a camera in a state where the housing is held by the user; a communication means for communicating with a counterpart information terminal device via a predetermined communication line; Video transmission control means for transmitting an image including the user's subject to the counterpart information terminal device by the communication means, detection means for detecting displacement of hand shake when the housing is shaken by hand shake, Means for canceling a hand-shake component in the video by adding a hand-shake displacement and a hand-shake component detected by the detection means to a video of the user's subject and the background captured by a camera; Means for detecting a shaking component of the subject of the user from the image in which the component of the hand shaking is canceled, and the magnitude of the shaking component of the subject detected by the means is predetermined. When exceeding the threshold value, the portable information terminal apparatus, characterized in that at the user side and means for displaying the object and the background of the user.