JP3339949B2 - Control method of video conference communication terminal - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for controlling a video conference communication terminal capable of remotely controlling a video camera device installed at a partner terminal.

[0002]

2. Description of the Related Art For example, CCITT recommendation H.264. There has been proposed a video conference communication device having a video conference communication function specified by H.320 and exchanging video and audio between remote locations to realize conference communication. In particular, in recent years, a video conference communication device has been regarded as important as a communication application using ISDN as a line, and is gradually spreading.

[0003] In such a video conference communication device, a user of the own terminal can arbitrarily operate a photographing screen of a video camera device of the other terminal on the video monitor device of the own terminal. In order to be able to do so, a device having a function of remotely controlling a video camera device of a partner terminal has been put into practical use.

[0004]

However, such a conventional apparatus has the following inconveniences.

That is, since video information exchanged with a partner terminal is in a state of being coded and compressed by a predetermined coding and compression method, the video information is encoded and processed (the partner terminal side). For the decoding process (own terminal side), for example, between the shooting screen of the video camera device of the partner terminal and the display screen of the video monitor device of the own terminal,
A time delay of about 1 second occurs (CCITT Recommendation H.26)
1 CIF format). Therefore, the user of the own terminal sees the screen one second before the video camera device of the partner terminal.

On the other hand, an operation signal for remotely controlling the video camera device of the partner terminal is received by the partner terminal with a time delay substantially equal to the transmission delay time. This is reflected in the camera angle of the video camera device of the terminal.

For this reason, when the operation is stopped when the display screen of the video monitor device of the own terminal is at a desired camera angle, the screen seen by the user at the time of the stop is the one of the video camera device of the partner terminal. Since the screen is a few seconds before, the camera angle changes in the operation direction for about one second from the time when the operation is stopped, and a situation occurs in which the camera angle does not reach the camera angle desired by the user.

As described above, when remotely operating the video camera apparatus, the relationship between the remote operation by the user and the change in the camera angle lacks real-time properties, and thus the inconvenience of poor operability has been caused.

The present invention has been made in view of the above circumstances, and has as its object to provide a control method of a video conference communication terminal that can improve operability when remotely controlling a video camera device of a partner terminal. .

[0010]

SUMMARY OF THE INVENTION The present invention provides a method for controlling a video conference communication terminal capable of remotely controlling a video camera device installed in a partner terminal, a memory for storing received moving image data, and a memory for storing the moving image data in the memory. Out of the received moving image data, a display control means for cutting out data of a display area of a predetermined size and displaying a display screen of the cut out moving image data on a video monitor device. In addition to controlling the display control means to read out the corresponding image data, when remotely controlling the video camera device of the partner terminal to change the shooting range, the video camera device corresponding to the display screen at the time of starting the remote control is displayed. Still screen data is formed, and during the operation period, the data input by the display control means is switched to the still screen data, In a mode in which the display area is displaced from the center of the still screen data in accordance with the contents of the operation, the display control means is controlled to cut out the still screen data, and when the operation period ends, the operation period is stopped. A remote operation amount of the video camera device from a start point to an end point is transmitted to a partner terminal as a camera operation signal, and the display control means is returned to a state where data in the memory is input.

[0011]

[0012]

[0013]

[0014]

[0015]

[0016]

[0017]

[0018]

[0019]

[0020]

[0021]

Also, in the control method of the video conference communication terminal capable of remotely controlling the video camera device installed in the partner terminal, the memory for storing the received moving image data, and the received moving image data stored in the memory, Display control means for cutting out data of a display area having a predetermined size and displaying a display screen of the cut-out moving image data on a video monitor device, and always reading out image data corresponding to an image at a predetermined position in the display area. While controlling the display control means, when remotely operating the video camera device of the partner terminal to change the shooting range, form still screen data corresponding to the display screen at the start of the remote operation, During the operation period, the data input by the display control means is switched to the still screen data, and the data is switched according to the remote operation content. When the display area is displaced from the center of the still screen data, the display control means is controlled to cut out the still screen data, and when the display area deviates from the still screen, The still screen data is updated to the content of the received image, and when the operation period ends, the remote operation amount of the video camera device from the last update time of the still image data to the end of the operation period is used as a camera operation signal as a camera operation signal. To return to a state in which the display control means inputs data in the memory.

Also, in the control method of a video conference communication terminal capable of remotely controlling a video camera device installed in a partner terminal, the video camera device of the partner terminal is set to a minimum magnification applicable at the start of the video conference communication. In such a state, the initial still screen data corresponding to the received moving image at that time is formed and stored, and thereafter, the received moving image is always displayed, and the video camera device of the partner terminal is
When remotely operating to change the shooting range,
While displaying the display screen of the initial still screen data, on the display screen, a frame image indicating a display frame corresponding to the shooting range of the video camera device of the partner terminal is synthesized and displayed,
While the display area of the frame image is moved according to the operation content, when the remote operation ends, a video camera operation signal corresponding to the display area of the frame image at that time is transmitted to the other terminal, and then the reception screen is displayed. Is displayed again.

[0024]

Therefore, a still image of the received image at the start of the camera operation is formed, the still image is displayed, and the same screen as when the camera is actually operated is displayed in accordance with the camera operation. Since the display area of the still image is controlled, the user can perform a camera operation while referring to the still screen when performing operations such as pan, tilt, and zoom for the video camera of the partner terminal. Further, at the time of starting the video conference communication, a still image of the received image on the screen in a state where the TV camera device of the partner terminal is fully zoomed out is formed, and during the TV camera operation period, the still image is displayed, In response to the camera operation, the display area of the still image is controlled so that the same screen as when the camera is actually operated is operated, so that operations such as panning, tilting, and zooming are performed on the video camera of the partner terminal. At this time, the user can operate the camera while referring to the still screen.

[0025]

Embodiments of the present invention will be described below in detail with reference to the accompanying drawings.

FIG. 1 shows an example of a video conference communication device according to an embodiment of the present invention. Here, this video conference communication device is based on CCITT Recommendation H.264. It has a teleconference communication function specified by 320 and uses ISDN as a transmission line.

In FIG. 1, a system control unit 1 is for controlling the operation of each unit of the video conference communication terminal device and executing a video conference communication control procedure process on an information channel (B channel). A ROM (read only memory) 2 is for storing a part of a control processing program executed by the system control unit 1 and various data referred to when executing the program, and a RAM (random only memory).・ Access memory) 3
Configures a work area and the like of the system control unit 1.

The clock circuit 4 is for outputting current time information, and the image processing unit 5 is, for example, for reducing / enlarging an image, for processing a still image of a photographic screen, and for encoding / decoding a still image. This is for executing various kinds of image processing necessary for the device functions provided by the video conference communication terminal device, such as a conversion process (JPEG system) and a statistical process of images.

The magnetic disk device 6 includes a system controller 1
For storing program data such as other control programs and application programs to be executed by the CPU, or various other data.
Is for operating this video conference communication terminal device. In addition, the operation display unit 7 displays the operation of the camera angle of the video camera device of the own terminal, for example, “pan (right, left)” for shaking the camera angle in the left and right direction, and “tilt (up, down) for shaking the camera angle in the up and down direction. ) ", Changing the magnification," zoom (in (increase the target screen ratio), out (decreasing the target screen ratio)) ", and changing the focal position" focus (in Video camera operation keys for adjusting (matching), out (out of focus from the object)) and operation keys for switching the operation object between the own terminal and the partner terminal. The still image input device 8
This is for inputting a still image, for example, from a still image camera device.

The voice input / output device 9 is for exchanging voice information with a partner terminal to realize a conversation between users, and the voice CODEC 10 is for outputting analog voice input from the voice input / output device 9. It converts a signal into corresponding digital audio data, converts received digital audio data into a corresponding analog audio signal, and outputs it to the audio input / output device 9. The voice control unit 11 controls the operation of the voice input / output device 9.

The video camera device 12 is for photographing a state of a participant or the like on its own terminal side.
Reference numeral 13 denotes an NTSC output from the video camera device 12.
The video signal of the type is converted into video data of the corresponding digital data, and the video data is converted into video data of a predetermined CIF format or QCIF type.
CITT Recommendation H. 261 to form video information by encoding and compressing the encoded video information, convert the encoded and compressed video information into original CIF or QCIF type video data, and convert the video data to a corresponding analog signal. Convert the video signal to NTSC
It converts it into a video signal of a model.

The video signal converted and formed by the video CODEC 13 is output to the display control unit 14. The video signal output from the video camera device 12 is also applied to the display control unit 14. The display control unit 14 controls the contents of the display screen of the video monitor device 15. The video camera control section 16 is for controlling the camera angle of the video camera device 12.

The ISDN interface control unit 17 connects the video conference communication terminal device to the ISDN, and performs a layer 1 signal processing function, a D channel (signal channel) signal and two B channel (information channel) signals. It has an integration / separation function.

The D-channel transmission control unit 18 has a call control procedure function performed on a signal channel and a multiplex call control function.
TT Recommendation H. H.221 data multiplexing / demultiplexing function and a data frame synchronization function. Digital audio data,
Digital video data and general-purpose data are added, and the connection end on the multiplexing side is connected to the ISDN interface control unit 17.

The system control unit 1, ROM 2, R
AM3, clock circuit 4, image processing unit 5, magnetic disk device 6, operation display unit 7, still image input device 8, audio CODEC
10, an audio controller 11, a video CODEC 13, a display controller 14, a video camera controller 16, a D-channel transmission controller 18, and a multiplexer / demultiplexer 19 are connected to a system bus 20. Data exchange between elements is mainly performed via the system bus 20.

The multiplexer / demultiplexer 19 and the audio COD
Audio data is exchanged with the EC 10, and video data is exchanged between the multiplexer / demultiplexer 19 and the video CODEC 13.

Now, in this video conference communication terminal device,
In the B channel, the CCITT Recommendation H. 221 is exchanged in a multi-frame format.

As shown in FIG. 2, one multi-frame MFL is composed of eight sub-multi-frames SMF1 to SMF.
8 in each sub-multiframe SMF1 to SMF1.
The SMF 8 is composed of two frames each. That is, one multi-frame MFL is composed of 16 frames FLM0 to FLM15.

Each of the frames FLM0 to FLM15
Is composed of 80 octets of data, as shown in FIG. 3, and the respective bit positions where these octets are arranged in bit order constitute subchannels SCH1 to SCH8.

The eighth bit of the first to eighth octets is a frame synchronization signal (Frame Ali).
The 8th bit of the 9th to 16th octets constitutes a bit rate allocation signal (Bit rate Allocation).
n Signal) BAS. Also, in the part of the 8th bit of the 17th to 80th octets of the sub-channel SCH8, data of an encryption channel for exchanging key information and the like for encrypting data is set in a part thereof. May be (optional).

As described above, the frame synchronization signal FAS
Are arranged in one frame FLM0 to FLM15, and the bit allocation is configured in units of a multi-frame MFL as shown in FIG.

That is, even frames FLM0, FLM
2,..., FLM14, second to eighth octets, followed by odd frames FLM1, FLM3,.
A horizontal synchronizing signal composed of an 8-bit data pattern “00110111” is arranged in the FLM 15, and a first octet of the odd-numbered frames FLM 1, FLM 3,. A synchronization signal is provided.

By detecting the horizontal synchronizing signal and the vertical synchronizing signal, the synchronization of one multi-frame MFL can be detected.

The 0th frame, the 2nd frame, the 4th frame
Bits N1, N2, N3, N4, and N5 of the first octet of the frame, the sixth frame, and the eighth frame are used to indicate a multiframe number. Of these, bit N5 is used to indicate whether a multi-frame number is used. As described above, since the data used for the multi-frame number is 4 bits, the multi-frame number changes in a descending order with values from 0 to 15, and the same multi-frame number appears every 16 multi-frames.

The tenth frame, the twelfth frame,
And the bit L of the first octet of the thirteenth frame
1, L2 and L3 are used to display connection numbers indicating the order of connection of the information channels carrying the frame among the information channels currently used. The bit R of the first octet of the fifteenth frame is reserved (reserved) for a future recommendation, and is normally set to 0.

The bit TEA of the first octet of the fourteenth frame is used to indicate that data cannot be transmitted due to an internal failure of the data terminal device.

The odd frames FLM1, FLM3,
.., Bit A of the third octet of FLM 15 is used to indicate whether frame synchronization or multi-frame synchronization has been established, or whether synchronization has been lost.

The odd frames FLM1, FLM3,
,..., The fifth octet, the sixth octet, the seventh octet, and the eighth octet of the FLM 15
C2, C3 and C4 are used to indicate a CRC (Cyclic Redundancy Check) code which is referred to for data error detection (ie, channel quality detection) of two consecutive frames (ie, sub-multiframes). The bit E of the fourth octet of the odd frames FLM1, FLM3,..., FM15 is used to indicate that a transmission error has been detected on the receiving side.

Further, as shown in FIG. 5, the bit rate allocation signal BAS includes even-numbered frames FLM0, FLM2,
, FLM14, 8-bit data representing a capability BAS or a BAS command is arranged, and subsequent odd-numbered frames FLM1, FLM3,.
A double error correction code for error correcting the value of the capability BAS or BAS command transmitted in the immediately preceding frame is arranged.

The transmission of the data of the multi-frame MFL is performed in the order of the frame number, and each of the frames FLM0 to FLM15 is transmitted in the octet order from the first octet to the 80th octet as shown in FIG. Octets are transmitted with the first bit first.

That is, each frame FLM0
In the FLM 15, the first bit of the first octet is transmitted first, and the eighth bit of the 80th octet is transmitted last.

In this video conference communication terminal device,
Normally, data transmission is performed using two information channels, and an example of a transmission procedure at that time is shown in FIG. This transmission procedure is described in CCITT Recommendation H.264. 242 and CCITT Recommendation H.264. Follow 320.

First, the calling terminal calls the destination terminal, performs a call setting procedure on the signal channel, secures one information channel (hereinafter, referred to as a first channel) (phase A), and sets the information channel. Frame synchronization is performed while exchanging frame data in which PCM audio data (A-law or μ-law, 64 Kbps) is set on one channel (frame mode). When frame synchronization is established, capability BAS data and command BAS data are mutually exchanged. Exchange (phase B1-1), the transmission mode to be used at that time is determined, and an additional call setting request for securing the second information channel is started (phase B1-2).

Then, the common mode having the highest function is selected from the contents exchanged between the transmission modes at that time (phase B1-3), and the calling terminal transmits the function to the called terminal in the selected operation mode. Is transmitted to set the parameters common to the device functions of the calling terminal and the called terminal (phase B2). Thus, in the first channel, for example, data transmission of audio data (16 Kbps) and data transmission of moving image data (46.4 Kbps) corresponding to the transmission mode selected at that time are performed (phase C). .

When the first channel starts data transmission in the frame mode, a call setup procedure is performed on a signal channel for a second information channel (hereinafter, referred to as a second channel) (phase CA), and the second channel is established. Then, frame data including only the frame synchronization signal FAS and the bit allocation signal BAS are exchanged using the second channel to establish frame synchronization and multi-frame synchronization (phase CB1-11). Establish synchronization between channels (phase CB1-1)
2).

When the synchronization of the two information channels is completed,
A transmission mode is set by transmitting a BAS command from the calling terminal side (phase CB1-2), the transmission mode is switched to the set contents (phase CB1-3), and common parameters are set (phase CB3).

When the initialization of the second channel is completed in this way, after that, in a state where the frame data exchanged on the first channel and the frame data exchanged on the second channel are synchronized, for example, Using two information channels, audio data, video data, and general-purpose data are 56 Kbps and 62.4 K, respectively.
Data transmission is performed with transmission rates of bps and 6.4 Kbps assigned (see FIG. 8).

When ending such data transmission, first, the second channel is disconnected. At this time, the first
A procedure for setting a common mode is performed for voice data transmission performed only on the channel (phase CD).
1) The mode of the second channel is switched to the mode 0F of the frame mode (phase CD2). At this time, the first channel and the second channel are asynchronous, and the second channel is in a state where a call is held in a transmission state of only the frame synchronization signal FAS and the bit allocation signal BAS, and the signal channel is disconnected. The call on the second channel can be released by the release procedure.

In the first channel, the phase CD1
For example, during the phase CD2, in the frame mode, for example, audio data and moving image data are transmitted at a total transmission speed of 62.4 Kbps, and when the operator of one terminal ends the call, the transmission of the moving image data ends. In order to use the transmission capacity of all the first channels including the transmission capacity of the moving image data for audio transmission, mode 0 is used.
F (Phase D2), after which the call of the first channel can be released by the call disconnection release procedure of the signaling channel.

As a result, the call disconnection / release procedure is executed on the signal channel for the first channel and the second channel (phase E), and the video conference communication between the two terminals ends.

As described above, in the video conference communication terminal device, first, one information channel (first channel) is secured and the frame mode is established, and then the transmission speed of the audio data and the moving image data in this first channel is reduced. If the second information channel (second channel) can be simultaneously set while performing data transmission in the transient mode by performing the assignment, the second channel is secured by the call setting procedure of the signaling channel. Then, the first channel transmitting in the transient mode is synchronized with the newly secured second channel, and when the channel synchronization is established, the transmission speed of the audio data and the moving image data is reset. In order to utilize the increased transmission capacity of the information channel, the encoding rule (encoding method) of the audio data and the moving image data is changed so as to exchange higher quality audio data and moving image data. I have.

When ending the data transmission, it is necessary to change from the mode using the transmission path in which both the first channel and the second channel are synchronized to the mode using only the first channel. is there. Therefore, in this case, first, each of the encoding rules of the audio data and the moving image data is set to a method of optimizing the transmission capacity of the first channel to 62.4 Kbps, and the transmission mode is changed to the transmission mode of only the first channel. In the second channel, the state of synchronization with the first channel is stopped, and after the mode shifts to mode 0F in which user data is in an idle transmission state, the call is released by releasing the signal channel. /release. Further, the first channel is switched from two types of media transmission of audio data and moving image data to mode 0F, which is a transmission mode of only audio data, and then disconnects / releases the call by a call disconnection / release procedure of the signaling channel. . The first channel can be disconnected / released after directly changing the mode to the mode 0F without switching the mode to the two types of media transmission of the audio data and the moving image data. Also,
The accounting information and the like obtained in the call disconnection release procedure are managed for each channel.

In addition, CCITT Recommendation H. In the video conference communication function specified in 320, video CODEC1
As an encoding method of CCITT Recommendation CC. The moving picture coding method specified in H.261 is applied. This CCIT
Recommendation T. H. Next, the moving image coding system of H.261 will be described. Here, two types of formats, CIF and QCIF, are defined as moving image information, but in the following description, only CIF will be described.

CIF (Common Intermed)
iate Format (common intermediate format) is a basic format of moving image information.
Displays a screen of 29.97 frames per second. Regarding the luminance component (Y), one frame (image frame) is composed of 288 lines, one line is composed of 360 pixels, and the color difference component (Y) is For Cb, Cr), one frame is composed of 144 lines, and one line is composed of 180 pixels.

Then, the frame is divided into 12 GOBs (Group Of Bloc) as shown in FIG.
k), and each GOB is divided into 33 macroblocks of 11 × 3, as shown in FIG.

Further, as shown in FIG. 3C, one macroblock has a luminance component of 2 × 2 = 4.
In addition to the division into one block, the color difference component is divided into 1 × 1 blocks. Therefore, one block of the luminance component and the chrominance component is composed of 8 pixels × 8 lines as shown in FIG.

Further, QCIF (Quarter CIF)
Is a format in which the number of pixels is set to 1/4 of CIF, as shown in FIG. Note that CCITT Recommendation H.
H.261 specifies that the moving image CODEC needs to be able to handle at least QCIF,
When F and QCIF face each other, it is defined that communication is performed by QCIF.

This moving picture information corresponds to the hierarchical structure of frame / GOB / macroblock / block as described above, and one example is shown in FIGS. 11 (a) to 11 (d). The following description is based on CCITT Recommendation H.264. H.261, which is a domestic standard corresponding to H.261.

First, as shown in FIG. 3A, one frame of moving image information includes a frame start code PSC composed of 20-bit data of a predetermined bit pattern representing the start of a frame, and a frame number TR representing the sequence number of the frame. It comprises type information PTYPE representing information on the entire frame, data insertion information for extension PEI, preliminary information PSPARE, and GOB data. Here, the extension data insertion information PEI may include at least one, and may not include the preliminary information PSPARE.
The frame start code PSC, the frame number TR, the type information PTYPE, the extension data insertion information PEI, and the spare information PSPARE are called a frame header.

One GOB data is arranged for each GOB. One GOB data is a G that indicates the start of the GOB data as shown in FIG.
OB start code GBSC, GOB number GN indicating the position of GOB, quantization characteristic information GQUA indicating information of quantization characteristic
It includes NT, extension data insertion information GEI indicating presence / absence of extension data area (preliminary information), preliminary information GSPARE, and macroblock data. Here, the extension data insertion information GEI may include at least one, and may not include the preliminary information GSPARE. Also, GOB
Start code GBSC, GOB number GN, quantization characteristic information G
QUANT, extension data insertion information GEI, and preliminary information GSPARE are called GOB headers.

One macro block data is arranged for each macro block. However, if there is no information in that part of the frame, it is not transmitted.

As shown in FIG. 9C, one macroblock data includes a macroblock address MBA for indicating the position of the macroblock, a type of the macroblock, and a type for indicating which data element appears. Information MTYPE, quantization characteristic information MQUANT representing quantization characteristics, motion vector information MV
D, a significant block pattern CB indicating that at least one transform coefficient is a transmitted macroblock
P and block data. Here, the quantization characteristic MQUANT, the motion vector information MVD, and the significant block pattern CBP appear when designated by the type information MTYPE. The macro block address MBA, the type information MTYPE, and the quantization characteristic information M
QUANT, motion vector information MVD, and significant block pattern CBP are called a macroblock header.

As shown in FIG. 7D, for example, the block data includes a transform coefficient TCOEFF representing transform coefficient data obtained by performing DCT (discrete cosine transform) processing on the image data of the block, and It consists of a block end code EOB indicating the end.

The CCITT Recommendation H. In the moving image coding method of H.261, for example, the quality of the moving image to be exchanged can be determined to some extent by appropriately changing the quantization step size of the moving image within a settable range (an even value of 2 to 62). Then, the quantization characteristic information GQUANT included in the GOB data is set to the content corresponding to the quantization step size applied to the subsequent GOB, and the quantization characteristic information MQUANT included in the macroblock data is It is set to the content corresponding to the quantization step size applied to the subsequent macroblock.

That is, conceptually, this CCIT
Recommendation T. H. In the moving image coding method of H.261, the dynamic range of a code can be limited by dividing a transform coefficient obtained by performing DCT processing on image data by a quantization step value.

Therefore, when the quantization step size is set to a small value, the quantization error is reduced, and the image quality when reproducing the encoded moving picture information is improved (ie,
Higher definition), but on the other hand, the data amount increases. In this way, the amount of data per frame increases,
When the data amount per second becomes larger than the data transmission speed assigned to the moving image information, the video CODEC1 is used to prevent the transmission buffer from overflowing.
No. 3 suppresses the data amount of the moving image information by performing frame skipping (dropping frames) of the moving image. That is, in this case, the ability to follow the movement of the image is reduced.

Conversely, if the quantization step size is set to a large value, the quantization error increases, and the image quality when reproducing the encoded moving picture information deteriorates (that is, the definition decreases). Since the data amount is reduced, the followability of the screen is improved.

Further, in the video CODEC 13, the setting of the quantization step size is performed adaptively.
As the application factor referred at that time, for example, the code data amount is used. Therefore, the video CODEC 13
Normally operates so as to adaptively change the quantization step size so as to reduce the amount of code data.
Further, the maximum value of the quantization step size is defined by the maximum value of the quantization step size, and the maximum value of the quantization step size can be designated to the video CODEC 13 from outside.

Therefore, if the maximum quantization step size specified in the video CODEC 13 is set to a large value,
Since the video CODEC 13 adaptively changes the quantization step size up to the quantization step size maximum value, for example, the transmission data amount per frame is reduced as compared with the case where the quantization step size maximum value is set small. can do.

In this embodiment, general-purpose data is
A video camera remote operation signal for operating the video camera device 12 on the partner terminal side is exchanged.

This video camera remote operation signal includes, for example, a start request command (START ACTION Re
quest) and a stop request command (STOP ACTION R) for requesting to stop the operation being executed.
request).

The start request command is composed of 2 bytes of data as shown in FIG. 12 (a), and the first byte has the first byte for indicating that the video camera remote operation signal is a start request command. A predetermined bit pattern (for example, 01H (representing an H hexadecimal number; the same applies hereinafter)) is set, and operation type data representing the content of the operation requested to be started is set in the second byte.

The stop request command is composed of 2 bytes of data, as shown in FIG. 8B, and the first byte has a video camera remote control signal indicating that the video camera remote operation signal is a stop request command. A predetermined bit pattern (for example, 02H) is set, and in the second byte, operation type data representing the operation content requested to be stopped is set.

Here, the eighth bit of the operation type data is assigned to a bit P for designating a pan operation,
The bit is a bit PD for specifying the direction of the pan operation
(Data 0 is left; data 1 is right)
The bit is allocated to a bit T for specifying a tilt operation, the fifth bit is allocated to a bit TD (data 0 is lower; data 1 is upper) for specifying a direction of the tilt operation, and the fourth bit is zoom. The third bit is assigned to a bit ZD (data 0 is out; the data 1 is in) for designating the direction of the zoom operation, and the second bit is designated to a focus operation. The first bit is a bit FD for designating the direction of the focus operation.
(Data 0 is out; data 1 is in).

In this embodiment, the video camera 1
As shown in FIG. 13, the amount of displacement in the operation direction when operating 2 is displaced by a constant amount of displacement per unit time.

For example, when the user operates the camera angle of the video camera device 12 of the partner terminal while viewing the display screen while displaying the received image from the partner terminal on the video monitor device 15 of the terminal itself, At the time of input, the system control unit 1 issues a start request command (STA
Sends an RT ACTION Request to the partner terminal.

Thus, the system control unit 1 of the partner terminal
Outputs a camera operation signal to the video camera control unit 16, whereby the camera angle of the video camera device 12 of the partner terminal gradually changes. Changes gradually.

Then, when the screen displayed on the video monitor device 15 becomes the desired camera angle of the user and the user stops operating the camera, at that time, the system control unit 1 issues a stop request command (START). A
Ction Request).

As a result, the system control unit 1 of the partner terminal
Stops outputting the camera operation signal to the video camera control unit 16, whereby the camera angle of the video camera device 12 of the partner terminal is fixed to the state at that time.

Here, the display screen of the video monitor device 15 viewed by the user is actually displayed at the time before the encoding / decoding processing time in the video CODEC 13 of the own terminal and the other terminal by the other terminal. Video camera device 12
This is the content shot by.

Therefore, the camera angle of the video camera device 12 is changed after the user completes the operation.
When the encoding / decoding processing time in the ODEC 13 elapses, the operation is stopped. For this reason, the camera is further displaced from the camera angle desired by the user by a delay time dT corresponding to the encoding / decoding processing time. Fixed.

Therefore, in the present embodiment, when the video camera device 12 is remotely operated and receives a stop request command from the partner terminal, the movement of the video camera device 12 in the operation direction is stopped at that time and the video camera device 12 is stopped. Coasting displacement dL obtained by displacing the camera device 12 during the delay time dT
Only by operating in the opposite direction to the operation direction at that time,
The camera angle desired by the user (operator) of the partner terminal can be obtained by a single operation.

Further, the delay time dT at this time can be mutually reported to the partner terminal as an arbitrary information element in an appropriate transmission control procedure signal exchanged in advance, for example.

FIG. 14 shows an example of processing when a camera operation is input through the operation display unit 7.

When a camera operation is input (decision 10)
1 is YES), it is checked whether the operation target at that time is remote (the partner terminal side) (decision 10).
2). If the result of determination 102 is YES, the above-described start request command (START ACTION Request) specifying the operation content at that time is transmitted (process 103), and the process waits until the camera operation at that time ends (determination 104). .

When the camera operation is completed and the result of determination 104 is YES, the above-described stop request command (STOP ACTION Re
quest) (process 105), and the camera operation process at this time ends.

When the operation target is local (own terminal side) and the result of determination 102 is NO, the value of the maximum quantization step size of the data transmission side of the video CODEC 13 is changed to a predetermined large value. (Process 10
6) In this state, an operation signal indicating the content of the operation at that time is transmitted to the video camera control unit 16 until the camera operation is completed (step 107, NO in determination 108).
loop).

When the operation of the camera ends and the result of decision 108 becomes YES, the value of the maximum quantization step size on the transmission side of the video CODEC 13 is returned to the original value (Process 1).
09), this operation ends.

FIG. 15 shows a processing example when a camera operation is requested from the partner terminal.

When a start request command is received (decision 20)
1 is YES), the value of the quantization step size maximum value on the data transmission side of the video CODEC 13 is changed to a predetermined large value (process 202), and the video camera remains in this state until a stop request command is received. An operation signal representing the operation content designated at that time is sent to the control unit 16 (process 203, NO loop of judgment 204).

When the stop request command is received and the result of determination 204 is YES, the coasting displacement dL corresponding to the delay time dT stored at that time is calculated (process 205). The operation information for operating the coast displacement amount dL in the direction opposite to the camera operation direction is sent to the video camera control unit 16 to correct the coast displacement amount dL (process 206).

Then, the maximum value of the quantization step size on the data transmission side of the video CODEC 13 is returned to the original value (process 207), and this process is terminated.

As described above, in the present embodiment, when a camera operation is requested from the partner terminal, the requested camera operation is executed, and the coasting displacement dL at that time is corrected to perform the operation. A camera angle desired by the terminal user is realized.

In this embodiment, when operating the camera angle, the camera angle is operated by changing the maximum value of the quantization step size on the data transmission side of the video CODEC 13 to a predetermined large value. When the content of the screen changes and the moving image portion of the shooting screen increases and the code data amount of the moving image information increases, a larger quantization step size can be set so that the code data amount does not increase so much. I have.

As a result, when operating the camera angle, it is possible to avoid a situation in which the amount of code data is increased and a frame is dropped, and a situation in which a jerky screen is obtained is prevented. be able to.

Here, the value of the maximum quantization step size after the change in this case can be obtained experimentally.
Further, a maximum value that can be taken may be set.

In the above embodiment, the maximum value of the quantization step size of the video CODEC 13 was changed to a large value in order to suppress the amount of code data. A video camera operation amount (operation amount per unit time) is limited according to a motion vector detection range. (B) A video camera operation amount is corresponding to a transmission speed (transmission capacity) allocated to moving image information. (C) Increasing the value of the transmission rate assigned to the moving image information (changing the transmission rate assignment)
and so on.

The amount of moving image information increases as the number of moving parts on the screen (moving image part) increases, and the amount of movement (speed) per unit time increases. When the camera angle is moved, all the contents in the screen move, so that the moving image portion sharply increases, whereby the amount of moving image information increases rapidly. When the amount of moving image information increases in this way, the time required for encoding / decoding increases, and the above-described delay time dT increases.

On the other hand, when the amount of information increases, the video CODEC 13 performs a so-called frame dropping process of reducing the number of frames (pictures) to be transmitted per unit time. Even when frame dropping does not occur, if the amount of information increases, a situation similar to that of frame dropping occurs on the receiving side, resulting in a jerky motion of the image.

Further, the transmission speed (transmission capacity) of moving image information
If is small, sufficient data cannot be transmitted per unit time. In this case, if the video CODEC 13 functions to maintain the number of frames per unit time, the video CODEC 13 operates adaptively in an attempt to reduce the amount of information per frame, so that, for example, the quantization step size is larger. Changes to a value. Even if the amount of information per unit time is reduced in this way, if data cannot be transmitted without loss, frame loss occurs. That is, if the transmission speed of the moving image information is low, the moving image has a jerky movement.

From these facts, in order to maintain a smooth screen when operating the camera, it is necessary to reduce the amount of moving image information and to reduce the amount of moving image information more than the transmission capacity allocated to the moving image information. The entire system needs to be moved so that becomes smaller, and as a method therefor, the above-described treatment can be taken.

From such a viewpoint, as a method for preventing frame dropping or jerky movement during operation of the camera, other than the above-described four methods are conceivable. For example, a method of increasing the motion vector detection range of the video CODEC 13 or reducing the resolution of the original image by image processing can be used. Further, a more efficient method can be realized by appropriately combining two or more of the above-described methods.

FIG. 16 shows an example of processing on the camera operation side when the video camera operation amount is limited in accordance with the motion vector detection range of the video CODEC 13.

When a camera operation is input (decision 30)
1 is YES), it is checked whether the operation target at that time is remote (the partner terminal side) (decision 30).
2). When the result of the determination 302 is YES, the above-mentioned start request command (START ACTION Request) specifying the content of the operation at that time is transmitted (process 303), and the process waits until the camera operation at that time is completed (determination 304). .

When the camera operation is completed and the result of determination 304 is YES, the above-mentioned stop request command (STOP ACTION Re
quest) (process 305), and the camera operation process at this time ends.

If the operation target is local (own terminal side) and the result of determination 302 is NO, the size of the motion vector detection range on the data transmission side of the video CODEC 13 at that time is read out, and The operation amount having a value corresponding to the operation amount is designated to the video camera control unit 16 (processing 3
06) In this state, until the camera operation is completed, an operation signal representing the content of the operation at that time is transmitted to the video camera control unit 16 (process 307, N in decision 308).
O loop).

When the camera operation is completed and the result of determination 308 is YES, the operation amount set in the video camera control unit 16 is returned to the original value (process 309), and this operation ends.

FIG. 17 shows an example of processing when a camera operation is requested from the partner terminal in this case.

When a start request command is received (decision 40)
1 is YES), the video CODEC 13 at that time
Of the motion vector detection range of the data transmission side of the data transmission side, and specifies an operation amount of a value corresponding to the size to the video camera control unit 16 (process 402), and receives a stop request command in that state. Up to the video camera controller 16
Then, an operation signal representing the operation content designated at that time is transmitted (step 403, NO loop of judgment 404).

When the stop request command is received and the result of determination 404 is YES, the coasting displacement dL corresponding to the delay time dT stored at that time is calculated (process 405). The operation information for operating the coast displacement amount dL in the direction opposite to the camera operation direction is sent to the video camera control unit 16 to correct the coast displacement amount dL (process 406).

Then, the operation amount set in the video camera control section 16 is returned to the original value (process 407), and this operation ends.

As described above, since the operation amount of the video camera is limited in accordance with the motion vector detection range of the video CODEC 13, the moving image component appearing in the moving image information can be limited, and the data amount of the moving image information can be reduced. Since it is possible to reduce the number of frames, it is possible to prevent a frame from being dropped or a jerky movement during operation of the camera.

FIG. 18 shows an example of processing on the camera operation side when the video camera operation amount is limited in accordance with the transmission speed (transmission capacity) assigned to the moving image information.

When a camera operation is input (decision 50)
1 is YES), it is checked whether the operation target at that time is remote (the partner terminal side) (decision 50).
2). When the result of determination 502 is YES, the above-described start request command (START ACTION Request) specifying the operation content at that time is transmitted (process 503), and the process waits until the camera operation at that time ends (determination 504). .

When the camera operation is completed and the result of determination 504 is YES, the above-described stop request command (STOP ACTION Re
quest) (process 505), and the camera operation process at this time ends.

If the operation target is local (own terminal side) and the result of determination 502 is NO, the operation is performed in accordance with the transmission speed assigned to the moving image information at that time.
The operation amount of the video camera device 12 per unit time is calculated, and the calculated operation amount is designated to the video camera control unit 16 (processing 506). In this state, the video camera control unit 16 is operated until the camera operation ends. Then, an operation signal representing the operation content at that time is transmitted (step 507, NO loop of judgment 508).

When the camera operation is completed and the result of determination 508 is YES, the operation amount set in the video camera control unit 16 is returned to the original value (process 509), and this operation ends.

FIG. 19 shows an example of processing when a camera operation is requested from the partner terminal in this case.

When the start request command is received (decision 60)
1 is YES), the operation amount of the video camera device 12 per unit time is calculated according to the transmission speed assigned to the moving image information at that time, and the calculated operation amount is sent to the video camera control unit 16. In this state, an operation signal representing the content of the operation specified at that time is sent to the video camera control unit 16 until the stop request command is received (processing 603, judgment 604).
NO loop).

When the stop request command is received and the result of determination 604 is YES, the coasting displacement dL corresponding to the delay time dT stored at that time is calculated (process 605). The operation information for operating the coast displacement amount dL in the direction opposite to the camera operation direction is sent to the video camera control unit 16 to correct the coast displacement amount dL (process 606).

Then, the operation amount set in the video camera control section 16 is returned to the original value (step 607), and this operation is terminated.

As described above, since the operation amount of the video camera is limited in accordance with the transmission speed assigned to the moving image information, the moving image component appearing in the moving image information can be limited. Since the data amount of information can be reduced, the transmission speed of the moving image information at that time,
The moving image information can be transmitted appropriately, and as a result, dropping of frames or jerky movement of the screen during operation of the camera can be prevented.

FIG. 20 shows an example of processing on the camera operating side when the value of the transmission speed assigned to the moving image information is increased.

When the camera operation is input (decision 70)
1 is YES), it is checked whether the operation target at that time is remote (the partner terminal side) (decision 70).
2). If the result of decision 702 is YES, the above-mentioned start request command (START ACTION Request) designating the content of the operation at that time is transmitted (process 703), and the process waits until the camera operation at that time ends (decision 704). .

When the camera operation is completed and the result of determination 704 is YES, the above-described stop request command (STOP ACTION Re
quest) (process 705), and the camera operation process at this time ends.

If the operation target is local (own terminal side) and the result of determination 702 is NO, the transmission speed assigned to the audio information, moving image information, and The assignment is changed so that the information has a larger value (step 706). This change of the transmission rate assignment is set in the own terminal and specified by the BAS command to the partner terminal.

In this state, an operation signal representing the operation content at that time is transmitted to the video camera control section 16 until the camera operation is completed (processing 707, NO loop of judgment 708).

When the operation of the camera is completed and the result of decision 708 is YES, the status of the transmission speed allocation to the audio information, moving image information and general-purpose data at that time is returned to the original state (step 709). This operation ends.
Here, the return of the transmission rate assignment is performed in the same manner as described above.
Designated to the partner terminal by the BAS command.

FIG. 21 shows a processing example when a camera operation is requested from the partner terminal in this case.

When the start request command is received (decision 80)
1 is YES), the audio information, video information,
And the transmission rate assigned to general-purpose data,
The allocation is changed so that the moving image information has a larger value (process 802). This change of the transmission rate assignment is set in the own terminal and specified by the BAS command to the partner terminal.

In this state, an operation signal indicating the operation content designated at that time is sent to the video camera controller 16 until the stop request command is received (Process 8).
03, NO loop of decision 804).

When the stop request command is received and the result of determination 804 is YES, the coasting displacement dL corresponding to the delay time dT stored at that time is calculated (process 805). The operation information for operating the coast displacement amount dL in the direction opposite to the camera operation direction is sent to the video camera control unit 16 to correct the coast displacement amount dL (process 806).

The sound information, the moving picture information,
Then, the state of the transmission rate allocation to the general-purpose data is returned to the original state (process 807), and this operation ends. Here, the return of the transmission rate assignment is specified to the partner terminal by the BAS command in the same manner as described above.

As described above, since the size of the transmission rate assigned to the moving picture information is increased, the moving picture information can be appropriately transmitted at the transmission rate of the moving picture information at that time. In addition, it is possible to prevent a frame from being dropped or a jerky movement during operation of the camera.

In the above-described embodiment, when the camera is operated, the correction operation is performed in the direction opposite to the operation direction by the coast displacement amount dL corresponding to the delay time dT, so that the screen desired by the user can be operated once. However, for example, if the content of the screen viewed by the user at the time of operating the camera is changed to a mode in which the time lag corresponding to the delay time dT is eliminated, the above-described correction operation is unnecessary. Become.

For example, as shown in FIG. 22A, while the entire screen of the received image is an area R1, the display control unit 14 displays the contents of a partial area R2 at the center of the area R1. 15, a margin area is provided on the display screen in advance. Here, the region R1
Is equivalent to a virtual screen space of a video ram (VRAM) for storing all data of one screen of a received image, and a region R
2 corresponds to an actual screen space displayed on the video monitor device 15 by the display control unit 14.

When a tilt-up operation is designated, for example, as shown in FIG. 13B, the region R2 is moved in the operation direction (in this case, the displacement amount corresponding to the delay time dT).
The real screen space is set in a state (R2 ′) shifted upward (upward), so that the user can operate the camera while watching the screen after the delay time dT.

When the user completes the camera operation, the area R2 returns to the original state as shown in FIG. At this time, immediately after the start of the operation, the region R2 is gradually changed to the state of the region R2 ′ according to the screen change rate due to the camera operation, and immediately after the operation is stopped, the region R2 is changed according to the screen change rate due to the camera operation. Region R2 ′ so that the region R2 ′ gradually returns to the state of the region R2.
When the real screen space to be cut out is changed, the screen seen by the user is smoothly continuous, so that the operability is further improved.

In the case of the tilt-down operation, the region R
2 'is set in the mode shown in FIG. In the case of the pan left operation, the region R2 'is set in the mode shown in FIG. In the case of the panlight operation, the region R2 'is set in the mode shown in FIG. Also,
In the case of the zoom-in operation, the region R2 'is set to the mode shown in FIG. In the case of zoom out operation,
The region R2 'is set in the mode shown in FIG.

FIG. 23 shows an example of processing on the camera operation side in this case.

When a camera operation is input (judgment 90)
1 is YES), it is checked whether the operation target at that time is remote (the partner terminal side) (decision 90).
2). When the result of determination 902 is YES, as described above, the display area is gradually changed to the final position according to the operation content at that time (step 90).
3), the above-described start request command (START ACTION Requests) specifying the operation content at that time.
t) (process 904), and waits until the camera operation at that time ends (determination 905).

When the camera operation is completed and the result of decision 905 is YES, the above-described stop request command (STOP ACTION Re
quest) (process 906), and gradually moves the display area to the original position (process 907).
At this time, the camera operation processing ends.

If the operation target is local (own terminal side) and the result of decision 902 is NO, the value of the maximum quantization step size on the data transmission side of the video CODEC 13 is changed to a predetermined large value. (Process 90
8) An operation command corresponding to the operation content at that time is output to the video camera control unit 16 (process 909), and in this state, the process waits until the camera operation is completed (decision 910).
NO loop).

When the camera operation is completed and the result of decision 910 is YES, an instruction to stop the operation is issued to the video camera control unit 16 (process 911), and the video CODEC 13
Then, the value of the maximum quantization step size of the transmitting side is returned to the original value (process 912), and this operation ends.

FIG. 24 shows an example of processing when a camera operation is requested from the partner terminal in this case.

When the start request command is received (decision 10
01 is YES), the value of the quantization step size maximum value on the data transmission side of the video CODEC 13 is changed to a predetermined large value (processing 1002), and the video camera control unit 16 is operated at that time. (Operation 909), and waits until a stop request command is received (NO loop in decision 1004).

When the stop request command is received and the result of determination 1004 is YES, a command to stop the operation is issued to the video camera controller 16 (step 100).
5), the value of the maximum quantization step size on the transmission side of the video CODEC 13 is returned to the original value (Process 100)
6), this operation ends.

In the above-described embodiment, the camera operation amount per unit time is constant. As described above, when the camera operation amount per unit time is constant, the operation of the user is likely to be monotonous, and an erroneous operation may occur.

Therefore, another embodiment of the present invention, in which the user can operate the camera in a fast or slow motion so that the user can operate the camera in an arbitrary mode, will be described below.

In this embodiment, the operation keys for operating the camera output, for example, operation signals of four levels. This can be realized, for example, by applying a function switch element whose resistance value changes according to pressure to the operation key.

FIGS. 25A to 25C show examples of video camera remote control signals exchanged in this embodiment.

The video camera remote operation signal includes a start request command (START ACTION Requests) for requesting the start of operation of the video camera device 12.
tNL (NL is a non-linear meaning; the same applies hereinafter)), a continuation request command (CONTINU
E ACTION Request NL), and
There is a stop request command (STOP ACTION Request NL) for requesting a stop of the operation under execution.

The start request command is composed of three bytes of data, as shown in FIG. 17A, and the first byte indicates that the video camera remote operation signal is a start request command. A predetermined bit pattern (for example, 01H) is set, the operation type data representing the operation content requested to be started is set in the second byte, and the displacement amount per unit time (in the operation direction) in the operation direction is set in the third byte. Hereinafter, displacement level data representing displacement level) is set.

The continuation request command is composed of three bytes of data, as shown in FIG. 17B, and the first byte indicates that the video camera remote operation signal is a continuation request command. A predetermined bit pattern (for example, 03H) is set, operation type data indicating the content of an operation continuously requested is set in the second byte, and displacement level data indicating a displacement level in the operation direction is set in the third byte. Is set.

The stop request command is composed of 2 bytes of data, as shown in FIG. 17C. The first byte indicates that the video camera remote operation signal is a stop request command. A predetermined bit pattern (for example, 06H) is set, and in the second byte, operation type data indicating the content of the operation requested to be stopped is set.

Here, the eighth and seventh bits of the displacement level data are data PA indicating the displacement level of the pan operation in four stages.
L, and the sixth and fifth bits of the displacement level data are data TA indicating the displacement level of the tilt operation in four stages.
L, and the fourth and third bits of the displacement level data are data ZA indicating the displacement level of the zoom operation in four stages.
L, and the second and first bits of the displacement level data are assigned to data FAL indicating the displacement level of the focus operation in four stages.

For example, as shown in FIG. 26, when the user displays an image received from the partner terminal on the video monitor device 15 of the terminal, the camera of the video camera device 12 of the partner terminal looks at the display screen. When the angle is operated, at the time of the operation input, the system control unit 1 starts a start request command (START ACTION Req.) Specifying a displacement level corresponding to the operation pressure of the user at that time.
estNL) to the partner terminal.

As a result, the system control unit 1 of the partner terminal
Outputs a camera operation signal to the video camera controller 16 in a state where the displacement level specified at that time is specified,
As a result, the camera angle of the video camera device 12 of the partner terminal gradually changes in a speed mode corresponding to the displacement level.
The screen displayed on 5 also changes gradually.

Thereafter, when the user appropriately changes the pressure of the operation key during operation, at that time, the system control unit 1 causes the continuation request command (CONTINUE ACTION Reaction) specifying the displacement level corresponding to the operation pressure.
request NL). As a result, the rate of change of the screen displayed on the video monitor device 15 is appropriately switched according to the displacement level of the user.

Then, when the screen displayed on the video monitor device 15 becomes the desired camera angle of the user and the user stops the camera operation, the system control unit 1 at that point in time issues a stop request command (START). A
CANCE Request NL).

As a result, the system control unit 1 of the partner terminal
Stops the output of the camera operation signal to the video camera control unit 16 and then calculates a displacement dL which is traced back from the time when the stop request command is received by a delay time dT, and calculates the displacement dL by the displacement dL. Operate in the opposite direction.

FIGS. 27 and 28 show an example of processing on the camera operation side in this case.

When a camera operation is input (judgment 11)
01 is YES), it is checked whether the operation target at that time is remote (the partner terminal side) (decision 110)
2). When the result of determination 1102 is YES, the above-described start request command (START A) specifying the displacement level of the operation key at that time and the operation content at that time
Ction Request NL) (Process 1)
103), the displacement level at that time is stored (Process 11)
04).

Then, it waits until the camera operation at that time ends or the displacement level changes (decision 1).
105, 1106 NO loop). When the pressure at which the user presses the operation key changes and the displacement level changes and the result of determination 1106 is YES, the displacement level of the operation key at that time and the continuation described above specifying the operation content at that time Request command (CONTINUE ACT
ION Request NL) (Process 110
7) Return to the process 1104 to store the displacement level at that time and execute the subsequent processes.

When the camera operation is completed and the result of determination 1105 is YES, the above-mentioned stop request command (STOP ACTION R) designating the operation content at that time.
request NL) (process 1108), and the camera operation process at this time ends.

If the operation target is local (own terminal side) and the result of determination 1102 is NO, the displacement level of the operation key at that time is stored (step 110).
9) Change the maximum value of the quantization step size on the data transmission side of the video CODEC 13 to a predetermined large value corresponding to the displacement level at that time (step 111).
0), a camera operation signal corresponding to the operation content at that time is output to the video camera control unit 16 in a state where the displacement level stored at that time is specified (processing 111).
1) In that state, the camera operation ends, or
Wait until the displacement level changes (decisions 1112, 111
3 NO loop).

In the case where the pressure at which the user presses the operation key changes and the displacement level changes, the result of decision 1113 is YE
When it becomes S, the process returns to the process 1109, the displacement level at that time is stored, and the subsequent processes are executed.

When the camera operation is completed and the result of determination 1112 is YES, the video camera control unit 16
Command operation stop (process 1114), and video CODEC
In step 1115, the maximum value of the quantization step size of the transmission side 13 is returned to the original value (processing 1115), and this operation ends.

FIG. 29 shows an example of processing when a camera operation is requested from the partner terminal in this case.

When the start request command is received (decision 12)
01 is YES), after saving the operation content and displacement level of the command received at that time (step 120
2) The value of the maximum quantization step size on the data transmission side of the video CODEC 13 is changed to a predetermined large value corresponding to the displacement level at that time (process 1203). An operation command corresponding to the displacement level at the time and the operation content is output (step 120).
4) The timer is started (process 1205), and in this state, the process waits until a stop request command is received or a continuation request command is received (NO loop of determinations 1206 and 1207).

When the continuation request command is received and the result of determination 1207 is YES, the timer is stopped (step 1208), and the time value of the timer at that time is updated to the latest (previously saved) operation signal (operation 1208). The processing is stored in association with the operation content and displacement level pair (processing 1209) and processing 12
Returning to step 02, the contents of the newly received command are stored, and the subsequent processing is executed.

When the stop request command is received and the result of determination 1206 is YES, an instruction to stop the operation is issued to the video camera controller 16 (step 121).
0), the timer is stopped (process 1211), and the count value of the timer at that time is stored in association with the latest operation signal (process 1212).

The set of the operation signal and the time value stored up to that point is traced back from the latest one, and the operation amount by the operation signal received from the present time to the time before the delay time dT, that is, the coasting in the delay time dT. The running displacement dL is calculated (step 1213).

Next, the time required for the correction operation (correction operation time) is calculated by dividing the coasting displacement amount dL by the finally set displacement level (processing 1214). Then, an operation command corresponding to the finally set displacement level and the content of the correction operation is output (process 1215), and the process waits until the correction operation time elapses (process 1216).

When this correction operation has been completed, the video camera controller 16 is instructed to stop the operation (step 121).
7), the value of the maximum quantization step size on the transmission side of the video CODEC 13 is restored to the original value (processing 121).
8), end this operation.

By the way, in the above-described embodiment, the remote operation of the video camera device 12 of the partner terminal is performed while displaying the received image to the user, but only the camera angle of the video camera device 12 is determined. If so, there is no need to display the video and perform the remote operation.

For example, if a still image for one screen of the display screen at the time of the start of the camera operation is formed and the user operates the camera while showing the still image to the user, during the camera operation, Since the image quality of the moving image information received on the camera operation side does not matter, it is not necessary to control the constant of the moving image CODEC 13 on the camera operated side, and the control is simplified. In addition, since the camera operation amount may be specified for the partner terminal in a state where the final operation content is determined, the control becomes easier.

Here, in this case, an example of the camera operation signal transmitted to the partner terminal is shown in FIG. In this case, the camera operation signal includes a start request command (START
ACTION Request MV (MV is a displacement amount)).

The start request command is composed of 4 bytes of data. The first byte has a predetermined bit pattern (for example, 07H) for indicating that the video camera remote operation signal is a start request command. Is set in the second byte, operation information related to the pan operation is set in the third byte, and operation information related to the zoom operation is set in the fourth byte.

Here, the eighth bit of the second byte is assigned to a bit P for designating that there is a pan operation, and the seventh bit is a bit PD (data 0 is left Data 1 is assigned to the right), and the sixth to first bits are assigned to data PMV representing the operation amount of the pan operation.

The eighth bit of the third byte is assigned to bit T for designating that there is a tilt operation, and the seventh bit is bit TD (data 0 is lower; Data 1 is assigned to (upper), and the sixth to first bits are assigned to data TMV representing the operation amount of the tilt operation.

The eighth bit of the fourth byte is assigned to bit Z for designating that there is a zoom operation, and the seventh bit is bit ZD (data 0 is out; Data 1 is assigned to IN), and the sixth to first bits are assigned to data ZMV representing the operation amount of the zoom operation.

FIGS. 31 and 32 show an example of processing on the camera operating side in this case. In this case, FIG.
In the same manner as shown in (a), a display control unit 1 stores a virtual screen space for storing all data for one screen of a still image.
4 sets a small area of the real screen space to be displayed on the video monitor device 15 so that the display screen can be changed in the operation direction of the user.

When a camera operation is input (decision 13)
01 is YES), it is checked whether the operation target at that time is remote (the partner terminal side) (decision 130).
2). If the result of determination 1302 is YES, a still image of the display screen of the received image at that time is created and saved in a predetermined still image data area (process 1303).
The display content of the display controller 14 is switched to the still image data area, and the still image formed at that time is displayed on the video monitor device 15 (processing 1304).

At the same time, an operation of sampling the key operation history of the user's camera operation (hereinafter referred to as key operation history sampling) is started (step 130).
5). Then, in response to the user's key operation, the display range of the screen displayed on the video monitor device 15 is moved in the operation direction (process 1306), and the process 1306 is terminated when the user's key operation ends or It monitors whether the display range deviates from the virtual screen space by a predetermined range (for example, １ ／) or more (decisions 1307, 130).
8). Here, when the display range deviates outside the virtual screen space, the portion of the display range not included in the virtual screen space is filled with solid blue (blue back) or a uniform color of any color. Is displayed with.

When the user's key operation proceeds and the display area deviates from the virtual screen space by a predetermined range or more,
When the result of the determination 1308 is YES, the key operation history sampling is stopped at that time (step 130).
9) Based on the sampling result of the key operation history up to that time, the final movement amount of the display range at that time is calculated,
The above-described start request command (START ACTION Request Request) in which the movement amount and the key operation content are set
st MV) is sent (process 1310), and key operation history sampling is started to obtain a history of subsequent user operations (process 1311).

In this state, the process waits until the content of the moving image information received from the partner terminal becomes the screen content at the time when the camera operation corresponding to the operation amount designated by the start request command is completed (process 1312). When the time has elapsed, a still image of the reception screen at that time is created and stored in a still image data area different from that at the start of the operation (process 1313).

At that time, the key operation history sampling is stopped (process 1314), and the moving amount of the display range at that time is calculated based on the key operation history sampling result up to that time (process 1315).

Next, by switching the content displayed by the display control unit 14 to the still image data area saved in the processing 1313, the display screen is switched to a new screen (processing 1316), and the display range calculated in the processing 1315 is changed. The display range is moved from the initial position in accordance with the amount of movement (step 1317), and the display screen is updated to the contents of the user operation until immediately thereafter. Next, returning to the process 1305,
Execute the subsequent processing.

If the key operation by the user is completed and the result of determination 1307 is YES, the key operation history sampling is stopped at that time (step 1318).
The above-described start request command (START ACTION Request MV) in which the movement amount of the display range at that time is calculated based on the sampling result of the key operation history up to that time, and the movement amount and the key operation content are set.
(Process 1319), and waits until the content of the moving image information received from the partner terminal becomes the screen content at the time when the camera operation corresponding to the operation amount designated by the start request command is completed (process 1319). 1320) When the waiting time has elapsed, the display state of the display control unit 14 is returned to the moving image display state 1321).

When the operation target is local (own terminal side) and the result of determination 1302 is NO, the value of the maximum quantization step size on the data transmission side of the video CODEC 13 is changed to a predetermined large value. At the same time (process 1322), a camera operation signal corresponding to the operation content at that time is output to the video camera control unit 16 (process 1323), and in this state, the process waits until the camera operation ends (NO in decision 1324). loop).

When the camera operation ends and the result of decision 1324 is YES, the video camera control unit 16
Command operation stop (process 1325), video CODEC
The value of the maximum quantization step size on the transmitting side of No. 13 is returned to the original value (process 1326), and this operation ends.

Thus, when operating the camera angle of the video camera of the partner terminal, the user can operate the camera while referring to the still screen. In this case, two still image data areas may be prepared, and the still image data created alternately may be stored in these two still image data areas.

FIG. 33 shows an example of processing when a camera operation is requested from the partner terminal in this case.

When the start request command is received (decision 14)
01 is YES), the operation time of the camera operation of the specified operation content is calculated based on the operation amount of the start request command received at that time (process 1402).

Next, an operation command corresponding to the operation content designated at that time is output to the video camera control section 16 (step 1403), and the operation waits until the operation time calculated in the step 1402 elapses (decision 1404). NO loop).

When the operation time has elapsed and the result of determination 1404 is YES, a command to stop the camera operation is issued to the video camera operation unit 16 (process 1405), and this operation ends.

In this embodiment, a still image of the received image at the start of the camera operation is formed, the still image is displayed, and the same screen as when the camera is actually operated is displayed according to the camera operation. Although the display area of the still image is controlled to be displayed, as described above, when the display area deviates from the saved still image area, the display screen needs to be updated, and the user greatly When attempting to operate the camera, the display screen is frequently updated, which may affect the overhead processing time.

To eliminate such inconvenience, for example, as shown in FIG. 34 (a), a conference scene at the start of the conference is photographed with the video camera device 12 zoomed out to the maximum. Then, the user operates the camera while forming the still image and displaying the still image. At this time, a screen that is expected to be displayed when the shooting range of the video camera device 12 changes according to a user's camera operation is indicated by a frame image PL,
The user can determine what screen is to be displayed by the user's operation.

[0210] When the user operation is completed, the display returns to the moving image display state corresponding to the designated camera operation as shown in FIG.

FIG. 35 shows an example of processing on the camera operation side in this case.

First, when the video conference communication is started, at the start time, the video camera device 12 of the partner terminal is zoomed out by the maximum amount (process 1501), and a still image of the received image at that time is formed. The still image is stored as an initial screen (process 1502).

Then, when a camera operation is input (the result of judgment 1503 is YES), it is checked whether or not the operation target at that time is remote (the partner terminal side) (judgment 1504). When the result of determination 1504 is YES, the above-described initial screen is displayed (process 1505), and a frame image PL having a size corresponding to the zoom ratio set at that time is displayed on the moving image display screen at that time. Superimpose display (composite display) at the display position (process 150)
6).

Thereafter, the display position of the frame image PL is appropriately moved or deformed according to the user's operation of the camera operation key (process 1507), and this process 1507 is repeated until the user ends the camera operation. Execute (decision 15)
08 NO loop).

When the user ends the camera operation,
If the result of step 08 is YES, the operation amount of the camera operation finally performed by the user is calculated based on the display position and size of the frame image PL at that time (step 1509), and the movement amount and the key operation content Start request command (START ACTION Requests)
tMV) (process 1510).

In this state, the process waits until the content of the moving image information received from the partner terminal becomes the screen content at the time when the camera operation corresponding to the operation amount specified by the start request command is completed (process 1511). When the time has elapsed, the display state of the display control unit 14 is returned to the moving image display state (step 1512).

When the operation target is local (own terminal side) and the result of determination 1502 is NO, the value of the maximum quantization step size on the data transmission side of the video CODEC 13 is changed to a predetermined large value. At the same time (process 1513), a camera operation signal corresponding to the operation content at that time is output to the video camera control unit 16 (process 1514), and in this state, the process waits until the camera operation ends (NO in determination 1515). loop).

When the camera operation ends and the result of determination 1515 is YES, the video camera control unit 16
Command operation stop (process 1516), video CODEC
Then, the maximum value of the quantization step size on the transmission side of No. 13 is returned to the original value (process 1517), and this operation ends.

Also in this case, similarly to the case of FIGS. 31 and 32, the operation history of the user is sampled,
It is necessary to calculate the operation amount based on the sampling result.

By the way, in the above embodiment, the video C
Although only the encoding / decoding processing time of the ODEC 13 is handled as the delay time, the camera operation can be corrected using the delay time including the transmission delay time. In this case, since the transmission delay time varies depending on the data amount and the transmission capacity of the moving image information, the measured representative value can be used. Video COD
The encoding / decoding processing time of EC13 is described in CCITT recommendation H.264. In addition to the calculation method defined in H.261, it can be determined by actually measuring. However, since the encoding / decoding processing time changes according to the data amount of the moving image information, for example, the encoding / decoding processing time corresponding to the representative data amount is measured and stored in advance, It is also possible to select an appropriate one from the stored encoding / decoding processing times according to the data amount at the time of actually operating the camera. However, if there is a difference in delay time that does not cause a problem in the human visual system, there may be no inconvenience even if only a representative value is used.

Further, in the above-described embodiment, the case where the present invention is applied to the video conference communication terminal has been described. However, other video communication devices, such as a video telephone terminal device and a still video conference The present invention can be similarly applied to a communication terminal device and the like.

[0222]

As described above, in the control method of the video conference communication terminal capable of remotely controlling the video camera device installed in the partner terminal, the memory for storing the received moving image data and the memory for storing the moving image data are stored in the memory. It includes display control means for cutting out data of a display area of a predetermined size out of the received moving image data and displaying a display screen of the cut out moving image data on a video monitor device, and always corresponds to an image at a predetermined position in the display area. When controlling the display control means so as to read out image data and remotely controlling the video camera device of the partner terminal to change the shooting range, a still screen corresponding to the display screen at the time of starting the remote control Data is formed, and during the operation period, the data input by the display control means is switched to the still screen data, and the remote control To the aspects of the display area is displaced from a central portion of the still picture data in accordance with the capacity to control the display control means so as to extract the still picture data,
When the operation period ends, the remote control amount of the video camera device from the start time to the end time of the operation period is transmitted to the partner terminal as a camera operation signal, and the display control means inputs the data of the memory. Since the state is restored, the user can operate the video camera of the partner terminal without being affected by the delay time associated with encoding and decoding of the moving image signal. Further, in the method for controlling a video conference communication terminal capable of remotely controlling a video camera device installed at a partner terminal, a memory for storing received moving image data, and a memory having a predetermined size among the received moving image data stored in the memory. Display control means for cutting out data in the display area and displaying a display screen of the cut-out moving image data on a video monitor device. The display control means always reads out image data corresponding to an image at a predetermined position in the display area. When controlling the means and remotely controlling the video camera device of the partner terminal to change the shooting range, still screen data corresponding to the display screen at the time of starting the remote operation is formed, and during the operation period, Switches the data input by the display control means to this still screen data, and raises the display area in accordance with the remote operation. In a mode displaced from the center of the still screen data, the display control means is controlled to cut out the still screen data, and when the display area deviates from the still screen, the content of the received image at that time The still screen data is updated, and when the operation period ends, the remote operation amount of the video camera device from the last update time of the still image data to the end of the operation period is transmitted to the partner terminal as a camera operation signal, Since the display control means is returned to the state in which the data in the memory is input, the user can operate the video camera of the partner terminal without being affected by the delay time associated with the encoding / decoding of the moving image signal. Is obtained. Further, in the control method of the video conference communication terminal capable of remotely controlling the video camera device installed in the partner terminal, the video conference device of the partner terminal is set to the applicable minimum magnification at the start of the video conference communication. Initial still screen data corresponding to the received moving image at that time is formed and stored, and thereafter, the received moving image is always displayed, and the video camera device of the partner terminal is changed to a mode in which the shooting range is changed. At the time of remote operation, the display screen of the above initial still screen data is displayed, and a frame image indicating a display frame corresponding to the shooting range of the video camera device of the partner terminal is displayed on the display screen, and the operation content is displayed. While moving the display area of the frame image accordingly, when the remote operation ends,
After transmitting the video camera operation signal corresponding to the display area of the frame image to the partner terminal at that time, the receiving screen is returned to the display state, so that the user can encode and decode the moving image signal. The effect that the video camera operation of the partner terminal can be performed without being affected by the delay time associated with.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a video conference communication device according to one embodiment of the present invention.

FIG. 2 is a schematic diagram illustrating an example of a frame configuration of a multi-frame.

FIG. 3 is a schematic diagram showing an example of a signal configuration of one frame.

FIG. 4 is a schematic diagram showing an example of a frame adjustment signal.

FIG. 5 is a schematic diagram showing an example of a bit rate assignment signal.

FIG. 6 is a schematic diagram for explaining a signal transmission order.

FIG. 7: CCITT Recommendation H. 242 is a time chart showing a general procedure example of H.242.

FIG. 8 is a schematic diagram showing an example of allocation of transmission speeds of audio data, general-purpose data, and moving image data when two information channels are reserved and frame mode data transfer is being performed.

FIG. 9 is a schematic diagram showing an example of a signal format of moving image information.

FIG. 10 is a schematic diagram showing another example of the signal format of the moving image information.

FIG. 11 is a schematic diagram showing an example of a data format of moving image information.

FIG. 12 is a schematic diagram showing an example of a camera operation signal.

FIG. 13 is a graph for explaining an example of camera operation correction.

FIG. 14 is a flowchart illustrating a processing example on the camera operation side.

FIG. 15 is a flowchart showing a processing example on the camera operated side.

FIG. 16 is a flowchart showing another example of the processing on the camera operation side.

FIG. 17 is a flowchart illustrating another example of processing on the camera operated side.

FIG. 18 is a flowchart showing still another example of processing on the camera operation side.

FIG. 19 is a flowchart showing still another example of the processing on the camera operated side.

FIG. 20 is a flowchart illustrating still another example of processing on the camera operation side.

FIG. 21 is a flowchart showing still another example of the processing on the camera operated side.

FIG. 22 is a schematic diagram for explaining another example of display at the time of operating the camera.

FIG. 23 is a flowchart illustrating another example of processing on the camera operation side.

FIG. 24 is a flowchart showing another example of processing on the camera operated side.

FIG. 25 is a flowchart showing another example of the camera operation signal.

FIG. 26 is a graph showing another example of the camera correction operation.

FIG. 27 is a flowchart showing a part of still another example of the processing on the camera operation side.

FIG. 28 is a flowchart showing another part of still another example of the processing on the camera operation side.

FIG. 29 is a flowchart showing still another example of processing on the camera operated side.

FIG. 30 is a schematic diagram showing still another example of the camera operation signal.

FIG. 31 is a flowchart showing a part of still another example of the processing on the camera operation side.

FIG. 32 is a flowchart showing another part of still another example of the processing on the camera operation side.

FIG. 33 is a flowchart showing still another example of the processing on the camera operated side.

FIG. 34 is a schematic view for explaining still another example of the camera operation.

FIG. 35 is a flowchart showing still another example of the processing on the camera operation side.

──────────────────────────────────────────────────続 き Continued on front page (58) Field surveyed (Int.Cl. ⁷ , DB name) H04N ^7/ 14-7/173

Claims (3)

(57) [Claims] 1. A method for controlling a video conference communication terminal capable of remotely controlling a video camera device installed in a partner terminal, comprising: a memory for storing received moving image data; Display control means for cutting out data of a display area of a predetermined size and displaying a display screen of the cut-out moving image data on a video monitor device, and always reading out image data corresponding to an image at a predetermined position in the display area. While controlling the display control means, when remotely controlling the video camera device of the partner terminal to change its shooting range, it forms still screen data corresponding to the display screen at the start of the remote operation, During the operation period, the data input by the display control means is switched to the still screen data, and the data is displayed in accordance with the remote operation contents. In a mode in which the area is displaced from the center of the still screen data, the display control means is controlled so as to cut out the still screen data, and when the operation period ends, a period from the start time to the end time of the operation period is controlled. A method for controlling a video conference communication terminal, comprising transmitting a remote operation amount of the video camera device to a partner terminal as a camera operation signal, and returning the display control means to a state in which data in the memory is input. 2. A method of controlling a video conference communication terminal capable of remotely controlling a video camera device installed in a partner terminal, comprising: a memory for storing received moving image data; Display control means for cutting out data of a display area of a predetermined size and displaying a display screen of the cut-out moving image data on a video monitor device, and always reading out image data corresponding to an image at a predetermined position in the display area. While controlling the display control means, when remotely controlling the video camera device of the partner terminal to change its shooting range, it forms still screen data corresponding to the display screen at the start of the remote operation, During the operation period, the data input by the display control means is switched to the still screen data, and the data is displayed in accordance with the remote operation. In a mode in which the area is displaced from the center of the still screen data, the display control means is controlled to cut out the still screen data, and when the display area deviates from the still screen, the reception at that time is performed. When the still screen data is updated to the content of the image and the operation period ends, the remote control amount of the video camera device from the last update time of the still image data to the end of the operation period is transmitted to the partner terminal as a camera operation signal. A method for controlling a video conference communication terminal, comprising transmitting the data and returning the display control means to a state in which data in the memory is input. 3. A method for controlling a video conference communication terminal capable of remotely controlling a video camera device installed in a partner terminal, wherein at the start of the video conference communication, the video camera device of the partner terminal is set to a minimum magnification applicable. In this state, the initial still screen data corresponding to the received moving image at that time is formed and saved.After that, the received moving image is always displayed, and the video camera device of the partner terminal is changed in its shooting range. When the remote operation is performed in such a mode, the display screen of the initial still screen data is displayed, and the display screen includes:
A frame image indicating a display frame corresponding to the shooting range of the video camera device of the partner terminal is synthesized and displayed, and the display area of the frame image is moved according to the operation content. A method for controlling a video conference communication terminal, comprising: transmitting a video camera operation signal corresponding to a display area of a frame image to a partner terminal, and then returning to a state in which a reception screen is displayed.