CN100584003C - Transmitting apparatus and method - Google Patents

Abstract

On the sending side, the background image and objects #1 to #3 are each sent at the transfer rate R/4. On the receiving side, an image composed of objects #1 to #3 and a background image is displayed at a specific spatial resolution and a specific temporal resolution. In this case, when the receiving side drags the object #1 at a certain time t1, the transmission of the background image and the objects #2 and #3 is stopped on the sending side. At this time, only object #1 is transmitted at the transmission rate R of the transmission path. Thus, an image with improved spatial resolution of dragged object #1 is displayed at the expense of temporal resolution of the image.

Description

Sending device and method

This case is a divisional application of an invention patent application with an application date of August 9, 2000, an application number of 00802048.5, and an invention title of "sending, receiving, and sending and receiving equipment and methods, recording media, and signals".

technical field

The present invention relates to a transmitting device, a transmitting method, a receiving device, a receiving method, a transmitting and receiving device, a transmitting and receiving method, a recording medium, and a signal. In particular, the present invention relates to a transmitting device, a transmitting method, a receiving device, a A receiving method, a transmitting and receiving device, a transmitting and receiving method, a recording medium, and a signal.

Background technique

For example, a related reference, Japanese Patent Laid-Open Publication Hei 10-112856, discloses an image transmission device. In this image sending device, a sender sends image data of a specific area of an image, and image data of other areas of the image are sent with different amounts of information corresponding to commands issued by a receiver. The receiving side displays images in an area containing a designated point of a higher spatial resolution (resolution in a spatial direction), and displays images in other areas of a lower spatial resolution.

In other words, when a sender sends image data to a receiver via a send path, the sender cannot send images at a data rate higher than that of the send path. Thus, when the receiving side displays a data on the actual time base, the sending side will send the image data to the receiving side at the sending rate corresponding to the sending path. Therefore, if the transmission rate is insufficient, the spatial resolution in the spatial direction of the image displayed on the receiving side deteriorates completely.

In order to solve this problem, in the image transmission device of Japanese Patent Laid-Open Publication Hei 10-112856, image data in one area of an image and image data in other areas of the image are transmitted with different information amounts. The image in this area containing a specified point and the images in other areas are displayed on the receiving side with high spatial resolution and low spatial resolution, respectively.

In other words, in the image transmission device of Japanese Laid-Open Publication Hei 10-112856, the improvement of the spatial resolution of a part that the user is to watch carefully comes at the expense of the spatial resolution (reduction) of the other part.

On the other hand, in the image transmission device of Japanese Patent Laid-Open No. Hei 10-112856, since the improvement of the spatial resolution of the part that the user wants to watch carefully is at the expense of the spatial resolution of other parts, the image that the user wants to watch carefully is improved. Partial spatial resolution is only improved at the expense of sacrificed spatial resolution.

In addition, when the transmission rate of the transmission path is very low, if the user wants to carefully watch the improvement of the spatial resolution of some parts at the expense of the spatial resolution of other parts, the spatial resolution of other parts will be seriously deteriorated. In the worst case, the user cannot view other parts clearly.

The present invention is made based on the above points of view. The object of the invention is to allow a better improvement of the spatial resolution of the image corresponding to the user's preference.

Summary of the invention

The first sending device of the present invention is a sending device for sending data to a receiving device, comprising: receiving means for receiving control information sent from the receiving device; control means for controlling the time direction of data sent to the receiving device corresponding to the control information , a resolution in at least two directions of a spatial direction and a level direction; and transmitting means for transmitting data corresponding to the control information having controlled the resolution in at least two directions to the receiving device.

The receiving device of the present invention is a receiving device that receives data transmitted from a transmitting device, and includes: transmitting means for transmitting control information to the transmitting device so as to control at least one of the time direction, space direction and level direction of the data corresponding to the control information resolution in two directions; receiving means for receiving data transmitted from the transmitting device, the resolution of which has been controlled in at least two directions of the data; and output means for outputting the data received by the receiving means.

The transmitting and receiving device of the present invention is a transmitting and receiving device having a transmitting device for transmitting data and a receiving device for receiving data, wherein the transmitting device includes: control information receiving means for receiving control information transmitted from the receiving device ; The control device is used to control the resolution in at least two directions of the time direction, space direction and level direction of the data sent to the receiving device corresponding to the control information; and the data sending device is used to send the corresponding data to the receiving device. The data that controls the resolution of at least two directions based on the control information; and wherein the receiving device includes: control information sending means for sending control information; data receiving means for receiving data sent from the sending device, the resolution of the data controlled in at least two directions of the data; and output means for outputting the data received by the data receiving means.

The second transmitting device of the present invention is a transmitting device for transmitting data to a receiving device, comprising: receiving means for receiving control information transmitted from the receiving device; sorting means for sorting the data corresponding to the control information; and transmitting means for The data is transmitted to the receiving device corresponding to the classification result of the data.

The first sending method of the present invention is a sending method for sending data to a receiving device, comprising the following steps: receiving control information sent from the receiving device; resolutions in at least two directions; and transmitting data having controlled resolutions in at least two directions corresponding to the control information to the receiving device.

The receiving method of the present invention is a receiving method for receiving data sent from a sending device, comprising the following steps: sending control information to the sending device so as to control at least two of the time direction, the space direction and the level direction of the data corresponding to the control information resolution in directions; receiving data sent by the transmitting device, the resolution of which is controlled in at least two directions of the data; and outputting the data received in the receiving step.

The transmitting and receiving method of the present invention is a transmitting and receiving method having processing of a transmitting device for transmitting data and processing of a receiving device for receiving data, wherein the processing of the transmitting device includes the steps of: receiving control information transmitted from the receiving device; corresponding to The control information controls the resolution of the data sent to the receiving device in at least two directions of the time direction, the space direction and the level direction; and the data that has controlled the resolution of at least two directions corresponding to the control information is sent to the receiving device ; wherein the processing of the receiving device includes the steps of: sending control information; receiving data sent from the sending device, the resolution of which is controlled in at least two directions of the data; and outputting the data received in the data receiving step.

The second transmitting method of the present invention is a transmitting method for transmitting data to a receiving device, comprising the steps of: receiving control information transmitted from the receiving device; classifying the data corresponding to the control information; and sending the classified result corresponding to the data to the receiving device send data.

The first recording medium of the present invention is a recording medium recording a program that causes a computer to execute transmission processing of transmitting data to a receiving device, the transmission processing including the steps of: receiving control information transmitted from the receiving device; controlling the resolution of the data transmitted to the receiving device in at least two directions of time direction, space direction and level direction; and transmitting the data having the resolution controlled in at least two directions corresponding to the control information to the receiving device.

The second recording medium of the present invention is a recording medium recording a program that causes a computer to execute receiving processing for receiving data transmitted from a transmitting device, the receiving process including the steps of: transmitting control information to the transmitting device so as to correspond to the control information control data resolution in at least two directions of a time direction, a space direction, and a level direction; receiving data transmitted from a transmitting device, the resolution of which has been controlled in at least two directions of data; and outputting The data received in the receive step.

A third recording medium of the present invention is a recording medium recording a program that causes a computer to execute transmission processing of a transmission device that transmits data and reception processing of a reception device that receives data, wherein the transmission processing of the transmission device includes the steps of: receiving Control information sent from the receiving device; corresponding to the control information to control the resolution of the data sent to the receiving device in at least two directions of the time direction, space direction and level direction; and controlling at least two resolutions that have been corresponding to the control information The data of the resolution of the direction is sent to the receiving device; wherein the receiving process of the receiving device includes the following steps: sending control information; receiving the data sent from the sending device, the resolution of the data has been controlled in at least two directions of the data; And output the data received on the data receiving step.

A fourth recording medium of the present invention is a recording medium recording a program that causes a computer to execute transmission processing of transmitting data to a receiving device, the transmission processing including the steps of: receiving control information transmitted from the receiving device; categorizing the data; and transmitting the data to a receiving device corresponding to the categorization of the data.

The first signal of the present invention is a signal containing a program that causes a computer to execute a transmission process of transmitting data to a receiving device, the transmission process including the steps of: receiving control information transmitted from the receiving device; controlling transmission corresponding to the control information resolution in at least two directions of time direction, space direction and level direction of the data to the receiving device; and transmitting the data having the resolution in the at least two directions controlled corresponding to the control information to the receiving device.

The second signal of the present invention is a signal containing a program that causes a computer to execute receiving processing for receiving data transmitted from a transmitting device, the receiving process including the steps of: transmitting control information to the transmitting device so as to control the data corresponding to the control information resolution of data in at least two directions of time direction, space direction, and level direction; receiving data transmitted from a transmitting device, the resolution of which has been controlled in at least two directions of data; and outputting in the receiving The data received in the step.

The third signal of the present invention is a signal containing a program that causes a computer to execute a transmission process of a transmission device that transmits data and a reception process of a reception device that receives data, wherein the transmission process of the transmission device includes the steps of: receiving The control information sent by the device; corresponding to the control information to control the resolution of the data sent to the receiving device in at least two directions of time direction, space direction and level direction; and controlling the resolution of at least two directions corresponding to the control information The resolution data is sent to the receiving device; wherein the receiving process of the receiving device includes the following steps: sending control information; receiving data sent from the sending device, the resolution of which is controlled in at least two directions of the data; and outputting in The data received on the data receiving step.

A fourth signal of the present invention is a signal containing a program that causes a computer to execute a transmission process of transmitting data to a receiving device, the transmission process including the steps of: receiving control information transmitted from the receiving device; data; and transmitting the data to a receiving device corresponding to the classification result of the data.

According to the first transmitting device, the first transmitting method, the first recording medium and the first signal of the present invention, control information transmitted from a receiving device is received. Corresponding to the control information, resolutions in at least two directions of time direction, space direction and level direction are controlled. Data for controlling resolution in at least two directions corresponding to the control information is transmitted to the receiving device.

According to the receiving device, the receiving method, the second recording medium and the second signal of the present invention, the control information is sent to the sending device, and the sending device controls at least two directions of the time direction, the space direction and the level direction corresponding to the control information. resolution. Furthermore, data corresponding to resolutions in at least two directions which the control information has controlled is transmitted, received and output from the transmitting device.

According to the transmitting and receiving device, the transmitting and receiving method, the third recording medium and the third signal of the present invention, the transmitting device receives the control information transmitted from the receiving device. Corresponding to the control information, resolutions in at least two directions of time direction, space direction and level direction are controlled. Data corresponding to resolutions in at least two directions controlled by the control information are transmitted to the receiving device, and furthermore, the receiving device transmits the control information to the transmitting device. Data corresponding to resolutions in at least two directions controlled by the control information are transmitted, received and output from the transmitting device.

According to the second transmitting device, the second transmitting method, the fourth recording medium and the fourth signal of the present invention, a control signal transmitted from a receiving device is received, and data is classified corresponding to the control information. Data is transmitted to the receiving device corresponding to the classification result of the data.

A brief description of the drawings

FIG. 1 is a schematic diagram showing a structural example of a transmission system of an embodiment of the present invention;

FIG. 2 is a detailed schematic diagram showing an example of a first structure of the transmission system shown in FIG. 1;

FIG. 3 is a block diagram showing a structural example of the transmitting device (terminal unit) 1 shown in FIG. 1;

FIG. 4 is a flow chart explaining the processing of the transmitting device shown in FIG. 3;

FIG. 5 is a block diagram showing a structural example of the receiving device (terminal unit) 2 shown in FIG. 1;

FIG. 6 is a flow chart explaining the processing of the receiving apparatus 2 shown in FIG. 5;

FIG. 7 is a block diagram showing a structural example of the transmission processing section 16 shown in FIG. 3;

FIG. 8 is a block diagram showing a structural example of the coding section 31 shown in FIG. 7;

9A, 9B and 9C are diagrams for explaining layered encoding/decoding;

FIG. 10 is a flow chart explaining the transmission processing of the transmission processing section 16 shown in FIG. 7;

FIG. 11 is a block diagram showing a structural example of the reception processing section 21 shown in FIG. 5;

FIG. 12 is a block diagram of a structural example of the decoding section 53 shown in FIG. 11;

FIG. 13 is a block diagram showing a structural example of the combination processing section 22 shown in FIG. 5;

FIG. 14 is a flowchart for explaining combination processing of the combination processing section 22 shown in FIG. 13;

15A, 15B and 15C are diagrams showing an image display example of the image output section 23 shown in FIG. 5;

16A and 16B are schematic diagrams explaining the relationship between the spatial resolution and the temporal resolution of images transmitted from the transmitting device 1 to the receiving device 2 shown in FIG. 1;

FIG. 17 is a block diagram showing a structural example of the object extraction section 14 shown in FIG. 3;

18A and 18B are diagrams explaining the processing of the start area dividing section 83 shown in FIG. 17;

19A, 19B, 19C and 19D are diagrams for explaining the processing of the region combining section 84 shown in FIG. 17;

FIG. 20 is a schematic diagram of the processing of the joined area processing section 85 and the separated area processing section 86 shown in FIG. 17;

FIG. 21 is a flowchart explaining object extraction processing of the object extraction section 4 shown in FIG. 17;

FIG. 22 is a flowchart explaining the details of the area stitching process at step S43 shown in FIG. 21;

FIG. 23 is a flow chart showing the details of the merged region processing in step S44 shown in FIG. 21;

FIG. 24 is a flowchart explaining the details of the separation area processing at step S44 shown in FIG. 21;

FIG. 25 is a block diagram showing a structural example of the control section 35 shown in FIG. 7;

Fig. 26 is a schematic diagram explaining a feature amount (feature amount) of an object;

FIG. 27 is a flow chart explaining the processing details of the control section 35 shown in FIG. 25;

FIG. 28 is a detailed schematic diagram showing an example of a second configuration of the transmission system shown in FIG. 1;

FIG. 29 is a block diagram showing a structural example of the transmitting device 1 shown in FIG. 28;

FIG. 30 is a flowchart explaining the processing of the transmission device 1 shown in FIG. 29;

FIG. 31 is a block diagram showing a structural example of the receiving apparatus 2 shown in FIG. 28;

FIG. 32 is a flowchart explaining the processing of the receiving apparatus 2 shown in FIG. 31;

FIG. 33 is a block diagram showing a structural example of the transmission processing section 1016 shown in FIG. 29;

FIG. 34 is a flowchart for explaining the transmission processing of the transmission processing section 1016 shown in FIG. 33;

Fig. 35 is a block diagram showing a structural example of a combination processing section shown in Fig. 31;

FIG. 36 is a flowchart explaining combining processing of the combining processing section shown in FIG. 35;

FIG. 37 is a schematic diagram showing the actual structure of the object extraction part 1014 shown in FIG. 29;

FIG. 38 is a flowchart explaining extraction processing of a moving object image or a still object image;

FIG. 39 is a flowchart explaining still area or active area determination processing;

40A, 40B, 40C, 40D and 40E are schematic views explaining the calculation method of the frame difference (inter-frame difference);

Fig. 41 is a flowchart explaining continuous click determination processing;

42A, 42B, 42C, 42D, 42E and 42F are schematic diagrams for explaining the object number allocation method;

Fig. 43 is a flowchart explaining stationary object combination processing;

Fig. 44 is a flowchart explaining active object combination processing;

Fig. 45 is a flowchart explaining object extraction processing;

46A, 46B, 46C, 46D and 46E are schematic diagrams for explaining the object extraction method;

FIG. 47 is a block diagram showing another structural example of the combination processing section 1022 shown in FIG. 31;

FIG. 48 is a block diagram showing another structural example of the transmitting device 1 shown in FIG. 28;

FIG. 49 is a block diagram showing a structural example of the variation determination and classification section 240 shown in FIG. 48;

50A and 50B are flow charts explaining the object of interest area change processing and object of interest area classification processing of the change determination and classification section 240;

FIG. 51 is a flowchart explaining interest change determination processing;

FIG. 52 is a schematic diagram of a detailed example of a third structure of the transmission system shown in FIG. 1;

FIG. 53 is a block diagram showing a structural example of the charging server 4 shown in FIG. 52;

FIG. 54 is a flow chart explaining the processing of the charging server 4 shown in FIG. 53;

Fig. 55 is a block diagram showing an example of the structure of a computer of an embodiment of the present invention.

The best form of carrying out the invention

FIG. 1 shows the structure of an image transmission system according to an embodiment of the present invention (the system is one entity as a logical combination of a plurality of devices, and it does not matter whether each device is provided in a rack or not).

The sending device consists of at least two terminal units 1 and 2 . As far as the relationship between the terminal units 1 and 2 is concerned, one is a transmitting device and the other is a receiving device. Images (image data) are sent from the sending device to the receiving device via the transmission path 3 .

According to the embodiment of the present invention, for example, it is assumed that the terminal unit 1 is a transmitting device, and the terminal device 2 is a receiving device, and image data is assumed to be transmitted and received between them. Hereinafter, the terminal unit 1 or 2 may be referred to as a transmitting device 1 or a receiving device 2, respectively.

In this case, the transmitting device 1 transmits image data to the receiving device 2 via the transmission path 3 . The reception device 2 receives image data from the transmission device 1 . The display of image data on the image output section 23 (see FIG. 5) will be described below. The image output section 23 is composed of, for example, a liquid crystal display or a CRT (cathode ray tube). Furthermore, the receiving section 2 transmits control information, which controls the spatial resolution in the spatial direction and the temporal resolution in the temporal direction of the image displayed by the image output section 23 , to the transmitting device 1 via the transmission path 3 .

The sending device 1 receives the control information from the receiving device 2, and controls the transmission of image data corresponding to the control information, so that the spatial resolution and time resolution of the image displayed by the receiving device 2 change when a predetermined condition is satisfied.

As the transmitting device 1 and the receiving device 2, a terminal unit such as a PHS (Personal Handyphone System) (trademark) unit and a portable terminal unit such as a portable telephone unit can be used. When the PHS unit is used, the transmission path 3 has a frequency band of 1895.1500 to 1905.9500 Mhz, and its transmission rate is 128 Kbps (bits per second).

FIG. 2 shows an example of a first configuration of the image transmission system shown in FIG. 1; here, portable terminal units such as a PHS unit and a portable telephone unit are used as the transmitting device 1 and the receiving device 2 shown in FIG.

According to the embodiment shown in FIG. 2, the transmission path 3 shown in FIG. 1 is composed of a radio base station 3-1 or 3-2 and a switching station 3-3. The switching station 3-3 is, for example, a telephone station connecting the wireless base stations 3-1 and 3-2. The wireless base station 3-1 or 3-2 transmits and receives a wireless signal to/from the terminal unit 1 or 2. Between terminal units 1 and 2, each of them can transmit a signal to the other and receive a signal from the other terminal via a transmission path 3 composed of wireless base stations 3-1 and 3-2, charging server 4, and the like. The type of base station 3-1 may be the same as or different from that of base station 3-2.

Referring to FIG. 2, the terminal unit 1 includes a camera section 1-1, a display section 1-2, a key section 1-3, a speaker section 1-4, and a microphone 1-5. The camera section 1-1 has an image pickup device and an optical system that can pick up moving images. The display section 1-2 can display characters, symbols, images, and the like. The key sections 1-3 are operated when inputting telephone numbers, characters, commands, and the like. Speakers 1-4 output sound. Microphones 1-5 input sound. Likewise, the terminal unit 2 includes a camera section 2-1, a display section 2-2, a key section 2-3, a speaker 2-4, and a microphone 2-5, which have the same functions as the camera section 1-1, the display section 1-2, The button part 1-3, the speaker 1-4 and the microphone 1-5 have the same structure.

Voice signals collected by the microphones 1-5 and 1-6 are sent and received between the terminal units 1 and 2. In addition, image data picked up by the camera sections 1-1 and 2-1 can be sent and received. Therefore, the display section 1 - 2 of the terminal unit 1 can display image data acquired by the camera section 2 - 1 of the terminal unit 2 . Also, the display section 2-2 of the terminal unit 2 can display image data acquired by the camera section 1-1 of the terminal unit 1.

In other words, the image data picked up by the camera section 1-1 of the sending device 1 and required information such as frame rate are sent to the Receive device 2. The receiving section 2 displays received image data (corresponding image) on a display section 2-1 composed of, for example, a liquid crystal display. On the other hand, the receiving device 2 transmits control information for controlling the spatial resolution and the temporal resolution of the image displayed on the display section 2-1 to the transmitting device 1 via the transmission path 3. The sending device 1 receives the control information from the receiving device 2 and controls the transmission of image data corresponding to the control information, so that the spatial resolution and temporal resolution of the image displayed by the receiving device 2 changes when a predetermined condition is met.

FIG. 3 below shows an example of the structure of the transmission device 1 shown in FIG. 2 .

The image input section 11 corresponds to the camera section 1-1 shown in FIG. 2 . The image input section 11 picks up a specific object and sends its image to the preprocessing section 12 . The preprocessing section 12 is composed of a background image extraction section 13 , an object extraction section 14 and an additional information calculation section 15 . The preprocessing section 12 performs preprocessing of the image input from the image input section 11 and supplies the synthesized image to the transmission processing section 16 .

In other words, the background image extraction section 13 extracts a so-called background image from the image supplied from the image input section 11 , and supplies the extracted background image to the transmission processing section 16 . Furthermore, the background image extracted by the background image extraction section 13 is also supplied to the object extraction section 14 and the additional information calculation section 15 .

As a method of extracting a background image, the frequency of occurrence of pixel values of a plurality of pixels at the same spatial position of a plurality of consecutive frames (for example, the current frame and the past 10 frames) is acquired. The pixel value with the highest frequency can be treated as background for that location. On the other hand, the average value (moving average value) of the pixel values at the same position in multiple frames can be obtained as the background image at the position.

The object extraction section 14 performs, for example, subtraction of the background image extracted by the background image extraction section 13 from the image supplied from the image input section 11, obtains a so-called foreground image, and supplies the foreground image to the transmission processing section 16 . When the image input by the image input section 11 contains a plurality of entities as the foreground, the object extraction section 14 extracts a foreground image corresponding to each entity and supplies the extracted foreground background to the transmission processing section 16 . On the other hand, the foreground image extracted by the object extraction section 14 is also supplied to the additional information calculation section 15 . Thereafter, the foreground image corresponding to each entity can be regarded as an object.

The additional information calculation section 15 detects a background image movement vector and an object movement vector. The background image movement vector represents the movement of the background image extracted by the background image extraction section 13 (the movement of the background image corresponding to the movement of the shooting direction of the image input section 11). The object movement vector represents the movement of the object extracted by the object extraction section 14 . In addition, the additional information calculation section 15 supplies the position information and the like of an object in one frame supplied from the object extraction section 14 as additional information to the transmission processing section 16 . In other words, when the object extraction section 14 extracts an object, the object extraction section 14 also extracts information related to the object, such as position information of the object, etc., and supplies the information to the additional information calculation section 15 . The additional information calculation section 15 also outputs the position information and the like as additional information.

The transmission processing section 16 multiplexes the background image extracted by the background image extraction section 13, the object extracted by the object extraction section 14, and the additional information acquired by the additional information calculation section 15, and synthesizes them via the transmission path 3 at their data rates. The multiplexed data is sent to the receiving device 2. The sending processing part 16 receives the control information sent from the receiving device 2 via the transmission path 3, and controls the sending of background images, objects and additional information corresponding to the control information, so that the spatial resolution and time resolution of the image displayed by the receiving device 2 The rate changes when predetermined conditions are met.

The processing of the sending device 1 shown in FIG. 3 will be briefly described below with reference to the flow chart shown in FIG. 4 .

The image output from the image input section 11 is supplied to the preprocessing section 12 . At step S1, the processing section 12 performs preprocessing of the image. In other words, at step S1, the background image extracting section 13 or the foreground image extracting section 14 respectively extracts a background image or an object from the image input by the image input section 11, and supplies the extracted background image or the extracted object to the sending process Part 16. Further, at step S1 , the additional information calculation section 15 obtains the above-mentioned additional information from the image input by the image input section 11 , and supplies the obtained additional information to the transmission processing section 16 . The transmission processing section 16 transmits the background image, foreground image, and additional information via the transmission path 3 . Thereafter, the flow returns to step S1, and the transmitting device 1 repeats similar processing.

FIG. 5 shows an example of the structure of the receiving device 2 shown in FIG. 2 .

The transmission device 1 transmits multiplexed data via the transmission path 3 . This multiplexed data is received by the reception processing section 21 . The reception processing section 21 separates the multiplexed data into background images, objects, and additional information, and supplies them to the combination processing section 22 .

The combination processing section 22 combines the background image, object, and additional information received from the reception processing section 21 into an original image, and supplies the synthesized image to the image output section 23 . The combination processing section 22 changes the spatial resolution and temporal resolution of the combined image corresponding to high-resolution information and the like to be described below.

The image output section 23 is composed of, for example, a liquid crystal display or a CRT. The image output section 23 displays the image output from the combination processing section 22 .

The control information input section 24 outputs the control information to the combination processing section 22 and the control information transmission section 25 . In other words, the control information input section 24 constitutes the key section 2-3. The control information input section 24 is composed of, for example, a pointing device such as a trackball. When the user designates a specific position of the image displayed by the image output part 23 with the control information input part 24, the control information input part 24 detects the position as a consideration point that the user is considering, and puts the coordinates of the consideration point into the control information, and output the resulting control information. On the other hand, the control information input section 24 is composed of, for example, a video camera. When the user takes and approves an image, the control information input section 24 detects a point that the user is considering as a considered point, puts the coordinates of the considered point into the control information, and outputs the synthesized control information. The control information input section 24 may be structured so that the user can directly input the spatial resolution and temporal resolution of the image displayed by the image output section 23 as control information.

When the control information transmission section 25 receives the control information from the control information input section 24 , the control information transmission section 25 transmits the control information to the transmission device 1 via the transmission path 3 .

The processing of the receiving device 2 shown in FIG. 5 will be briefly described below with reference to the flowchart shown in FIG. 6 .

In the reception device 2 , the reception processing section 21 receives the multiplexed data from the transmission device 1 via the transmission path 3 . In step S11, the reception processing section 21 performs a reception process of separating the multiplexed data into background images, objects and additional information. The reception processing section 21 supplies the combination processing section 22 with a background image, an object, and additional information as a result of the reception processing. In step S12 , the combination processing section 22 combines the background image, the object, and the additional information into an original image and supplies the composed image to the image output section 23 . The image output section 23 displays the combined image. Thereafter, the flow returns to step S11, and then the receiving device repeats similar processing.

In the receiving apparatus 2, when the user designates a point of consideration of an image displayed by the image output section 23 with the control information input section 24, the coordinates of the point of consideration are put into the control information. This synthesized control information is supplied to the control information transmission section 25 . The control information transmission section 25 transmits the control information to the transmission device 1 via the transmission path 3 . When the sending device 1 receives the above-mentioned control information, the sending processing part 16 controls the sending of background images, objects and additional information, so that the spatial resolution and temporal resolution of the image displayed by the receiving device 2 can meet the agreed conditions change from time to time. Thereafter, since the background image, object and additional information that have been controlled are transmitted from the transmitting device 1 to the receiving device 2, the image having changed spatial resolution and temporal resolution is displayed by the receiving device 2 when a predetermined condition is satisfied.

FIG. 7 shows a configuration example of the transmission processing section 16 of the transmission device 1 shown in FIG. 3 .

Background images, objects, and additional information are supplied from the preprocessing section 12 (see FIG. 3 ) to the encoding section 31 and the control section 35 . The encoding section 31 encodes a background image, an object, and additional information, and supplies the synthesized encoded data to a MUX (Multiplexer) 32 . The MUX 32 selects background images, objects and additional information received from the encoding section 31 under the control of the control section 35 and supplies the selected encoded data to the transmitting section 33 as multiplexed data. The transmitting section 33 modulates the multiplexed data received from the MUX 32 , and transmits the modulated data to the receiving device 2 via the transmission path 3 . The data amount calculation section 34 monitors the multiplexed data output from the MUX 32 to the transmission section 33 , calculates the data rate of the multiplexed data, and supplies the calculated data rate to the control section 35 .

The control section 35 controls the output of the multiplexed data of the MUX 32 so that the data rate of the data amount calculation section 34 does not exceed the transmission rate of the transmission path 3 . Further, the control section 35 receives control information transmitted from the receiving apparatus 2 via the transmission path 3, and controls selection of coded data of the MUX 32 corresponding to the control information.

FIG. 8 shows an example of the structure of the coding section 31 shown in FIG. 7 .

The background image is supplied to the difference calculation section 41B. The difference calculation section 41 subtracts the background image of the just-processed previous frame from the background image of the processed current frame (hereinafter, this frame may be referred to as a considered frame), and takes the difference data of the background image as the subtraction result It is supplied to the hierarchical encoding section 42B. The hierarchical encoding section 42B performs hierarchical encoding on the difference data of the background image received from the difference calculation section 41B, and supplies the encoded result to the storage section 43B. The storage section 43B temporarily stores the layered encoding result received from the layered encoding section 42 . The layered encoding result stored in the storage section 43B is supplied to the MUX 32 as encoded data of the background image (see FIG. 7 ).

The layered encoding result stored in the storage section 43B is supplied to the local decoder 44B. The local encoder 44B decodes the layered encoding result into an original background image, and supplies the decoded background image to the difference calculation section 41B. The background image decoded by the local decoder section 44B is used by the difference calculation section 41B. The difference calculation section 41B uses the decoded background image to obtain difference data of the background image of the next frame.

The objects are supplied to the difference calculation section 41F. Difference calculation section 41F, layered encoding section 42F, storage section 43F, and local decoder section 44F perform the same processing as difference calculation section 41B, layered encoding section 42B, storage section 43B, and local decoder 44B described above. Thus, in the same manner as the background image, the object is hierarchically coded and supplied to the MUX 32 (see FIG. 7). When there are a plurality of objects, the difference calculation section 41F, the hierarchical encoding section 42F, the storage section 43F, and the local decoding section 44F hierarchically encode the plurality of objects in the above-described manner.

Additional information is provided to the VLC (Variable Length Coding) section 45 . The VCL section 45 encodes the additional information according to the variable length code encoding method, and supplies the encoded additional information to the MUX 32 (see FIG. 7).

Layered encoding/decoding performed by the encoding section 31 shown in FIG. 8 will be described below with reference to FIG. 9. FIG.

It is assumed that an average value of four pixels of 2×2 pixels (horizontal×vertical) of the lower hierarchy is regarded as one pixel (pixel value) of the higher hierarchy, and it is assumed that one image is encoded in three hierarchical levels. In this case, for an image in the lowest hierarchical level considering 8×8 pixels as shown in FIG. 9(A), four pixels h00, h01, h02 of the upper left portion of 2×2 pixels are calculated and the mean m0 of h03. This average value is considered by m0 to be a pixel in the upper left portion of the second hierarchical level. The average value m1 of the four pixels h10, h11, h12, and h13 of the upper right portion of the lowest hierarchical level is calculated. The mean value m2 of the four pixels h20, h21, h22, and h23 of the lower left portion of the lowest hierarchical level is calculated. The average value m3 of the four pixels h30, h31, h32, and h33 of the upper right portion of the lowest hierarchical level is calculated. Mean values m1, m2, and m3 are regarded as pixels on the upper right portion, lower left portion, and upper right portion of the second hierarchical level, respectively. Furthermore, an average value q of four pixels m0, m1, m2, and m3 of 2×2 pixels of the second hierarchical level is calculated. The average value is regarded as a pixel of the third hierarchical level of the highest hierarchical level.

According to the hierarchical coding described above, the spatial resolution of the images in the highest hierarchical level is the lowest. The spatial resolution of an image is inversely proportional to the level of hierarchy. In other words, the spatial resolution of the images in the lowest hierarchical level is the highest.

The amount of data in the case where all pixels h00 to h03, h10 to h13, h20 to h23, h30 to h33, m0 to m3, and q are transmitted is larger than the lowest pixel transmitted only by pixels m0 to m3 and q of a higher hierarchical level. The amount of data in the case of hierarchical images.

Therefore, as shown in FIG. 9(B), the pixel q of the third hierarchical level is transmitted instead of the pixel m3 at the lower right position of the pixels m0 to m3 of the second hierarchical level, for example.

Furthermore, as shown in FIG. 9(C), the pixel m0 of the second hierarchical level is transmitted instead of, for example, the pixel h03 at the lower right position of the pixels m0 to m3 of the lowest hierarchical level. Likewise, the remaining pixels m1, m2 and q of the second hierarchical level are transmitted instead of the pixels h13, h23 and h33 of the first hierarchical level. Although pixel q is not a pixel of the second hierarchical level, it is transmitted instead of pixel m3 obtained directly from pixels h30 to h33. Thus, pixel q is sent instead of pixel m3 instead of pixel h33.

Thus, as shown in FIG. 9(C), the number of transmitted pixels becomes 4×4 pixels=16 pixels. The number of pixels transmitted is therefore the same as in the case of transmitting only pixels of the lowest hierarchical level shown in FIG. 9(A). In this case, the amount of data sent can be avoided from increasing.

The pixel m3 transmitted instead of the pixel q and the pixels h03 , h13 , h23 , and h33 transmitted instead of the pixels m0 to m3 can be decoded as follows.

In other words, since q is the average value of m0 to m3, the equation q=(m0+m1+m2+m3)/4 is satisfied. Therefore, the pixels of the second hierarchical level can be obtained (decoded) using the pixel q of the third hierarchical level and the pixels m0 to m2 of the second hierarchical level corresponding to the equation m3=4×q-(m0+m1+m2) m3.

Furthermore, since m0 is an average value of h00 to h03, the equation m0=(h00+h01+h02+h03)/4 is satisfied. Therefore, corresponding to the equation h03=4*m0-(h00+h01+h02), the pixel h03 of the first hierarchy level can be obtained with the pixel m0 of the second hierarchy level and the pixels h00 to h02 of the first hierarchy level . In the same way, h13, h23 and h33 can be obtained.

Therefore, a pixel not transmitted in a particular hierarchical level can be decoded with a pixel transmitted in that hierarchical level and a pixel of a hierarchical level one level above.

The transmission processing of the transmission processing section 16 shown in FIG. 7 will be described below with reference to the flowchart shown in FIG. 10 .

In step S21, in the transmission processing section 16, the control section 35 determines whether the receiving device 2 has transmitted a control signal. When the determined result of step S21 indicates that the control signal has not been transmitted from the receiving device 2 (ie, the control section 35 has not received the control information), the flow proceeds to step S22. The control section 35 controls the MUX 32 to select coded data of background images, objects, and additional information, and multiplex them so that the receiving device 2 can display an image with a normal time resolution (for example, a default time resolution).

When the normal temporal resolution is, for example, 30 frames/second, MUX 32 selects coded data of a background image, an object, and additional information, multiplexes the coded data, and outputs the multiplexed data so that the image is Displays the maximum time resolution of multiplexed data transmitted at the transmission rate of transmission path 3.

Actually, for example, in the case of performing hierarchical encoding by three hierarchical levels, when (only) the data of the third hierarchical level is transmitted at the transmission rate of the transmission path 3, the encoded data of the background image, object, and additional information are selected to display the image of the third level hierarchy. Therefore, in this case, the receiving device 2 displays an image with a spatial resolution (horizontal direction and vertical direction) of 1/4 (horizontal direction and vertical direction) and a temporal resolution of 30 frames/second which are 1/4 of the original image (first hierarchical level).

Thereafter, the flow advances to step S23. In step S23 , the transmission section 33 transmits the multiplexed data output by the MUX 32 via the transmission path 3 . Thereafter, the step returns to step S21.

When the determined result at step S21 indicates that the control signal has been transmitted from the receiving device 2 (ie, the control section 35 has received the control information), the flow proceeds to step S24. The control section 35 manipulates the control information input section 24 corresponding to the control signal, thereby approving the designated point of consideration. Thereafter, the flow advances to step S25.

In step S25, the control section 35 designates an area like a rectangle having a predetermined size around a consideration point (the rectangular frame has parallel sides in the frame horizontal direction and the vertical direction and the consideration point is on the center of gravity of the rectangular frame) as a priority range , in which priority is used to improve the spatial resolution and to detect background images, objects and additional information that make up the image in this priority range.

Thereafter, the flow proceeds to step S26. In step S26, the control section 35 controls the MUX 32 to select and multiplex the coded data of the background image, the object, and the additional information, and multiplexes the selected coded data so that the receiving device 2 can display with a higher temporal resolution. Images within this priority range.

In other words, when the control section 35 receives the control signal from the receiving device 2, the control section 35 controls the MUX 32 to improve the spatial resolution of images within the priority range at the expense of temporal resolution.

Therefore, the MUX 32 selects with priority the coded data of the background image, object, and additional information required to display the image of the third hierarchical level and the image of the second hierarchical level, multiplexes the selected coded data and outputs the multiplexed data. used data.

Furthermore, at step S26, the control section 35 controls the MUX 32 to insert positional information of the priority range, size, etc. into the additional information selected as the multiplexed data. Thereafter, the flow advances to step S23.

In step S23, the transmission section 33 transmits the multiplexed data output by the MUX 32 via the transmission path 3 as described above. Thereafter, the flow returns to step S21.

For the sake of simplicity, assume that on step S26, for an image of the third hierarchical level as an image in the non-priority range, the coded data of background information, objects and additional information of the image of the third hierarchical level are continuously selected and displayed, which is multiplied The case where the multiplexed data amount is larger than the data of the second hierarchical level of the image within the priority range at step S22. According to the embodiment of the present invention, if the image is displayed at 30 frames per second in the above manner, only the data of the third level is transmitted at the transmission rate of the transmission path 3 . This makes it impossible to transmit the multiplexed data obtained at step S26 to display images at 30 frames/second. Thus, in the extreme case of step S23, the multiplexed data obtained at step S26 is transmitted with a time resolution of 0 frame/second.

Therefore, in this case, for images within the priority range, the spatial resolution of each of the horizontal direction and the vertical direction of the image displayed by the receiving device 2 is 1/1 of the spatial resolution of the original image (that is, the first hierarchical level). 2. In other words, the reception device 2 displays an image (second hierarchical level) whose spatial resolution in each of the horizontal direction and vertical direction is twice the spatial resolution of the image in the third hierarchical level. However, its temporal resolution is 0 frames per second. In other words, the receiving device 2 displays a still image.

After transmitting the data of the second hierarchical level for the image within the priority range, when the determined result of step S21 indicates that it has been transmitted again from the receiving device 2 (that is, the user continuously operates the control information input section 24 and designates the same point of consideration) control signal, the flow proceeds to step S24. In step S24, the control section 35 recognizes the same point of consideration. Thereafter, the flow advances to step S25. In step S25, the control section 35 designates the same priority range. Thereafter, the flow proceeds to step S26.

In step S26, the control section 35 controls the MUX 32 to select the coded data of the background image, the object, and the additional information and multiplex the selected coded data, so that the receiving device 2 can display the images within the priority range with a higher spatial resolution. Image.

In this case, for the images within the priority range, the encoded data of background images, objects, and additional information of the third hierarchical level and the second hierarchical level are selected with priority. In this case, the coded data of the background image, object and additional information of the first hierarchical level are selected with priority and output as multiplexed data. Furthermore, at step S26, as described above, high-resolution information is inserted into the additional information. Thereafter, the flow advances to step S23. In step S23 , the transmission section 33 transmits the multiplexed data output by the MUX 32 via the transmission path 3 . Thereafter, the flow proceeds to step S21.

Therefore, in this case, for the images within the priority range, the receiving device 2 displays an image having the same spatial resolution as the original image (first hierarchical level). In other words, the reception device 2 displays an image having a spatial resolution (in each of the horizontal direction and the vertical direction) four times that of the image of the third hierarchical level. However, the temporal resolution is, for example, 0 frames/second. In other words, the receiving device 2 displays a still image.

Thus, when the user continuously operates the control information input section 24 and designates the same point of consideration, for images within the priority range containing the point of consideration, the spatial resolution is improved by priority transmission at the expense of the temporal resolution of the image. data. Thus, although the temporal resolution of the images deteriorates, the spatial resolution of the images within the priorities containing the consideration points is improved. Therefore, images within the priority range are displayed more clearly. In other words, the image of the portion considered by the user is displayed more clearly.

Therefore, the transmission of the image data is controlled so that the spatial resolution and the temporal resolution of the image within a certain priority range including a point of consideration are varied within the resolution range of the image data transmitted at the transmission rate of the transmission path 3 . In other words, the transmission of image data is controlled so that the spatial resolution of the image within the priority range is improved and the temporal resolution of the image is deteriorated. Thus, an improved spatial resolution and a worsened temporal resolution are obtained by means of the image data transmitted at the transmission rate of the transmission path 3 . As a result, the spatial resolution of the image displayed by the receiving device 2 can be further improved at a limited transmission rate.

FIG. 11 below shows a configuration example of the reception processing section 21 shown in FIG. 5 .

The reception section 51 receives the multiplexed data via the transmission path 3 . After the multiplexed data is demodulated, it is supplied to a DMUX (Demultiplexer) 52 . The DMUX 52 decomposes the multiplexed data received from the receiving section 51 into encoded data of background images, objects, and additional information, and supplies the decomposed data to the decoding section 53 . The decoding section 53 decodes the coded data of the background image, objects, and additional information into their original data and supplies to the combination processing section 22 (see FIG. 5).

In other words, FIG. 12 shows a structural example of the decoding section 53 shown in FIG. 11 .

The layered-encoded difference data, which is encoded data of the background image, is supplied to the adding device 61B. In addition, a background image of the previous frame which has been decoded and stored in the storage section 62B is also supplied to the adding means 61B. The adding means 61B adds the difference data of the background image and the background image of the previous frame received from the storage section 62B to decode a background image of a desired hierarchical level. This decoded background image is supplied to the storage section 62B. The storage section 62B stores the background image received from the adding device 61B. This decoded background image is supplied to the adding device 61B and the combination processing section 22 (see FIG. 5 ).

The hierarchically encoded difference data as object encoded data is supplied to the adder 61F. The addition device 61F and the storage section 62F perform similar processing to the addition device 61B and the storage section 62B. Thus, in the same manner as the background image, difference data of objects is decoded into objects of desired hierarchical levels, and then supplied to the combination processing section 22 (see FIG. 5 ). When there are a plurality of objects, the addition section 61F and the storage section 62F decode difference data of the plurality of objects (hierarchical decoding).

Additional information encoded with a variable length code is supplied to the reverse VLC section 63 as additional information encoded data. The reverse VLC section 63 decodes the additional information with variable length coding. The original additional information is thereby obtained, and supplied to the combination processing section 22 .

The local decoder 44B shown in FIG. 8 is constituted as an adding means 61B and a storage section 62B. Also, the local decoder 44F is constituted as an adding means 61F and a storage section 62F.

FIG. 13 shows a structural example of the combination processing section 22 shown in FIG. 5 .

The background image (see FIG. 1 ) output by the decoding section 53 is supplied to the background image writing section 71 . The object output by the decoding section 53 is supplied to the object writing section 72 . The additional information output by the decoding section 53 is supplied to the background image writing section 71 , the object writing section 72 and the combining section 77 .

The background image writing section 71 continuously writes the background image into the background image memory 73 . When an image is captured with the camera panned or tilted, the background image moves. At this point, the background image writing section locates the background image, and then writes the positioned background image into the background image memory 73 . Therefore, the background image memory 73 can store an image that is spatially wider than an image of one frame. The background image is positioned corresponding to a background image movement vector included in the additional information.

When the background image writing section 71 writes a background image with a high spatial resolution into the background image memory 73, the background image writing section 71 changes the value of the background image flag memory 74 corresponding to the pixels of the background image from 0 to 1. Address stores the value of the background image flag. When the background image writing section 71 writes a background image into the background image memory 73 , the background image writing section 71 checks the background image flag memory 74 . The background image writing section 71 does not write a background image with low spatial resolution on a pixel whose background image flag is 1 (indicating that the pixel stores a background image with high spatial resolution). Thus, basically, whenever a background image is supplied to the background image writing section 71 , it is written to the background image memory 73 . However, the background image writing section 71 does not write the background image with low spatial resolution on the pixels storing the background image with high spatial resolution. As a result, whenever a background image of high spatial resolution is supplied to the background background image writing section 71, the range of high spatial resolution is widened within the background image memory.

The object writing section 72 continuously writes the supplied objects onto the object memory 75 . When there are a plurality of objects, the object writing section 72 writes each object to the object memory 75 . When the object writing section 72 writes the same object (allocated with the same flag) to the object memory 75, the object writing section 72 replaces the old object with a new object (an object newly supplied to the object writing section 72).

In addition, when the object writing section 72 writes an object of high spatial resolution into the object memory 75, the object writing section 72 changes the pixel value stored in an address of the object identification memory 76 corresponding to the object from 0 to 1. A background image flag. When the object writing section 72 writes an object into the object memory 75 , the object writing section 72 checks the object flag memory 76 . The object writing section 72 does not write an object with a low spatial resolution on the object memory 75 storing an object (object flag=1) with a high spatial resolution. In the same manner as the background image memory 73 , whenever an object is supplied to the object writing section 72 , the object is written to the object writing section 72 . However, the object writing section 72 does not write an object of low spatial resolution to a pixel storing an object of high spatial resolution. As a result, whenever an object of high spatial resolution is supplied to the object writing section 72 , the number of objects of high spatial resolution is incremented in the object memory 75 .

The background image memory 73 stores the background image supplied from the background image writing section 71 . The background image flag memory 74 stores the above-mentioned one-bit background image flag according to one address of the background image memory 73 . The background image flag memory 74 indicates whether a background image of high spatial resolution has been stored at an address of the background image memory 73 or not. The object memory 75 includes at least one memory section for storing each object supplied from the object writing section 72 . The object flag memory section 76 stores the above-mentioned one-bit object flag representing whether an object of high spatial resolution has been stored in the object memory 75 or not.

In this case, for simplicity, the background image flag and the object flag are one-bit flags. Alternatively, the background image flag and the object flag may be multi-bit flags. In this case, background image flags and object flags can represent multi-level resolutions. In other words, a one-bit flag represents only two levels, high resolution and low resolution. However, a multi-bit flag represents multiple levels of resolution.

The combining section 77 reads out the background image of the current frame (this frame may be referred to as a considered frame) stored in the background image memory 73 corresponding to a background image movement vector contained in the additional information. Furthermore, the combining section 77 combines the background image and an object stored in the object memory 75 corresponding to an object movement vector contained in the additional information. As a result, the combining section 77 generates an image in consideration of the frame, and supplies the image to a display memory 78 .

Furthermore, when the combination section 77 receives control information from the control information input section 24 (see FIG. 5), the combination section 77 reads out an object at the point of consideration contained in the control information from the object memory 75, and supplies the object to Child window memory 79 .

The display memory 78 functions as a so-called VRAM (Video Read Only Memory). The display memory 78 temporarily stores the image of the considered frame supplied from the combining section 77 .

The sub-window memory 79 temporarily stores an object supplied from the combining section 77 .

The superimposing section 80 reads out the stored content of the display memory 78, and supplies the stored content to the image output section 23 (see FIG. 5). The image output section 23 displays the stored content. When necessary, the superimposing section 80 opens a sub-window (to be described later) on the image output section 23, reads out the stored content of the sub-window memory 79, and displays the content on the sub-window.

Next, processing (combining processing) performed by the combining processing section 22 shown in FIG. 13 will be described with reference to FIG. 14 .

First, in step S31, the background image writing part 71 or the object writing part 72 respectively correspond to the background image flag stored in the above-mentioned manner in the background image flag memory 74 or the object flag stored in the above-mentioned manner in the object flag memory 75. A background image or an object supplied by section 53 (see Fig. 12) is written into the associated memory.

In other words, the background image writing part 71 checks the background image flag memory 74, and writes the supplied background image to the address of the background image memory 73, and the background image flag of the pixel corresponding to the address is 0 at this moment. On the contrary, if the spatial resolution of the supplied background image is very high, the background image writing part 71 writes the background image into an address of the background image memory 73, and the background image flag of the pixel corresponding to the address is 1 at this moment.

Also, when the object flag of the object memory 75 is 0, the object writing section 72 writes the supplied object to the object memory 75 . When the object flag of the object memory 75 is 1, the object writing section 72 writes the supplied object into the object memory 75 if the spatial resolution of the supplied object is high.

When the background image writing part 71 writes the background image into a specific address of the background image memory 73, if a background image has been stored at this address, then the background image writing part 71 overwrites the background image on the address . This processing also applies to the object memory 75 .

Thereafter, the flow proceeds to step S32. In step S32, the background image writing section 71 and the object writing section 72 determine whether the additional information contains high-resolution information. When the determined result at step S32 shows that the additional information contains high-resolution information (that is, the user has operated the control information input section 24 (see FIG. When an image within the range sends a background image and an object with high spatial resolution), the flow proceeds to step S33. In step S33 , the background image writing section 71 or the object writing section 72 changes the background image flag of the background image flag memory 74 or the object flag of the object flag memory 76 to 1.

In other words, when the transmitting device 1 transmits a background image and an object with high spatial resolution for an image within the priority range, in step S31, the background image writing section 71 and the object writing section 72 will have high spatial resolution. The background image and objects at spatial resolution are written to background image memory 73 and object memory 75, respectively. Thus, at step S33, the background image flag and the object flag of one pixel constituting the background image and the object with high spatial resolution are changed to 1.

Thereafter, the flow advances to step S34. The combination section 77 reads out an object within the priority range from the object memory 75 and writes the object into the sub-window memory 79 .

In other words, when the determined result at step S32 shows that the additional information contains high-resolution information, it means that the user has operated the control information input section 24 (see FIG. 5 ), the control information has been sent to the sending device 1, and Device 1 has then sent a background image and an object with high resolution for an image in the priority range. The control information sent to the transmitting device 1 is also supplied to the combining section 77 . Thereby, when combining section 77 receives control information, on step S34, combining section 77 corresponds to the coordinate identification (recognize) its priority range of the point of consideration contained in the control information, reads out from object memory 75 that transmission device 1 has transmitted. The high spatial resolution object within the priority range, and write this object to the sub-window memory 79.

Thereafter, the flow advances to step S35. In step S35, the combining section 77 reads out from the background image memory 73 the background image of the current frame (this frame is referred to as a considered frame) displayed corresponding to the background image movement vector contained in the additional information. Furthermore, the combining section 77 reads out an image displayed with respect to the considered frame from the object memory 75 . The combining section 77 combines the background image of the considered frame with the object read from the object memory 75 corresponding to the object movement vector contained in the additional information, and writes the combined image of the considered frame to the display memory 78 . In other words, the combining section 77 writes a background image to, for example, the display memory 78 and then overwrites an object onto the display memory 78 . As a result, the combining section 77 writes, onto the display memory 78, an image in which a background image and an object are combined into a considered frame image.

In this manner, an image of the considered frame written to the display memory 78 and an object written to the sub-window memory 79 are supplied to the image output section 23 (see FIG. 5 ). The image output section 23 displays supplied images and objects.

In the sending device 1 a so-called soft key can be contained eg in additional information. In this case, the combining section 77 combines the object and the background image using soft keys.

In contrast, when the determined result at step S32 indicates that the additional information does not contain high-resolution information (ie, the user has not operated the control information input section 24 (see FIG. 5 )), the flow advances to step S35, skipping steps 33 and 34. As described above, the combining section 77 reads out a background image of the considered frame from the background image memory 73 . Furthermore, the combining section 77 reads out a desired object from the object memory 75, and combines the background image of the considered frame with the object read out from the object memory 75 corresponding to the additional information. As a result, an image considering the frame is generated and written to the display memory 78 . Thereafter, the flow returns to step S31. Then, the combination processing section 22 repeats similar processing.

In the combination processing described above, when the user does not operate the control information input section 24 (see FIG. 5 ) (that is, the control information input section 24 composed of a pointing device such as a mouse that is not dragged (or clicked)), as shown in FIG. As shown in FIG. 15(A), the low spatial resolution image is displayed with a default temporal resolution on the image output section 23 (see FIG. 5). In Fig. 15(A), the low spatial resolution image moves over the low spatial resolution background image.

When the user operates the control information input section 24 (see FIG. 5), moves the cursor to the object, and drags the cursor to the position of the object, the control information is transmitted to the transmission device 1 as described above. The transmitting device 1 transmits the data required for displaying images of high spatial resolution for images within a priority range at the expense of temporal resolution. As a result, as shown in FIG. 15(B), although the temporal resolution is (for example) 0 frames/second, the images within the priority range around the dragging position are displayed on the image output section 23 (see FIG. 5 ). An image in which the spatial resolution of an object and a background image is progressively improved. In other words, corresponding to the dragging time (or the number of clicks), the spatial resolution of images within the priority range is gradually improved.

In this case, as shown in FIG. 15(B), a sub-window is opened on the image output section 23 (see FIG. 5). An object within the priority range around the drag position is displayed in such a manner that the spatial resolution of the object is gradually improved.

Thereafter, when the user stops the dragging operation with the control information input section 24 (see FIG. 5 ), as described above, the combination section 77 reads out a background image of the considered frame from the background image memory 73 at step S35, and from the object memory 75 reads out an object, combines the background image of the considered frame with the object read from the object memory 75 corresponding to the additional information, and writes the combined image to the display memory 78 . As described above, since the object of high spatial resolution as a result of the drag operation is stored in the object memory 75, the image output section 23 (see FIG. 5) is displayed with the conventional temporal resolution shown in FIG. 15(C) in consideration of Paste an image of an object with high spatial resolution as a result of the drag operation at the relevant position of the frame.

Thereafter, the image output section 23 (see FIG. 5) displays an image of an object of high spatial resolution moved corresponding to the additional information with conventional time resolution.

Thus, when the user performs a drag operation on the position of the object he or she wants to watch carefully, he or she can watch the object with high spatial resolution. In other words, the user can view an object in detail.

When the user performs a drag operation, a background image of high spatial resolution within a priority range is stored in the background image memory 73 . Thus, even if the user stops the drag operation, the background image of high spatial resolution stored in the background image memory 73 is displayed. Thus, when the user performs a drag operation, since the background image in the priority range around the drag position is improved, no matter when the user performs With the drag operation, the background image with high spatial resolution is gradually expanded according to a mosaic pattern. Finally, the image output section 23 displays the entire high spatial resolution background image.

Furthermore, according to this embodiment, as described above, since a background image is stored in the background image memory 73, the transmitting device 1 does not need to transmit an already transmitted one with a low spatial resolution. Thus, the transmission bandwidth (transmission rate) of the background image can be allocated to one object and one background image with high spatial resolution.

In the above case, an object of high spatial resolution as a result of the drag operation is stored to the object memory 75 . After stopping the dragging operation, the high spatial resolution object is pasted onto the background image. Thus, the spatial resolution of an object displayed by the receiving device 2 becomes high. However, the state of the object photographed by the sending device 1 does not affect the object displayed by the receiving device 2 .

In order to solve this problem, after stopping the dragging operation, an object of high spatial resolution stored in the object memory 75 can be replaced with an object stored in the storage section 62F of the decoding section 53 (see FIG. 12 ), while ignoring Object sign. In other words, the objects transmitted by the transmitting device 1 are successively stored on the storage section 62F of the decoding section 53 (see FIG. 12 ). Thus, when an object is written to the object memory 75, the object of an image displayed on the image output section 23 is affected by the state of the object photographed by the transmitting device 1 (however, the spatial resolution of the object is lowered).

The MUX 32 shown in FIG. 7 including the transmission processing section 16 of the transmission device 1 controls the presence/absence of high-resolution information, a frame rate ( time resolution) and a spatial resolution, and a frame rate and a spatial resolution within the non-priority range are placed on a header or the like of the multiplexed data. The receiving device 2 recognizes the presence/absence of high-resolution information, the frame rate and spatial resolution within the priority range, and the non-priority frame rate and spatial resolution within the right range.

In the transmitting device 1 , the header information may contain, for example, (coordinates) of the considered point contained in the control information transmitted by the receiving device 2 . In this case, the combining section 77 shown in FIG. 13 can identify an object at a position of a consideration point corresponding to the header information transmitted by the transmitting device 1 . In other words, in the above case, the combination section 77 reads out the object from the object memory 75 corresponding to the control information supplied from the control information input section 24 (see FIG. , and supply the object to the child window memory 79. However, when the header information contains a consideration point, the combination section 77 identifies an object at a position of the consideration point corresponding to the header information. In this case, in FIG. 5 , the control information input section 24 does not need to supply the control information to the receiving device 2 .

Next, the relationship between the spatial resolution and the temporal resolution of an image transmitted from the transmitting device 1 to the receiving device 2 via the transmission path 3 will be described with reference to FIG. 16 .

Assume that the transmission rate of the transmission path 3 is R (bps) and that one image consisting of one background image and three objects #1 to #3 is transmitted. For simplicity, additional information is not considered in this embodiment. Also, assume that in order to display a background image and objects #1 to #3 with a certain spatial resolution, each of them requires the same amount of data.

In this case, when the user does not perform a drag operation, in the transmission device 1, as shown in FIG. 16(A), the background image and the object #1 are transmitted at R/4 [bps] in which the transmission rate R is divided by 4 to #3. When the conventional temporal resolution is 1/T frame/sec, the transmitting device 1 can transmit the background image and one frame of data of each of the objects #1 to #3 within a period of at most T seconds. Thus, in this case, the reception device 2 displays a background image and objects #1 to #3 of one spatial resolution obtained with data of T×R/4 bits per frame.

When the user performs a drag operation on, for example, the position of object #1, as shown in FIG. R only sends object #1. Thereafter, at time t2 when a period of time 4T elapses from time t1 , when the user stops the drag operation, the transmission device 1 transmits the background image and objects #1 to #3 at each transmission rate R/4.

Thus, when the user performs a drag operation, the transmission device 1 transmits data of 4T×R bits for the object #1. Therefore, assuming that when the user performs a drag operation, the temporal resolution is 0 frames/second, and the receiving device 2 displays the object #1 at a spatial resolution obtained with data of 4T×R bits per frame. In other words, when the horizontal spatial resolution and the vertical spatial resolution are improved by the same amount, although the temporal resolution becomes 0 frame/second, the receiving device 2 displays four times as much as the case where the user does not perform the dragging operation (=( Object #1 of one spatial resolution in each of the horizontal direction and the vertical direction of 4T×R/(T×R/4 bits))).

In other words, objects of this spatial resolution can be improved at the expense of temporal resolution. Furthermore, the spatial resolution of the part considered by the user can be rapidly improved compared to the case where temporal resolution is sacrificed.

In the above case, when the user performs a drag operation for object #1, the background image and all data of other objects #1 and #3 are not transmitted. However, as shown in FIG. 16(B), a higher transfer rate may be assigned to data of object #1, and a lower transfer rate may be assigned to data of background images and other objects #2 and #3.

On the other hand, even if a drag operation is performed, the transfer rate assigned to each of the background image and objects #1 to #3 does not change from R/4. In other words, in this case, since the spatial resolution is improved at the expense of the temporal resolution, the spatial resolution can be improved without changing the transfer rate, for example, although it takes a long time to display the data.

According to this embodiment, as described above, an object with high spatial resolution as a result of the drag operation is stored in the object memory 75 . After the drag operation stops, the high spatial resolution object is pasted onto a background image. However, the position where the background image of the object of high spatial resolution is pasted depends on an object movement vector contained in the object additional information transmitted by the transmitting device 1 .

Thus, the receiving device 2 will agree on what object of a particular frame should be recognized corresponding to what object of an adjacent frame. Therefore, when the object extracting section 14 (see FIG. 3) of the transmitting device 1 extracts an object, the object extracting section 14 will allow the receiving device 2 to add information for performing such an identification operation to the additional information.

FIG. 17 shows a structural example of the object extraction section 14 shown in FIG. 3 .

An image output from the image input device 11 and a background image output from the background image extraction section 13 are supplied to a subtraction section 81 . The subtraction section 81 subtracts the background image output from the background image extraction section 13 from the image output from the image input section, and obtains a foreground image as an object. The foreground image obtained by the subtraction section 81 is supplied to the frame memory 82 and the initial area division section 83 .

The frame memory 82 temporarily stores the foreground image supplied from the subtraction section 81 . The initial area division section 83 uses the foreground image of the current frame (this frame is called a consideration frame) supplied from the subtraction section 81 and the foreground image of the previous frame stored in the frame memory 82 (this foreground image is called a previous frame). foreground image) to perform initial region segmentation processing.

In other words, the initial area dividing section 83 classifies each pixel constituting the foreground image of the considered frame by category corresponding to the pixel value. In fact, when a pixel value is represented by RGB (red, green, blue), the initial area division part 83 is based on a vector (this vector can be called a color vector) composed of elements of R, G, and B values and The relationship between multiple small regions in the RGB space classifies each pixel. Furthermore, the initial area dividing section 83 classifies each pixel including the foreground image of the previous frame stored in the frame memory 82 in the same manner.

When the considered frame is, for example, the n-th frame, the initial region dividing section 83 divides the foreground image of the n-th frame and the foreground image of the (n-1)-th frame of the previous frame into temporally and spatially adjacent A region of pixels grouped by category.

In other words, for example, it is assumed that each pixel constituting the foreground image of the nth frame and each pixel constituting the foreground image of the (n-1)th frame are classified by category as shown in FIG. 18(A). In FIG. 18 (also in FIGS. 19 and 20), a group of a character C and a numeral contained in a rectangle represents a category of a pixel.

In the case shown in FIG. 18(A), when the foreground image of the nth frame and the foreground image of the (n-1)th frame are divided into a plurality of regions composed of a plurality of pixels classified by the same category , forming the initial region represented by the dotted line shown in FIG. 18(B).

The initial area obtained by the initial area dividing section 83 is supplied to the area combining section 84 shown in FIG. 17 .

The area concatenation section 84 performs area boundary processing for the initial area supplied from the initial area division section 83 .

In other words, the area concatenation section 84 reads object information of an object included in the (n-1)th frame from the object information memory 88, and recognizes the position and range of the object included in the (n-1)th frame. In addition, the region concatenating section 84 identifies the pixels constituting the object contained in the (n-1)th frame, and concatenates the initial region containing the pixels.

In fact, for example, it is assumed that a specific object Obj classified into categories c2, c3, c4, and c5 is contained within a range (a box) indicated by a solid line shown in FIG. 19(A). composed of many pixels. The initial region containing these pixels is tiled. Thus, in this example, an initial area consisting of pixels of classes c2, c3, c4, and c5 (hereinafter, an initial area consisting of pixels of class c#i is referred to as an initial area c#i) is concatenated is the shaded area shown in Fig. 19(B).

Furthermore, the region merging section 84 calculates the distance between the merged initial region (hereinafter may be referred to as a merged region) and the original region adjacent to the merged region. In this instance, the distance between a spliced area and an adjacent initial area (hereinafter may be referred to as an adjacent initial area) may be the pixel value ( color) (this distance may be referred to as a distance in RGB space) or the continuity of pixel values (colors) of pixels in the vicinity of the boundary of two regions.

When the distance between the merged region and the adjacent initial region is less than (or equal to) a predetermined threshold, the region combining part 84 merges the merged region and the adjacent initial region, and generates a new merged region. The region merging section 84 calculates the distance described above, and repeats merging adjacent initial regions and merged regions until there is no adjacent initial region that can be merged into a merged region.

Thus, a stitched region shown in, for example, FIG. 19(C) is formed with the stitched region shown in FIG. 19(B). In FIG. 19C, since the distance between the initial regions c7 and c8 is close, they are merged into the merged region shown in FIG. 19(B). On the contrary, due to the large distance between the initial regions c1 and c6, they cannot be spliced into one spliced region.

When there is no adjacent initial region that can be stitched into a stitched region as shown in FIG. ) an object corresponding to the object Obj of the frame (this object can be referred to as a corresponding object), designate the same sign as the sign of the object Obj of the nth frame, and output the object to the splicing area processing part 85 and the separation area Processing section 86 .

In other words, according to this embodiment, objects that coincide in each frame are assigned the same flag. The receiving device 2 recognizes an object of the adjacent frame corresponding to an object of the adjacent frame corresponding to the flag.

The flag assigned to the object Obj of the (n−1)th frame is included in the object information and stored in the object information memory 88 . With respect to the object information memory 88, the area concatenation section 84 recognizes the object Obj flag assigned to the (n-1)th frame.

Furthermore, when the region concatenation section 84 extracts the objects of the nth frame corresponding to all the objects of the (n-1)th frame in the above-described manner, the region concatenation section 84 extracts each initial region of the nth frame and its other A region adjacent to the initial region is stitched into an object, and a new flag (rather than any flag assigned to the object at (n-1) frames) is assigned to the extracted object and the extracted object is output to the stitched region process section 85 and separate area processing section 86.

The joined-area processing section 85 and the separated-area processing section 86 execute a joined-area process and a separated-area process, respectively. The stitched region processing is performed when stitching objects. Separation region processing is performed when separating stitched objects.

In other words, for example, regarding three consecutive frames of the (n-1)th frame, the nth frame, and the (n+1)th frame, as shown in FIG. 20, in the (n-1)th frame, Objects A and B are close to each other. In frame n, objects A and B overlap each other. In other words, in the nth frame, objects A and B can be spliced into one object. In this case, the objects A and B that have been spliced into one object move continuously in their directions. In the (n+1)th frame, the stitched object is separated into two objects A and B.

In this case, in the region-merging processing in the region-merging section 84, the merged object of the n-th frame corresponds to both the objects A and B in the (n-1) frame. On the contrary, the separated objects A and B in the (n+1)th frame are equivalent to one concatenated object in the Nth frame. According to this embodiment, it is assumed that an object of a certain frame is equivalent to an object of the previous frame. Thus, as mentioned above, it is best not to have one object relate to two objects. Instead, it's better not to have two objects related to one object.

Thus, the mosaic processing section 85 executes a mosaic area process by calculating a mosaic object of the n-th frame with respect to one of the objects A and B of the (n-1)-th frame. The separated region processing section 86 performs separated region processing by correlating one of the two objects A and B of the (n+1)th frame with one spliced object of the nth frame.

In other words, the merged area processing section 85 assigns the same flag as one of the objects A and B of the (n-1)th frame to one merged object of the nth frame. On the other hand, the separated area processing section 86 assigns the same flag as the joined object of the nth frame to one of the two separated objects A and B of the (n+1)th frame, and assigns a new flag to the ( n1) Another detached object of frame.

The object extraction result as a result of the combined area processing performed by the combined area processing section 85 and the object extraction result as a result of the separated area processing performed by the separated area processing section 86 are combined by the area combining section 84 and supplied to a new Area processing section 87 .

When the object extraction results of the joined area processing section 85 and the separated area processing section 86 contain a new object, the new area processing section 87 performs new area processing of the new object.

There are three types of objects to which new flags are assigned in the object extraction results of the merged area processing section 85 and the separated area processing section 86 . A first type of object is one that the region stitching section 84 does not extract the object of the frame of interest like the corresponding object of the previous frame because the object moves fast and the object does not spatially overlap in the frame of interest and the previous frame. The second type of object is an object for which it is impossible for the separation region processing section 86 to correlate the object of the considered frame with the corresponding frame of the previous frame because the corresponding object of the previous frame is spliced with another object. A third type of object is an object to which a new flag is assigned due to the appearance of a new object in the frame under consideration.

Among these three types of objects, a new flag should be assigned to an object of the third type. Thus, for objects of the first and second types, the new area processing section 87 detects a corresponding object from the previous frame, and reassigns the same flag as the object of the considered frame to the corresponding object of the previous frame.

In fact, the new area processing section 87 checks the object information memory 88, recognizes the objects of several past frames of the considered frame, and obtains the distance between each of the recognized objects and the considered object of the considered frame, which is assigned a new logo. Like the distance between objects, the distance between feature quantities of objects may also be used. Examples of object feature quantities are an area of the object, a bar graph in the tangential direction of the contour line made up of each pixel of the object (for example, along the upper, lower, left, right, upper left, lower left, upper right, and upper right eighth) bars in each of the directions), and a movement vector for the object.

The new area processing section 87 obtains the minimum value of the distance between the recognized object and the considered object. When the minimum value is less than (or equal to) the predetermined threshold, the new area processing section 87 processes an object having the minimum distance from the considered object as an object corresponding to the considered object, and assigns the same flag as the obtained object flag to Consider an object, and output the considered object. When the minimum value of the distance between the agreed object and the considered object is not less than a predetermined threshold (that is, there is no such object whose distance from the considered object was short in the past frame), the new area processing section 87 converts the considered object to The process is to assign a new flag to the consideration frame for an object newly appearing in the consideration frame, and output the consideration frame.

The output of the new area processing section 87 is supplied to the additional information calculation section 15 (see FIG. 3 ) and the transmission processing section 16 (see FIG. 3 ). Furthermore, the output of the new area processing section 87 is supplied to the object information storage 88 . The object information memory 88 temporarily stores an object (position and size (outline) of the object, pixel values of pixels constituting the object, etc.) and a flag assigned as object information.

Object extraction processing for extracting an object from an image will be described below with reference to the flowchart shown in FIG. 21 . The object extraction processing is performed by the object extraction section 14 shown in FIG. 17 .

One image output from the image output section 11 and one background image output from the background image extraction section 13 are supplied to the subtraction section 81 . In step S41, the subtraction section 81 subtracts the background image output from the background image extraction section 13 from the image output from the image input section 11 to obtain a foreground image as an object. The foreground image obtained by the subtraction section 81 is supplied to the frame memory 82 and the initial area division section 83 . The frame memory 82 stores the foreground image output from the subtraction section 81 .

On the other hand, in step S42, the initial area dividing section 83 checks a foreground image of a frame preceding the considered frame, performs the initial area dividing process described in accordance with FIG. 18, obtains an initial area, and supplies the obtained initial area to the area Splicing part 84. In step S43, the region mosaic section 84 checks the object information of the previous frame stored in the object information memory 88, performs region mosaic processing for the initial region output by the initial region dividing section 83 in the manner described in conjunction with FIG. 19 , and extracts the considered frame an object of .

The objects extracted by the area concatenation section 84 are supplied to the concatenation area processing section 85 and the separation area processing section 86 . In step S44, the joined area processing section 85 or the separated area processing section 86 executes joined area processing or separated area processing in the manner described in conjunction with FIG.

In step S45 , the new area processing section 87 performs the above-described new area processing for the outputs of the merged area processing section 85 and the separated area processing section 86 . Thus, the new area processing section 87 outputs the final object extraction result of the considered frame. The extraction result of the object is supplied to the additional information calculation section 15 (see FIG. 3 ) and the transmission processing section 16 (see FIG. 3 ). Furthermore, the extraction result of the object is supplied to and stored in the object information storage 88 .

Thereafter, the flow returns to step S20, at which the next frame is designated as a new frame for consideration. The object extraction section 14 then repeats similar processing.

The area combining process performed by the area combining section 84 at step S43 in FIG. 21 will be described in detail below with reference to the flowchart shown in FIG. 22 .

In the area mosaic processing, first, at step S51, the area mosaic section 84 checks the object information about an object contained in the previous frame (previous frame) of the considered frame, and processes the object of the previous frame as a considered object. In step S51, using the object of consideration, as shown in FIG. 19(B), the region concatenating part 84 concatenates the initial regions output by the initial region dividing part 83 to form a concatenated region.

Thereafter, the flow advances to step S52. In step S52, the area combining section 84 searches for an initial area adjacent to the merged area (this initial area may be referred to as an adjacent initial area). Thereafter, the flow advances to step S53. In step S53, the area merging section 84 calculates the distance between the merging area and the considered initial area. Thereafter, the flow proceeds to step S54. In step S54, the region combining section 84 determines whether the distance between regions is smaller than a predetermined threshold.

When the determined result at step S54 indicates that the distance between the stitched area and the considered initial area is smaller than the predetermined threshold, the flow advances to step S55. In step S55, the region merging section 84 merges the considered initial region into the merged region, thereby forming a new merged region. Thereafter, the flow proceeds to step S56.

On the contrary, when the determined result at step S54 indicates that the distance between the concatenated area and the considered initial area is not smaller than the predetermined threshold, the flow proceeds to step S56, skipping step S55. In other words, the region merging section 84 does not merge the considered initial region into the merged region. In step S56, the area mosaic section 84 determines whether or not all initial areas adjacent to the merged area have been searched. When the determined result at step S56 indicates that all initial areas adjacent to the stitched area have not been searched, the flow returns to step S52. In step S52, the area concatenation section 84 searches for adjacent initial areas that have not been searched. Thereafter, the region mosaic section 84 repeats similar processing.

When the determined result at step S56 indicates that all initial areas adjacent to the stitched area have been searched, the flow proceeds to step S57. In step S57, the region mosaic section 84 determines whether or not each of all objects contained in the previous frame has been designated as an object of consideration. When the determined result at step S57 signifies that each of all the objects contained in the previous frame is not specified as the object of consideration, the flow returns to step S51. In step S51, the region concatenation section 84 designates an object contained in the previous frame as the object of consideration, and then repeats similar processing for the newly considered object.

In contrast, when the determined result at step S57 signifies that each of all objects contained in the previous frame has been designated as an object of consideration, the flow returns to the called process.

Next, the merged region processing performed by the merged region processing section 85 at step S44 shown in FIG. 21 will be described in detail with reference to the flowchart shown in FIG. 23 .

First, at step S61, the concatenated area processing section 85 designates a processed frame as a considered frame, checks the object information memory 88, and recognizes the previous frame under the condition that the object of the previous frame spatially overlaps with the considered object of the considered frame. The number of objects contained in a frame (the previous frame) (ie, the number of objects in the previous frame of the considered frame), and adjust the number of objects to a variable N.

Thereafter the flow advances to step S62. In step S62, the merged area processing section 85 determines whether the variable N is 2 or greater than 2. When the variable N is less than 2 (ie, there is no object in the previous frame that spatially overlaps with the considered object or the number of objects is 1), the process proceeds to step S65, and steps S63 and S64 are skipped.

In contrast, when the determined result at step S62 indicates that the variable N is equal to or greater than 2 (ie, two or more objects spatially overlapping with the considered object are contained in the previous frame), the flow proceeds to step S63. In step S63 , the stitched area processing section 85 calculates the distance between the object under consideration and each of two or more objects of the previous frame that spatially overlap the object under consideration. Thereafter the flow advances to step S64.

At step S64, the merged area processing section 85 selects an object having the smallest distance to the considered frame from among the objects obtained at step S63, and assigns the same flag as that of the selected object to the considered object.

Thereafter, the flow advances to flag S65. In step S65 , the merged area processing section 85 determines whether or not each of all the objects contained in the considered frame has been designated as the considered object. When the determined result at step S65 indicates that each of all the objects contained in the considered frame has not been designated as the considered object, the merged area processing section 85 designates one of these objects as the considered frame. Thereafter, the flow returns to step S61. In step S61, the merged area processing section 85 repeats similar processing.

On the other hand, when the determined result at step S65 signifies that each of all objects contained in the consideration frame has been designated as an object of consideration, the flow returns to the called process.

Next, the separation region processing performed by the separation region processing section 86 at step S44 shown in FIG. 21 will be described in detail with reference to the flowchart shown in FIG. 24 .

The separated area processing section 86 checks the object information memory 88 and designates one of the objects contained in the previous frame (previous frame) of the considered frame as a considered object. Further, in step S71, the separated area processing section 86 agrees the number of objects of the frame under consideration corresponding to the object under consideration (the agreed object is referred to as a corresponding object), and adjusts the number of objects to a variable N.

Thereafter, the flow proceeds to step S72. In step S72, the separation area processing section 86 determines whether the variable N is equal to 2 or greater than 2.

When the determination result on the step S72 shows that the variable N is not greater than 2 (that is, the considered frame does not contain an object that overlaps with the considered object in space or the number of objects is 1), the flow process advances to the step S76, skipping the steps S73 to S75.

On the contrary, when the determined result at step S72 shows that the variable N is equal to 2 or greater than 2 (that is, when the consideration frame contains two or more objects (objects corresponding to the consideration objects) that overlap with the consideration object in space), the flow proceeds Go to step S73. In step S73, the separation area processing section 86 calculates the distance between each of these objects and the object of consideration. Thereafter, the flow proceeds to step S74.

In step S74, the separation area processing section 86 selects an object having the smallest distance from the object under consideration from among the objects, and assigns the same flag as that of the selected object to the object under consideration.

Thereafter, the flow proceeds to step S75. At step S75, the separated area processing section 86 assigns a new flag to one of the objects not selected at step S74 (ie, objects other than the object at the shortest distance from the considered object). Thereafter, the flow proceeds to step S76.

In step S76, the separated area processing section 86 determines whether or not each of all objects contained in the previous frame has been designated as an object of consideration. When the determined result at step S76 indicates that each of all the objects contained in the previous frame has not been designated as the object of consideration, the separated area processing section 86 designates one of these objects as the object of consideration. Thereafter, the flow returns to step S71. In step S71, the separated area processing section 86 repeats similar processing.

When the determined result at step S76 signifies that each of all objects contained in the previous frame has been designated as an object of consideration, the flow returns to the called process.

In the above case, when the user designates a point of consideration with the control information input section 24, the transmission device 1 controls the transmission of data so as to improve the quality of an image within the priority range containing the point of consideration at the expense of the temporal resolution of the image. spatial resolution. On the other hand, the transmitting device 1 can learn the user's preference, detect an object or the like that the user intends to view with a high spatial resolution corresponding to the learning result, and control transmission of data so as to display the object with a high spatial resolution .

FIG. 25 shows an example of the structure of the control section 35 shown in FIG. 7 in the case where the transmitting device 1 executes such a control.

The priority range specifying section 91 receives a control signal transmitted from the receiving device 2, specifies the priority range in the above-mentioned manner, and supplies the specified priority range to the selection control section 92 and the feature quantity extraction section 93.

The selection control section 92 controls data selection of background images, objects and additional information of the MUX 32 (see FIG. 7). In other words, when the selection control section 92 receives the priority range from the priority range designation section 91, the selection control section 92 controls the MUX 32 (see FIG. spatial resolution. Furthermore, when the selection control section 92 receives a flag from the object detection section 95, the selection control section 92 controls the MUX 32 (see FIG. 7) to improve the spatial resolution of the object with the flag at the expense of the temporal resolution of the image.

The data amount calculation section 34 (see FIG. 7 ) supplies the selection control section 92 with the data rate of the multiplexed data output by the MUX 32 . The selection control section 92 controls the data selection of the MUX 32 so that the multiplexed data rate does not exceed the transfer rate of the transfer path 3 .

The background image, object, and additional information output by the preprocessing section 12 (see FIG. 3 ), and the priority range output by the priority range specifying section 91 are supplied to the feature quantity extraction section 93 . The feature quantity extracting section 93 extracts feature quantities of images within the priority range output from the priority range specifying section 91 . In other words, the feature amount extraction section 93 extracts the feature amount of the objects contained in the priority range such that the feature amount reflects the trend of the image the user is considering.

Actually, for example, as shown in FIG. 26 , the feature quantity of an object of a specific person extracted by the feature quantity extracting section 93 indicates that an object is a person, the movement of the object is consistent, and the position of the object in the depth direction (depth of field) is Foreground, the position of the displayed object is the center, the object is moving (the object is a position that is moving), the area of the object contains eyes, nose and mouth (the area of the object consists of eyes, nose and mouth), the pattern of the object is stripes The pattern (the object is a striped part), the color of the object is red (the object is a red part).

The feature quantity extraction part 93 obtains a vector (element with a feature quantity) of an extraction object (this vector can be referred to as a feature quantity vector), and obtains the bar graph of the feature quantity vector stored in the bar graph storage part 94 The frequency is incremented by 1.

The bar graph storage section 94 stores a bar graph of the feature quantity vector as a user's preferred learning result.

The object detection section 95 detects an object from those supplied from the preprocessing section 12 (see FIG. 3 ) to obtain a feature quantity vector having the highest frequency of the histogram from the histogram storage section 94 . In other words, the object detection section 95 obtains a feature vector for an object supplied from the preprocessing section 12 (see FIG. 3 ) in the same manner as the feature amount extraction section 93 . Furthermore, the object detection section 95 checks the histogram stored in the histogram storage section 94 to determine whether the feature amount vector of the object supplied from the preprocessing section 12 (see FIG. 3 ) is contained in the feature amount vector of the feature amount vector having the highest frequency. within a predetermined range of vector space. When the feature quantity vector is contained within a predetermined range, the object detection section 95 specifies the object as the object that the user intends to consider, and supplies the flag of the object to the selection control section 92 .

Next, the control processing of the MUX 32 (see FIG. 7 ) will be described with reference to the flowchart shown in FIG. 27 . This control processing is executed by the control section 35 shown in FIG. 25 .

First, at step S81 , the priority range specifying section 91 determines whether or not a control signal has been transmitted from the receiving device 2 . When the determined result at step S81 indicates that the control signal has been transmitted from the receiving device 2, the flow proceeds to step S82. At step S82, the priority range specifying section 91 specifies a priority range corresponding to the control signal in the above-described manner, and supplies the priority range to the selection control section 92 and the feature quantity extraction section 93.

In step S83, the selection control section 92 controls the MUX 32 (see FIG. 7) to improve the space of an image (an object and a background image) within the priority range supplied from the priority range specifying section 91 at the expense of the temporal resolution of the image. resolution.

At step S84, the feature amount extracting section 93 extracts feature amounts of images within the priority range supplied from the priority range specifying section 91, obtaining a feature amount vector having elements composed of each feature amount of the object. At step S85 , the feature amount extraction section 93 increments the frequency of the histogram of the feature amount vector stored in the histogram storage section 94 by one. The flow returns to step S81.

In the loop from step S81 to step S85, the histogram storage section 94 forms a histogram of the object feature vectors that the user intends to consider. In other words, the bar graph storage section 94 learns the user's preference.

Furthermore, the feature amount extraction section 93 may quantize the obtained feature amount vector and increment the frequency of the code corresponding to the quantization result of the feature amount vector. In this case, the bar graph storage section 94 temporarily stores the bar graph of the code.

In contrast, when the determined result at step S81 indicates that the receiving device 2 has not transmitted the control signal, the flow proceeds to step S86. At step S86, the object detection section 95 obtains the feature amount vector of the object supplied from the preprocessing section 12 (see FIG. 3 ) in the same manner as the feature amount extraction section 93 . Furthermore, at step S87, the object detection section 95 checks the histogram stored in the histogram storage section 94, and determines whether the predetermined range is contained within a predetermined range of the feature quantity vector space around the feature quantity vector having the highest frequency. A feature quantity vector of the object supplied from the processing section 12 (see FIG. 3). In other words, in step S87, the object detection section 95 determines whether the distance between the feature amount vector having the highest frequency and the feature amount vector of the object supplied from the preprocessing section 12 is equal to or smaller than a predetermined value.

As described above, when the histogram storage section 94 stores the code histogram as a vector quantization result, the object detection section 95 quantizes the obtained feature quantity vector. In step S87 , the object detection section 95 determines whether the code that is the vector quantization result matches the code of the highest frequency of the histogram stored in the histogram storage section 94 .

When the result of the determination at step S87 shows that the distance between the feature quantity vector having the highest frequency and the feature quantity vector of the object supplied by the preprocessing section 12 is not less than a predetermined value (that is, the distance between the feature quantity vector supplied by the preprocessing section 12 due to the user's preference When the object is an object that the user does not intend to consider), the flow advances to step S88. At step S88, the selection control section 92 controls the MUX 32 (see FIG. 7) to cause the receiving device 2 to display an image with normal spatial resolution and normal temporal resolution. Thereafter, the flow returns to step S81.

On the contrary, when the result of the determination at step S87 shows that the distance between the feature quantity vector having the highest frequency and the feature quantity vector of the object supplied from the preprocessing section 12 is equal to or smaller than a predetermined value (that is, due to the user's preference, the preprocessing section 12, when the object supplied by the user is the object that the user intends to consider), the object detection section 95 outputs the flag of the object supplied by the preprocessing section 12 to the selection control section 92. Thereafter, the flow advances to step S89.

At step S89 , the selection control section 92 controls the MUX 32 (see FIG. 7 ) to improve the spatial resolution of the object with the marker supplied from the object detection section 95 at the expense of temporal resolution. Thereafter, the flow returns to step S81.

Thus, in this case, the receiving device 2 displays an object with the marker output from the object detection section 95 so as to increase its spatial resolution at the expense of temporal resolution. Thereafter, the receiving device 2 continuously displays objects with high spatial resolution.

As a result, the receiving apparatus 2 automatically displays the object that the user tends to consider, so that the spatial resolution of the object can be increased without the user controlling (intervention) controlling the information input device 24 . Thereafter, the receiving device 2 continuously displays objects of high spatial resolution (however, in this case, the temporal resolution of the image deteriorates as described above).

The bar graph of the feature quantity vector stored in the bar graph storage section 94 as a learning result of the user's preference may be reset periodically, aperiodically, or in response to a user's request of the receiving device 2 .

In the above case, the spatial resolution of objects whose feature quantity vectors are equal to (matched) or smaller than the feature quantity vector of the highest frequency of the bar graph is improved. On the other hand, the spatial resolution is also improved for all objects having a feature amount vector whose feature amount vector is equal to (matches) or smaller than a feature amount vector whose frequency in the bar graph exceeds a predetermined value.

The priority range is a predetermined range where the consideration point is located at the center of gravity (in the above case, the priority range is a rectangular range). However, it can also be said that an image area within a rectangular range including the consideration point is an image area that the user is interested in viewing (hereinafter, this area is referred to as an interest object area).

On the other hand, the moving image displayed by the receiving device 2 has a moving image area (hereinafter, this area may be referred to as a moving area) and a still image area (hereinafter, this area may be referred to as a still area).

In order to improve the spatial resolution of the object of interest region, it is necessary to identify (specify) the object of interest region that the user is considering in the moving region or the stationary region of the image. When the region of the object of interest that the user is considering can be specified, an image region not considered by the user (for example, a background image region) can be obtained.

Even if an interest object area considered by the user can be specified at a certain time, it can be changed thereafter. Thus, when one image area considered by the user is changed to another image area, the new image area must be identified as the object of interest area.

In addition, there may be multiple interest object areas considered by the user. In this case, these object-of-interest regions must be identified separately.

28 shows an example of the second configuration of the image transmission system shown in FIG. 1 in the case where a portable terminal unit is used as the sending device 1 and the receiving device 2 shown in FIG. 1 . In FIG. 28, parts similar to those in FIG. 2 are denoted by the same reference numerals, and their descriptions are omitted. In other words, the structure of the image transmission system shown in FIG. 28 is basically the same as that of the image transmission system shown in FIG. 2 .

In the embodiment shown in FIG. 2, all the information needed to control such as the spatial resolution and temporal resolution of the coordinates of the point of consideration that the user is considering, spatial resolution, temporal resolution and transmission rate are sent as control information from the receiving device 2 is sent to sending device 1. On the contrary, according to the embodiment shown in Fig. 28, as will be explained below, the information of the consideration point of the image displayed on the display section 2-2 operated (clicked) by the user with the key section 2-3 of the receiving device 2 is used as Control information is transmitted (hereinafter, the information on the considered point may be referred to as click data).

When the sending device 1 receives the click data from the receiving device 2, the sending device 1 specifies the image that the user is considering from the image displayed from the receiving device 2 (the image is captured by the camera section 1-1 of the sending device 1) corresponding to the click data. An image area (object area of interest) of the given image area, and controls the amount of information of the image data sent to the receiving device 2 so as to change the spatial resolution and temporal resolution of the designated image area when a predetermined condition is satisfied.

FIG. 29 below shows a structural example of the transmission device 1 shown in FIG. 28 . In FIG. 29, parts similar to those in FIG. 3 are denoted by the same reference numerals, and their descriptions are omitted. In transmission device 1 shown in FIG. 29 , background image extraction section 1013, object extraction section 1014, and transmission processing section 1016 are provided instead of background image extraction section 13, object extraction section 14, and transmission processing section 16, respectively. Furthermore, the click data transmitted by the reception device 2 is supplied not only to the transmission processing section 1016 corresponding to the transmission processing section 16 but also to the preprocessing section 12 . Except for these points (points), the configuration of the transmission device 1 shown in FIG. 29 is substantially the same as that of the transmission device 1 shown in FIG. 3 . However, in the transmission device 1 shown in FIG. 29 , the output of the background image extraction section 1013 is not supplied to the object extraction section 1014 . In contrast, in the transmission device 1 shown in FIG. 3 , the output of the background image extraction section 13 is supplied to the object extraction section 14 . Furthermore, in the transmitting device 1 shown in FIG. 29 , the output of the object extraction section 1014 is supplied to the background image extraction section 1013 . In contrast, in the transmission device 1 shown in FIG. 3 , the output of the object extraction section 14 is not supplied to the background image extraction section 13 .

Click data is supplied to the preprocessing section 12 . In the AND processing section 12 , the click data is supplied to the object extracting section 1014 . The object extracting section 1014 extracts (specifies) an image region (object of interest region) considered by the user of the receiving device 2 from the image captured by the image input section 11, and converts the image corresponding to the extracted (designated) object of interest region The data is supplied to the transmission processing section 1016 . When there are a plurality of object-of-interest areas in which the user of the receiving device 2 considers the image taken by the image input section 11 , the object extraction section 1014 supplies the image data of the object-of-interest areas of the plurality to the transmission processing section 1016 . Furthermore, the image data of the object region of interest extracted by the object extraction section 1014 is also supplied to the additional information calculation section 15 .

As an example, the object of interest area considered by the user is an object such as a real object. Next, a case where the object extracting section 1014 extracts an object (hereinafter referred to as an object image) as an example of the object area of interest will be described. It should be noted that the object area of interest is not limited to one object. On the contrary, an object area of interest may be an image area instead of an object, an image area within an object, or a background image portion (described below). However, according to this embodiment, a case where one object of interest area is one object will be described as an example. The object extraction processing (interest object area designation processing) performed by the object extraction section 1014 will be described below.

The background image extracting part 1013 extracts a part of a background image with the image (that is, an image region other than the object region of interest; The image portion is referred to as a signal (hereinafter referred to as background image data) corresponding to the image portion, and the extracted background image data is supplied to the transmission processing section 1016 and the additional information calculation section 15. In this example, a planar image region whose activity is low (ie, has no image-like value) is processed as a background image. Of course, like images of no value, background images can be objects of no interest to the user. In this example, for the sake of brevity, the above-mentioned planar image area will be said to be a background image.

The additional information calculation section 15 detects a background image movement vector representing the movement of the background image corresponding to the background image data supplied from the background image extraction section 1013 (the movement of the background image corresponds to the movement along the photographing direction of the image input section 11). Furthermore, the additional information calculation section 15 detects an object movement vector representing the movement of the object corresponding to the image data of the object image supplied from the object extraction section 1014 (hereinafter, the image data is referred to as object image data). The additional information calculation section 15 supplies the detected motion vector as a part of the additional information to the transmission processing section 1016 . Further, the additional information calculation section 15 supplies information on the object such as the object position of the image (frame) captured by the image input section 11 to the transmission processing section 1016 corresponding to the object image data supplied from the object extraction section 1014 as additional information. and contour. In other words, when object extraction section 1014 extracts an object image from image data, object extraction section 1014 also extracts information about the object, such as the position and outline of the object, and supplies the information to additional information calculation section 15 . The additional information calculation section 15 outputs information on the object as additional information.

The transmission processing section 1016 encodes the object image data supplied from the object extraction section 1014, the background image data supplied from the background image extraction section 1013, and the additional information supplied from the additional information calculation section 15 corresponding to the click data supplied from the receiving device 2, so that when When the spatial resolution of the object image of the image displayed by the receiving device 2 is improved, the condition of the data rate of the transmission path 3 is satisfied. Thereafter, the transmission processing section 1016 multiplexes the encoded object image data, the encoded background image data, and the encoded additional information, and transmits the multiplexed data, frame rate information, etc. to the reception device 2 via the transmission path 3 .

The processing performed by the transmission device 1 shown in FIG. 29 will be described below with reference to the flowchart shown in FIG. 30 .

In step S91 , the transmitting device 1 inputs the image data obtained by the image input section 11 to the preprocessing section 12 .

Thereafter, the sending device 1 receives the click data sent from the receiving device 2 and inputs the click data to the preprocessing section 12 at step S92.

In step S93, the preprocessing section 12 having received the image data and the click data performs preprocessing for extracting background images, objects, and additional information, and converts the background image data, object image data, and additional information obtained in the preprocessing to It is supplied to the transmission processing section 1016.

At step S94 , the transmission processing section 1016 calculates the data amounts of the object image data, the background image data, and the additional information so as to satisfy the condition of the data rate of the transmission path 3 . Thereafter, the transmission processing section 1016 encodes object image data, background image data, and additional information corresponding to the data amount, and then multiplexes them. Thereafter, the transmission processing section 1016 transmits the multiplexed data and frame rate information to the reception device 2 via the transmission path 3 .

Thereafter, the flow returns to step S1. At step S1, the transmitting device 1 repeats similar processing.

FIG. 31 shows a structural example of the receiving device 2 shown in FIG. 28 . In FIG. 31, parts similar to those in FIG. 5 are denoted by like reference numerals, and their descriptions are omitted. In other words, in the receiving apparatus 2 shown in FIG. 31 , a combination processing section 1022 is provided instead of the combination processing section 22 . Furthermore, a click data input section 1024 and a click data transmission section 1025 are provided instead of the control information input section 24 and the control information transmission section 25 respectively. Except for these points, the structure of the receiving device 2 shown in FIG. 28 is basically the same as that of the receiving device 2 shown in FIG. 5 .

The multiplexed data transmitted by the transmission device 1 via the transmission path 3 is received by the reception processing section 21 . The reception processing section 21 separates the received multiplexed data into coded background image data, coded object image data, and coded additional information data, and decodes the separated data. Thereafter, the reception processing section 21 supplies the decoded background image data, object image data, and additional information to the combination processing section 1022 .

The combination processing section 1022 groups and decodes the background image data, object image data, and additional information, and supplies the combined image signal to the image output section 23 . Furthermore, the combination processing section 1022 controls the spatial resolution and temporal resolution of the combined image corresponding to the click data supplied from the click data input section 1024 .

The click data input section 1024 generates click data representing the click position (coordinate position) and click time corresponding to the operation of the key section 2-3 by the user. The key section 2-3 serves as a pointing means for specifying the coordinate position of an image displayed on the image output section 23 corresponding to the display section 2-2 of the receiving apparatus 2 (see FIG. 28). In other words, when the user clicks the button section 2-3 for an expected image position (object area of interest) of an image displayed on the image output section 23, the click data input section 1024 generates click data representing coordinate information of the clicked position and the clicked position. time. The click data generated by the click data input section 1024 is sent to the combination processing section 1022 and the click data sending section 1025 .

When click data transmission section 1025 receives click data from click data input section 1024 , click data transmission section 1025 transmits the click data to transmission device 1 via transmission path 3 .

Processing performed by the receiving apparatus 2 shown in FIG. 31 will be briefly described below with reference to the flowchart shown in FIG. 32 .

First, in step S101 , the reception device 2 receives multiplexed data transmitted by the transmission device 1 via the transmission path 3 .

In step S102, the reception processing section 21 separates the multiplexed data into coded background image data, coded object image data, and coded additional information data, and decodes the separated coded data. The decoded background image data, object image data, and additional information are sent to the combination processing section 1022 .

At step S103, in the reception device 2, the click data input section 1024 obtains click data corresponding to the click operation of the key section 2-3 by the user, and supplies the click data to the combination processing section 1022 and the click data transmission section 1025. Thus, the click data is sent from the click data sending section 1025 to the sending device 1 .

In step S104, the combination processing section 1022 combines an image corresponding to the background image data, object image data, and additional information supplied from the reception processing section 21 and the click data supplied from the click data input section 1024 and controls the spatial resolution and the spatial resolution of the combined image. Time resolution. The sending device 1 may put the click data sent by the receiving device 2 into the header information of the multiplexed data, and send the synthesized header information to the receiving device 2 . In this case, the combination processing section 1022 of the reception device 2 can obtain the click data from the header information. Thus, there is no need to supply the click data from the click data input section 1024 to the combination processing section 1022 .

In step S105, the image combined by the combining processing section 1022 is displayed on a liquid crystal display of the image output section 23 or the like.

Thereafter, the flow returns to step S101. At step S101, the receiving device 2 repeats similar processing.

FIG. 33 shows a real (actual) structural example of transmission processing section 1016 of transmission device 1 shown in FIG. 29 . In FIG. 33, parts similar to those in FIG. 7 are denoted by like reference numerals, and their descriptions are omitted. In other words, the configuration of the transmission processing section 1016 shown in FIG. 33 is basically the same as that of the transmission processing shown in FIG. 7 except that the transmission processing section 1016 shown in FIG. Section 1016 has the same structure.

In FIG. 33 , background image data, object image data, and additional information are supplied to the transmission processing section 1016 from the preprocessing section 12 shown in FIG. 29 . Background image data, object image data, and additional information are input to the coding section 31 and the control section 35 . The encoding section 31 encodes the supplied background image data, object image data, and additional information in the above-described manner, and supplies the resulting encoded data to the MUX 32 . The MUX 32 selects encoded background image data, encoded object image data, and encoded additional information data under the control of the control section 35 , and supplies the selected data to the transmission section 33 as multiplexed data. The transmission section 33 modulates the multiplexed data supplied from the MUX 32 into a downstream portion (downstream) corresponding to the transmission standard of the transmission path 3, and transmits the modulated data to the reception device 2 via the transmission path 3.

On the other hand, the control section 35 controls the output of the multiplexed data supplied from the MUX 32 so that the data rate supplied from the data amount calculation section 34 does not exceed the transmission rate of the transmission path 3 . Further, the control section 35 receives the click data sent from the receiving device 2 via the transmission path 3, and controls the MUX 32 to select and multiplex encoded data corresponding to the click data.

Next, transmission processing performed by the transmission processing section 1016 shown in FIG. 33 will be described with reference to the flowchart shown in FIG. 34 .

First, in step S111 , the control section 35 of the transmission processing section 1016 determines whether click data has been transmitted from the reception device 2 . When the determined result at step S111 indicates that the receiving device 2 has not transmitted click data (ie, the control section 35 has not received click data), the flow advances to step S112. In step S112, as in the case of step S22 shown in FIG. 10, the control section 35 controls the MUX 32 to select encoded background image data, encoded object data, and encoded additional information data, and multiplex the selected data so that the receiving device 2 can display an image with conventional temporal resolution.

Thereafter, the flow advances to step S113. At step S113 , the transmission processing section 1016 transmits the multiplexed data supplied from the MUX 32 from the transmission section 33 via the transmission path 3 . Thereafter, the flow returns to step S111.

When the determined result at step S111 indicates that the receiving device 2 has transmitted the click data (ie, the control section 35 has received the click data), the flow advances to step S114. At step S114, the control section 35 recognizes the coordinates (click position) and click time of a point of consideration that the user has specified with the key section 2-3 of the receiving device 2 corresponding to the click data.

Thereafter, at step S115, the control section 35 designates the interest object area considered by the user on the receiving device 2 side in a manner to be described below corresponding to the coordinates (click position) and click time of the considered point, designates the interest object area as its space A priority range of an image whose resolution is preferentially improved, and an image within the priority range and its additional information are detected. In this case, the image within the priority range is the object image. An image in the non-priority range is, for example, an image like a background image in the area of uninteresting objects.

Thereafter, at step S116, the control section 35 controls the MUX 32 to select an image in the priority range (object image), an image in the non-priority range (background image), and coded data of additional information, and multiplex them. In other words, when the control section 35 receives the click data from the receiving device 2, as in the case of step S26 shown in FIG. Resolution has been improved at the expense of.

Further, at step S116, the control section 35 controls the MUX 32 to insert high-resolution information as the position and size information of the priority range into the additional information selected as the multiplexed data. Thereafter, the flow advances to step S113.

At step S113 , the transmission device 33 transmits the multiplexed data output by the MUX 32 via the transmission path 3 . Thereafter, the flow returns to step S111.

As described above, in the transmission processing shown in FIG. 34 , processing similar to the processing shown in FIG. 10 is executed. Thus, when the user of the receiving device 2 continuously operates the click data input section 1024 for images containing the priority points on the consideration points (for example, he and she continuously designate the same consideration point), the room for improvement is preferentially transmitted. resolution data. Thus, the spatial resolution of images within the priority range containing the point of consideration is gradually improved. As a result, images within the priority range are displayed more clearly. In other words, an image (an object area of interest and an object image) considered by the user on the receiving device 2 side is displayed more clearly.

As described above, the transmission of the image data is controlled corresponding to the transmission rate of the transmission path 3 so that the spatial resolution and temporal resolution of the images (one object region of interest and one object image) within the priority range are within the range of resolution Variety. Thus, at a limited transmission rate, the spatial resolution of an object image corresponding to a point of interest displayed by the receiving device 2 can be further improved. In other words, since the spatial resolution of the object image within the priority range is improved at the expense of the temporal resolution of the image, the object image displayed on the receiving device 2 (i.e., spatial resolution) can be displayed more clearly at a limited transmission rate. resolution can be further improved).

FIG. 35 below shows an example of the structure of the combination processing section 1022 shown in FIG. 31 . In FIG. 35, the same parts as those shown in FIG. 13 are denoted by the same reference numerals, and their descriptions are omitted. In other words, the background image flag memory 74 is not provided in the combination processing section 1022 . What is provided in the combination processing section 1022 is a combination section 1077 instead of the combination section 77 . Also, what is supplied to the combination section 1077 is click data instead of control information. Except for these points, the structure of the combining processing section 1022 shown in FIG. 35 is basically the same as that of the combining section 22 shown in FIG. 13 .

Referring to FIG. 35 , the background image data output by the reception processing section 21 (see FIG. 31 ) is input to the background image writing section 71 . The object image data output by the reception processing section 21 is input to the object writing section 72 . The additional information output by the reception processing section 21 is input to the background image writing section 71 , the object writing section 72 and the combining section 1077 .

The background image writing section 71 continuously writes the supplied background image data into the background image memory 73 . However, in the embodiment shown in FIG. 35, the background image flag memory 74 of the embodiment shown in FIG. 13 is not provided. Thus, in FIG. 35, when the background image writing section 71 writes the background image data on the background image memory 73, the background image writing section 71 does not check a background image flag.

The combining section 1077 reads out a background image of the frame displayed at the current time (current frame) from the background image data stored in the background image memory 73 corresponding to the background image movement vector contained in the additional information, corresponding to the background image contained in the additional information. An object movement vector combines an object image stored in the object memory 75 with the background image, and supplies the combined image of the current frame to the display memory 78 .

Furthermore, when the combination section 1077 receives the click data from the click data input section 1024 shown in FIG. Among the click data, and the obtained object image data is supplied to the sub-window memory 79 .

Processing (combination processing) executed by the combination processing section 1022 shown in FIG. 35 will be described below with reference to the flowchart shown in FIG. 36 .

First, at step S121, the object writing section 72 writes object image data supplied from the decoding section 53 shown in FIG.

Thereafter, the flow proceeds to step S122. The object writing section 72 determines whether the additional information contains high-resolution information. When the determined result at step S122 shows that the additional information contains high-resolution information (that is, the user of the receiving device 2 has operated the key part 2-3, the click data has been sent to the sending device 1, and the sending device 1 is subsequently within the priority range When one image of the object image data having a high spatial resolution is sent), the flow proceeds to step S123. At step S123, the object writing section 72 sets the relevant object flag of the object flag memory 76 to "1".

In other words, when the transmitting device 1 has transmitted object image data having a high spatial resolution for images within the priority range, the object image data having a high spatial resolution is written in the object memory 75 at step S121. Thus, at step S123, the object flags of the pixels constituting the object with high spatial resolution are adjusted to "1".

Thereafter, the flow proceeds to step S124. At step S124 , the combining section 1077 reads out the object image data within the priority range from the object memory 75 , and writes the obtained object image data into the sub-window memory 79 .

In other words, when the determined result at step S122 indicates that the additional information contains high-resolution information, as described above, the user has operated the key portion 2-3, the click data has been transmitted to the transmission device 1, and the transmission device 1 subsequently Send object image data with high spatial resolution for an image within the priority range. The click data sent to the sending device 1 is also supplied to the combining section 1077 . When the combination part 1077 receives the click data, at step S124, the combination part 1077 agrees with the priority range corresponding to the coordinates and the click time of the point of consideration contained in the click data, and reads out from the object memory 75 that has been sent from the transmission device 1. Send an object with high spatial resolution within the priority range and write the obtained object to the sub-window memory 79 . In addition, as described above, when the header information sent by the sending device 1 includes click data, the combining part 1077 can identify the priority range according to the click data included in the header information.

Thereafter, the flow advances to step S125. At step S125, the combining section 1077 reads out the background image data of the current frame from the background image memory 73 corresponding to the background image movement vector contained in the additional information. Furthermore, the combining section 1077 reads out the object image data of the displayed current frame from the object memory 75 . Thereafter, the combination section 1077 combines the background image data of the current frame with the object image data that has been read out from the object memory 75 corresponding to an object movement vector contained in the additional information, and writes the combined image of the current frame to the display memory 78. In other words, the combining section 1077 writes the background image data to the display memory 78 and then overwrites the object image data to the display memory 78 . As a result, the combining section 1077 writes, to the display memory 78, the image data of the current frame in which the background image and the object image have been combined.

In the manner described above, the image data of the current frame written to the display memory 78 and the object image data written to the sub-window memory 79 are supplied to and displayed on the image input section 23 shown in FIG. 31 .

In contrast, when the determined result at step S122 indicates that the additional information does not contain high-resolution information (ie, the user of the receiving device 2 has not operated the key portion 2-3), the flow advances to step S125, skipping steps S123 and S124. As described above, in step S125, the combination section 1077 reads out the background image data of the current frame from the background image memory 73, reads out the required object image data from the object memory 75, and converts the current frame's image data corresponding to the additional information. The background image and the object image data that has been read out from the object memory 75 are combined. As a result, image data for the current frame is formed and written to the display memory 78 . Thereafter, the flow returns to step S121. At step S121, the combining section 1077 repeats similar processing.

According to the combination processing described above, in the same manner as the case described in conjunction with FIG. 15, an image with a high spatial resolution that the user considers as an object is displayed. The combination processing section 1022 shown in FIG. 35 does not have the background image flag memory 74 shown in FIG. 13 . Therefore, the background image writing section 71 shown in FIG. 35 always writes the supplied background image data to the background image memory 73 . Therefore, in the combination processing shown in FIG. 36 , the spatial resolution of the background image cannot be improved as in the cases described with reference to FIGS. 13 to 15 .

Next, a method of extracting an object image (object area of interest) corresponding to the click data supplied from the receiving device 2 will be explained. This method is executed by the object extraction section 1044 shown in FIG. 29 .

FIG. 37 shows a structural example of the object extraction section 1014 of the preprocessing section 12 shown in FIG. 29 .

In FIG. 37 , image data supplied from the image input section 11 shown in FIG. 29 is stored in the image memory 201 . The image data is read out from the image memory 201 and supplied to the common terminal of the still area and moving area determining section 203 , the object image extracting section 213 and the selection switch 207 . The image memory 201 stores image data of at least several frames required for still area and moving area determination by the still area and moving area determination section 203 , which is a downstream portion of the image memory 201 .

Furthermore, the click data transmitted by the receiving device 2 via the transmission path 3 is stored in the click data memory 202 . The click data is read from the click data memory 202 and supplied to the common terminal of the still area and moving area determination section 204 , the continuous click determination section 204 and the selection switch 206 . Click data storage 202 stores click data required for continuous click determination for a predetermined period of time (for example, more than 500 to 700 milliseconds) performed by continuous click determination section 204 which is a downstream portion of click data storage 202 .

The static area and the moving area determining part 203 determine whether the image area (for example, 16×16 pixels) of the local small block around the click position (coordinate value on the image) represented by the current click data sent by the receiving device 2 is a moving area or a moving area. static area. In other words, the still area and moving area determination section 203 obtains the difference between the image area of the current frame and the image area of one past frame, which is several times earlier than the current frame, for 16×16 pixels around the clicked position. frame (hereinafter, this past frame is referred to as a past frame). When the difference between frames is equal to or smaller than a predetermined threshold, the still area and moving area determination section 203 determines that the image area is a still area. On the contrary, when the difference is greater than a predetermined threshold, the still area and moving area determination section 203 determines that the image area is a moving area. When processing a color image, the still area and moving area determining section 203 obtains the difference between the image area of the current frame and the image area of the past frame for each of R, G, and B images of 16×16 pixels. When the average value of the absolute values of frame differences obtained for R, G, B is equal to or smaller than a predetermined threshold (for example, equal to or smaller than 10), the still area and moving area determination section 203 determines that the image area is a still area. On the contrary, when the average value is greater than a predetermined threshold, the still area and moving area determination section 203 determines that the image area is a moving area. When the moving area and still area determining section 203 determines that the image area is a still area, the still area and moving area determining section 203 determines that the current click data output from the click data memory 202 is a still click (a click in the still area). When the still area and moving area determining section 203 determines that the image area is a moving area, the still area and moving area determining section 203 determines that the current click data output from the click data memory 202 is a moving click (a click in the moving area). Thereafter, the still area and moving area determining section 203 sends to the process determining section 205 information representing a still click and a moving click as a result of the still area and moving area determination.

The continuous click determination section 204 determines whether or not the user of the receiving device 2 has continuously performed click operations corresponding to the click time of the click data transmitted from the receiving device 2 . In other words, the continuous click determination section 204 obtains the time difference (ie, the click time interval) between the click time of the current click data sent by the receiving device 2 and the click time of the previous click data. When the time difference is equal to or smaller than a predetermined threshold, the continuous click determination section 204 determines that the user has not performed a continuous click operation. When the continuous click determination section 204 determines that the user has performed a continuous click operation, the continuous click determination section 204 processes the current click data output from the click data memory 202 as continuous clicks. On the contrary, when the continuous click determination part 204 determines that the user has not yet performed a continuous click operation (that is, the time difference between the current click time and the previous click time is equal to or smaller than a predetermined threshold), the continuous click determination part 204 processes the click from the click as a non-continuous click. The current click data output by the data memory 202 . Thereafter, the continuous click determination section 204 transmits to the process determination section 205 information indicating continuous clicks or discontinuous clicks as a result of the continuous click determination.

The processing determination section 205 controls the selection switch 206 , the selection switch 207 and the selection switch 208 corresponding to the still area and moving area determination results of the still area and moving area determination section 203 and the continuous click determination results of the continuous click determination section 204 .

In other words, corresponding to the still area and moving area determination results and the continuous click determination results, when the current click data output by the click data memory 202 is a still click and a continuous click, the processing determination part 205 controls the selection switch 206 so that the click data memory The current click data output at 202 is sent to the stationary object connection processing section 211 . Furthermore, the processing determination section 205 controls the selection switch 207 so that the image data output from the image memory 201 is sent to the still image connection processing section 211 . However, the processing determination section 205 controls the selection switch 208 so that the previous click data (to be described later) output by the object extraction result memory 214, the object number (equivalent to the above-mentioned flags) assigned to the click data, and classified (identified) object) and the object image data corresponding to the object number are sent to the stationary object connection processing section 211.

In addition, corresponding to the static area and moving area determination results and continuous click determination results, when the current click data output by the click data memory 202 is a moving click and a continuous click, the processing determination part 205 controls the selection switch 206 so that the click data memory The current click data output at 202 is sent to the mobile object connection processing section 210 . Furthermore, the processing determination section 205 controls the selection switch 207 so that the image data output from the image memory 201 is sent to the mobile object connection processing section 210 . Furthermore, the processing determination section 205 controls the selection switch 208 so that the previous click data (to be described later) output from the object extraction result memory 214, the object number assigned to the click data, and the object image data corresponding to the object number Send to the mobile object connection processing section 210.

In addition, corresponding to the static area and the moving area determination result and the continuous click determination result, when the current click data output by the click data memory 202 is a static click and a continuous click (the time difference between the current click time and the previous click time is equal to or greater than the predetermined threshold), the process determination section 205 controls the selection switch 206 so that the current click data output by the click data memory 202 is sent to the object number allocation section 209 . Furthermore, the processing determination section 205 controls the selection switch 207 so that the image data output from the image memory 201 is sent to the stationary object connection processing section 211 . At this time, the selection switch 208 controls the selection switch 208 . The previous click data, object number, and object image data output by the object extraction result processor 214 are caused not to be sent to the stationary object connection processing section 211 (in this case, for example, the selection switch 208 is turned on).

In addition, corresponding to the static area and the moving area determination result and the continuous click determination result, when the current click data output by the click data memory 202 is a moving click and a discontinuous click (the time difference between the current click time and the previous click time is equal to or greater than the predetermined threshold), the processing determining part 205 controls the selection switch 206 so that the current click data output by the click data memory 202 is not sent to the object number assigning part 209. Furthermore, the processing determination section 205 controls the selection switch 207 so that the image data output from the image memory 201 is sent to the mobile object connection processing section 210 . At this time, the processing determination section 205 controls the selection switch 208 so that the previous click data, object number, and object image data output by the object extraction result memory 214 are not sent to the moving object connection processing section 210 (for example, the selection switch 208 is turned on).

The object number assignment section 209 assigns a new object number to the click data (will be described below) as a discontinuous click rather than a continuous click that is processed by the stationary object connection processing section 211 and the moving object connection processing section 210 by one connection processing ), and send the object number and the click data to the object number memory 212.

When the processing determination part 205 determines that the current click data output by the click data memory 202 is a moving click and a continuous click, the moving object connection processing part 210 determines whether the previous click data is a moving click, whether the image feature near the current click position is included in Within the feature of the moving object image area with the object number assigned to the previous click data, or whether they (the current click data and the previous click data) are the same. When the determination result is positive, the mobile object connection processing section 210 determines that the current click is a click on the same object image. Thus, the moving object connection processing section 210 performs a connection process of assigning the same object number as that of the previous click data to the current click data and sending the object number and the click data to the object number memory 212 .

When the determined result of the processing determination section 205 shows that the current click data output by the click data memory 202 is a still click and a continuous click, the still object connection processing section 211 determines whether the previous click is a still click, and whether the current click position is included in the In the area of the still object image assigned to the object number of the previous click data, or whether the current click position is close to the area. When the determination result of the stationary object connection processing section 211 is affirmative, the stationary object connection processing section 211 determines that the current click is a click for the same object as the previous click. Thus, the stationary object connection processing section 211 performs a stationary object connection process of assigning the same object number as the previous click data to the current click data and sending the object number and the click data to the object number memory 212 .

The object number memory 212 stores click data of a plurality of past frames to which object numbers have been allocated by the object number assignment section 209, the moving object connection processing section 210, and the stationary object connection processing section 211, and transmits the stored click data and the object number to the object image extraction section 213.

The object image extraction section 213 extracts a still object image, a moving object image, a background image, etc. from the image data supplied from the image memory 201 corresponding to the click data which has been supplied from the object number memory 212 and to which an object number has been assigned a plurality of past frames. , and supply the extracted result to the object extraction result memory 214.

In other words, the object image extraction section 213 obtains from the image portion of click data having a high click data density as a still click corresponding to the click data supplied from the object extraction section 213 and to which an object number has been assigned a plurality of past frames. A predominate object number. The object image extracting section 213 forms the shape of an object corresponding to the assignment of the click data assigned with the dominant object number, and extracts an image configuring the object as an object image from the image data.

Further, the object image extracting section 213 performs a pattern matching operation on an image of a frame near the click position to which the same object number is assigned in the click data determined to be a movement click, and performs motion compensation for the image corresponding to the matching result. . However, the object image extraction section 213 obtains the dominant object number of an image area having a high click density (ie, an image area adjusted by motion compensation) from an image data determined as a similar image area. The object image extracting section 213 forms the shape of an image corresponding to the assignment of the click data to which the dominant object number is assigned, and extracts an image configured with an object image from the image area.

Thereafter, the object image extracting section 213 designates an image portion having a low still click density and a low moving click density as a background image.

The object extraction result memory 214 stores object image data extracted by the object image extraction section 213, as well as click data, object numbers, and the like. The object extraction result memory 214 supplies the object image data to the background image extraction section 1013, the additional information calculation section 15, and the transmission processing section 1016 shown in FIG. 29 as necessary.

Next, a process of extracting an object image (object region of interest) considered by the user of the receiving device 2 from an image being photographed corresponding to the click data transmitted by the receiving device 2 will be described with reference to the flowchart shown in FIG. 38 . This processing is performed by the object extraction device 1014 shown in FIG. 37 .

First, in step S131, the image memory 201 stores image data of one frame input by the image input section 11 (frame image data input whenever transmitted). The image memory 201 stores image data of at least several frames required to perform still area and moving area determination processing at step S133.

At step S131, when the receiving device 2 transmits click data via the transmission path 3, the click data memory 202 stores the click data. The click data memory 202 stores click data necessary for continuous click determination processing (to be described below) performed at step S133 for at least a predetermined period of time (for example, up to 500 to 700 milliseconds).

Thereafter, the flow advances to step S132, where the object extraction section 1014 determines whether or not click data that has been transmitted by the receiving device 2 and has not been processed is stored. When the determined result at step S132 indicates that the click data memory 202 does not store unprocessed click data, the flow returns to step S131. At step S131, the object extraction section 1014 waits for image data and click data to be input. In contrast, when the determined result at step S132 indicates that the click data memory 202 stores unprocessed click data, the flow proceeds to step S133. At step S133, the object extracting section 1014 designates the oldest click data that has not been processed as the current click data. The still area and moving area determination section 203, the continuous click determination section 204, and the processing determination section 205 execute still area and moving area determination processing and continuous click determination processing for this current click data.

In other words, at step S133, the still area and moving area determining section 203 uses the click position information (image coordinate value) contained in the current click data transmitted from the receiving device 2 to perform the determination processing of the still area and the moving area to determine the Whether an image area of a local patch around a click location is a moving area or a static area.

Next, the still area and moving area determination processing at step S133 shown in FIG. 38 performed by the still area and moving area determining section 203 will be described more practically. As shown in the flowchart shown in FIG. 39, at step S141, the still area and moving area determination section 203 reads out several frames of image data and click data from the image memory 201 and click data memory 202, respectively. When processing a color image, as shown in FIG. 40(A), the still area and moving area determining section 203 reads out image data for several frames of R (red), G (green), and B (blue). In addition, the still area and moving area determining section 203 reads out image data of several past frames including one frame corresponding to the current click data and images corresponding to the frames corresponding to the click data from the image data memory 201 and the click data memory 202, respectively. Click data for clicks performed.

Thereafter, in step S142, for a local small block consisting of 16×16 pixels in the horizontal direction and the vertical direction around the click position of the current click data, the still area and the moving area determining section 203 calculate an image of the current frame The difference between the region and the past frame that is several frames ahead of the current frame (the past frame is called the past frame). When a color image is processed, at step S142, as shown in FIGS. 40(B) and 40(A), the still area and moving area determination section 203 obtains a 16×16 pixel area for each of R, G, and B Compose the difference between frames of an image and obtain the average of the absolute values of the differences for each of R, G and B.

Thereafter, at step S143, the still area and moving area determination section 203 determines whether or not the inter-frame difference calculated at step S142 is equal to or smaller than a determined threshold. Thereafter, the flow advances to step S144. When the difference between frames is equal to or smaller than a predetermined threshold, the still area and moving area determining section 203 determines that a small block (the small block may be called a current block) containing the click position of the current click data is a still area. On the contrary, when the difference between frames is greater than a predetermined threshold, the still area and moving area determination section 203 determines that the current block is a moving area. Furthermore, when the determination result of the still area and moving area determining section 203 indicates that the current block is a still area, the still area and moving area determining section 203 designates the click data corresponding to the image area of the current block as a still click. On the contrary, when the determination result of the still area and moving area determining section 203 indicates that the current block is a moving area, the still area and moving area determining section 203 designates the image area click data corresponding to the current block as a moving click. The still area and moving area determination section 203 outputs the determination result as the still area and moving area determination result.

In the case of processing a color image, in step S144, as shown in FIG. 40(D), when the absolute value of the interframe difference for each block of 16×16 pixels for each of R, G, and B When the average value of is less than or equal to a predetermined threshold (for example, "10"), the still area and moving area determination section 203 adjusts the predetermined flag to, for example, "0". On the contrary, when the average value is larger than the predetermined threshold, the still area and moving area determination section 203 adjusts the predetermined flag to, for example, "1". At step S144, as shown in FIG. 40(E), when all the flags of R, G, and B for the current block of 16×16 pixels are “0”, the still area and moving area determination section 203 determines The current block is a still area, and the click data corresponding to the image area of the current block is designated as a still click. On the contrary, when one of the flags is "1", the still area and moving area determination section 203 determines that the current block is a moving area and specifies the click data corresponding to the image area of the current block as a moving click. The still area and moving area determination section 203 outputs information of a still click or a moving click as a result of the still area and moving area determination.

Returning to FIG. 38 , at step S133, the continuous click determination section 204 performs continuous click determination processing to determine whether the click operation performed by the user of the receiving device 2 is a continuous click operation corresponding to the click time included in the click data sent by the receiving device 2. Click Actions.

Next, the processing of the continuous click determination section 204 performed at step S133 shown in FIG. 38 will be described. According to the flowchart shown in FIG. 41, at step S151, the click data memory 202 reads out the click data stored in the click data memory 202.

Thereafter, at step S152, the continuous click determination section 204 calculates the time difference (click interval) between the click time of the current click data sent from the receiving device 2 and the click time (previous time) of the previous click data.

Then, at step S153, the continuous click determination section 204 determines whether the time difference is equal to or smaller than a predetermined threshold. When the determined result at step S153 indicates that the time difference is equal to or smaller than the predetermined threshold, the continuous click determination section 204 determines that the current click data is a continuous click. On the contrary, when the determined result at step S153 indicates that the time difference is greater than the predetermined threshold, the continuous click determination section 204 determines that the current click data is not a continuous click. Thereafter, the flow proceeds to step S154. At step S154, the continuous click determination section 204 outputs information representing continuous clicks or discontinuous clicks as a result of the continuous click determination.

In other words, when the determined result at step S153 indicates that the current click data is continuous clicks, the user of the receiving device 2 often performs click operations on one object image continuously. This is because, when the user of the receiving device 2 requests the transmitting device 1 to transmit object image data (data in the object region of interest) with a high spatial resolution, he or she tends to continuously click on the object whose spatial resolution the user wants to improve. An object image part (object region of interest). Thus, when the determination result of the continuous click determination section 204 indicates that the current click operation is a continuous click operation, the continuous click determination section 204 designates the current click data as a continuous click. On the contrary, when the determination result of the continuous click determination part 204 shows that the current click operation is not a continuous click operation (that is, the time difference between the click time of the current click and the click time of the previous click is equal to or greater than a predetermined threshold), the continuous click determination part 204 designates the current click data as a non-sequential click. The continuous click determination section 204 outputs a continuous click determination result.

Returning to Fig. 38, on step S133, when the determined result of static area and moving area determination section 203 and continuous click determination section 204 shows that the current click data is a static click and continuous click, the processing determination section 205 controls the selection switch in the above-mentioned manner 206 , selection switch 207 and selection switch 208 . Thus, at step S135, the still object connection processing section 211 executes a still image connection process. When the determination result indicates that the current click data is a moving click and a continuous click, the process determining section 205 controls the selection switch 206, the selection switch 207, and the selection switch 208 in the above-described manner. Thus, at step S136, the mobile object connection processing section 210 executes a mobile object connection process. When the determination result indicates that the current click data is a discontinuous click, the process determination section 205 controls the selection switch 206, the selection switch 207, and the selection switch 208 in the above-described manner. At step S136, the mobile object connection processing section 210 executes mobile object connection processing. When the determined result indicates that the current click data is a discontinuous click, the process determination section 205 controls the selection switch 206 in the above-described manner. Thus, at step S134, the object number assigning section 209 executes new object number assigning processing.

In other words, when the determined result at step S133 indicates that the current click data is a discontinuous click, the flow returns to step S134. At step S134, the object number assignment section 209 assigns a new object number to the current click data. Thereafter, the flow returns to step S131.

Actually, as shown in FIG. 42(A), when an object number assigned to the previous click data CL1 represented by the solid line X is, for example, "0", if represented by the dotted line X mark in Fig. 42(A), When the current click data CL2 (that is, the click data that does not specify an object number) is determined to be a discontinuous click, the object number assigning section 209 assigns a new object number to the object marked by the solid line X in FIG. 42(B). represents the current click data CL2 (in this example, the new object number is "1").

On the contrary, when the determined result at step S133 indicates that the current click data is a continuous click and a still click, if the previous click is a still click, and the current click position is contained in a corresponding object number assigned to the previous click data. In the image area, or the current click position is close to the image area, the stationary object connection processing section 211 judges that the current click is a click for the same object image as the previous click. Thus, at step S134, the still object connection processing section 211 executes still image connection processing for assigning the same object number as that of the previous click data to the current click data.

In other words, as shown in the flowchart shown in FIG. 43, at step S161, the still object connection processing section 211 determines whether the previous click data is a continuous click and a still click. When the determined result at step S161 indicates that the previous click data are continuous clicks and still clicks, the flow proceeds to step S162. In contrast, when the determined result at step S161 indicates that the previous click was not a continuous click and a still click, the flow advances to step S164.

When the determined result at step S161 shows that the previous click data is not a continuous and still click, at step S164, the still object connection processing section 211 follows the same manner as described in connection with FIGS. 42(A) and (B). Assign a new object number to the current click data. Thereafter, the flow advances to step S137 shown in FIG. 38 .

On the contrary, when the determined result at step S161 indicates that the previous click data is a continuous click and a still click, the flow returns to step S162. At step S162, the stationary object connection processing section 211 obtains the spatial distance between the current click position and the image area corresponding to the object number assigned to the previous click data. When the current click position is included in the image area corresponding to the object number assigned to the previous click data, or when the current click position is close to the image area, the still object connection processing section 211 determines that the current click data is related to the previous click data. Click data of the same object image. On the contrary, when the current click position is not included in the image area corresponding to the object number assigned to the previous click data, and the current click position is far from the image area, the still object connection processing section 211 determines that the current click data is the same as the previous click data. A click data of a different object image is clicked. When the determined result at step S162 indicates that the current click data is that of the same object image as the previous click, the flow proceeds to step S163. In contrast, when the determined result at step S162 indicates that the current click data is click data different from the object image of the previous click, the flow returns to step S164.

When the determined result at step S162 indicates that the current click data is click data different from the object image of the previous click, the flow returns to step S164. In step S164, the stationary object connection processing section 211 assigns a new object number to the current click data. Thereafter, the flow advances to step S137 shown in FIG. 38 .

On the contrary, when the determined result at step S162 shows that the current click data is the click data of the same object image as the previous click, at step S163, the still object connection processing section 211 executes still object connection processing for combining the same object image with the previous click. The same object number as the click data is assigned to the current click data.

Actually, as shown in FIG. 42(C), when an object number assigned to the previous click data CL1 represented by the solid line X mark is "0", if the dotted line X shown in FIG. 42(C) The current click mark CL2 indicated by the mark (that is, an object number is not assigned to the current click data) is determined as a continuous click and a still click, the previous point is a still click and the current click position is included in the In the image area corresponding to an object number, or the current click position is close to the image area, the still object connection processing section 211 assigns the same object number ("1" in this example) as the previous click data to the image area identified by the image. The current click data CL2 indicated by the solid line X mark shown in 42(D).

The stationary object connection processing section 211 assigns the same object number as that of the previous click data to the current click data. The flow advances to step S137 shown in FIG. 38 .

When the determination result on the step S133 shows that the current click data is a continuous click and a moving click, the previous click is a moving click, and image features near the current click position are included in the image area corresponding to the object number assigned to the previous click Among the features of (16×16 pixels), or when the former is close to the latter, the moving object connection processing section 210 determines that the click is a click of the same object image as the previous click. At step S136, the moving object connection processing section 210 executes moving object connection processing for assigning the same object number as that of the previous click data to the current click data.

In other words, when the determined result at step S133 indicates that the current click data is a continuous click and a moving click, then as shown in FIG. and mobile clicks. When the determined result at step S171 indicates that the previous click data is a continuous click and a moving click, the flow proceeds to step S172. In contrast, when the determined result at step S171 is not the continuous click and the moving click, the flow proceeds to step S174.

When the determined result at step S171 indicates that the previous click data is not a continuous click and a moving click, the flow proceeds to step S174. In step S174, the mobile object connection processing section 210 assigns a new object number to the current click data in the same manner as described in connection with Figs. 41(A) and (B). Thereafter, the flow advances to step S137 shown in FIG. 38 .

When the determined result at step S171 indicates that the previous click data is a continuous click and a moving click, the flow proceeds to step S172. In step S172, the moving object connection processing section 210 obtains features of the image data (16*16 pixels) around the current click position, and features of the image area corresponding to the object number assigned to the previous click. When the feature of the image area near the current click position is included in the feature of the image area corresponding to the object number assigned to the previous click data, and the former is close to the latter, the reception processing section 210 determines that the click is related to the previous click. Clicks on the same object image. On the contrary, when the characteristics of the image region near the current click position are not included in the characteristics of the image region corresponding to the object number assigned to the previous click data, or the former is far away from the latter, the reception processing section 210 determines that the current click data is Click data for an object image that is different from the previous click. In this case, the feature of the image area is, for example, a color (average color, typical color, or the like), a bar graph, or a pattern of a local area (16*16 pixels) near a clicked position. When the same object number is assigned to multiple movement hits, it means that one object is tracked in these hit data. When the determined result at step S172 indicates that the current click data is click data for the same object image as the previous click, the flow proceeds to step S173. In contrast, when the determined result at step S172 indicates that the current click data is click data for an object different from the previous click, the flow proceeds to step S174.

When the determined result at step S172 indicates that the current click data is click data for an object image different from the previous click, the flow proceeds to step S174. At step S174, the moving object connection section 210 assigns a new object number to the current click data in the above-mentioned manner. Thereafter, the flow advances to step S137 shown in FIG. 38 .

When the determined result at step S172 indicates that the current click data is click data for the same object image as the previous click, the flow proceeds to step S173. In step S173, the moving object connection processing section 210 assigns the same object number as that of the previous click data to the current click data.

Actually, when an object number assigned to the previous click data CL1 represented by the solid line X shown in FIG. 42(E) is, for example, "0", if the The current click data CL2 is determined as a continuous click and a moving click, and the feature of the image near the current click position is included in the feature of the object image corresponding to the object number assigned to the previous click, or the former is close to the latter, Then the moving object connection processing section 210 assigns the same object number ("0" in this example) as the previous click data to the current click data CL2 indicated by the solid line X mark shown in FIG. 42(F).

In step S173, after the moving object connection processing section 210 assigns the same object number as the previous click to the current click data, the flow proceeds to step S137.

When the flow advances to step S137 from step S135 shown in FIG. Still object images, moving object images, and other background images are extracted from the image data of past several frames stored in the image memory 20 . In other words, it seems that a still object image is contained in an image portion having a high still click data density. At this time, the object image extracting section 213 obtains the still click data density corresponding to the past several frames to which the object number is assigned, obtains the dominant object number of the image portion having a high still click density, forms and assigns the dominant object number A distribution of the click data corresponds to the shape of the object, and an image of the object configured according to the still object image is extracted from the image data.

When the flow advances from step S136 to step S137, the object image extraction section 213 performs a pattern matching operation for images of frames near the click position of the movement click data to which the same object number is assigned, and performs movement compensation of the image corresponding to the matching result. , obtain the dominant object number of the pattern-matched image region with high click density, form an object shape corresponding to the distribution of click data assigned the dominant object number, and extract from the image data the shape of the object by moving the object image an image.

At step S137, the object image extracting section 213 processes an image portion having low still click density or low moving click data as the current background image. In other words, the object image extracting section 213 processes an image portion extracted from image data other than the still object image and the moving object image as a background image.

The processing at step S137 will be described in detail below with reference to the flowchart shown in FIG. 45 . First, at step S181, the object image extracting section 213 captures several past frames of click data assigned object numbers and image data corresponding thereto. Thereafter at step S182, the object image extracting section 213 classifies the click data into a still click and a moving click. When the flow advances from step S135 shown in FIG. 38 to step S137, the flow advances from step S182 shown in FIG. 45 to step S184. In contrast, when the flow advances from step S136 shown in FIG. 38 to step S137, the flow advances from step S182 shown in FIG. 45 to step S184.

When the flow advances from step S135 shown in FIG. 38 to step S137, the flow advances from step S182 shown in FIG. 45 to step S184. At step S184, the object image extracting section 213 obtains a still click data density of each still click assigned an object number for each block of 16*16 pixels.

Thereafter, at step S185, the object image extracting section 213 determines the value of the still click indicated by the X mark shown in FIG. Whether the static click density is equal to or greater than a predetermined value.

In an image transmitted to the receiving device 2, an image portion with a high still click density tends to contain a still object image. Thus, when the still click density of a specific block is equal to or greater than the predetermined value, the flow proceeds to step S186. On the contrary, when the still click density of a specific block is less than the predetermined value, the flow proceeds to step S190.

In step S186, when the still click density of the block exceeds a predetermined value, as shown in FIG. 46(E), the object image extracting section 213 obtains the most dominant object number from the object numbers assigned to the click data of the block. Thereafter, the object image extraction section 213 combines the blocks (BK0, BK2, BK4, and BK5) corresponding to the dominant object numbers shown in FIG. 46(B) and configures the object. The object image extraction section 213 extracts an image of an object configured as a still object image from the image data. After step S186, the flow proceeds to step S138 shown in FIG. 38 .

On the other hand, when the flow advances from step S136 shown in FIG. 38 to step S137, the flow advances from step S182 shown in FIG. 45 to step S183.

At step S183, as shown in FIG. 46(C), the object image extracting section 213 assigns the same click data object number as that of the moving click indicated by the X mark shown in FIG. A pattern matching operation is performed on the images of the past frames, and motion compensation is performed on the images corresponding to the matching results.

Thereafter, at step S187, the object image extracting section 213 obtains a moving click density in the pattern matching image area.

Thereafter, at step S188, the object image extraction section 213 determines whether or not the movement click density of the movement clicks of the image indicated by the X mark shown in FIG. 46(D) is equal to or greater than a predetermined value.

An image portion that is motion compensated and has a high motion click density often contains a moving object image. Thus, when the movement click density of an image area subjected to movement compensation is equal to or greater than the predetermined value, the flow proceeds to step S189. On the contrary, when the motion click density of an image area subjected to motion compensation is smaller than the predetermined value, the flow proceeds to step S190.

At step S189, the object image extracting section 213 obtains the most dominant object number from object numbers assigned to the click data for an image region having a moving click density equal to or greater than the predetermined value. Thereafter, as shown in FIG. 46(D), the object image extracting section 213 combines the blocks (BK3 and BK6) corresponding to the dominant object numbers, and configures the object. Thereafter, the object image extracting section 213 extracts an image configured as a moving object image from the image data. After step S189, the flow proceeds to step S138 shown in FIG. 38 .

At steps S185 and S188, when the click density is less than the predetermined value, the flow proceeds to step S190. At step S190, the object image extracting section 213 processes an image portion having a low still click density or a low moving click density as a background image area of the current image. In other words, the object image extracting section 213 processes the image portion other than the still object image and the moving object image that has been extracted from the image data as a background image. After step S190, the flow advances to step S138 shown in step FIG. 38 .

After the object image extracting section 213 has extracted a still object image, a moving object image, and a background image from the image data, the flow advances to step S138 shown in FIG. 38 . At step S138, the object image extraction section 213 determines whether the object extraction process is completed. When the determined result at step S138 is negative, the flow advances to step S131. When the determined result at step S138 is affirmative, the object extraction section 213 completes the object extraction process.

In the processing described above, object extraction section 1014 of transmitting device 1 shown in FIG.

In the embodiment shown in FIG. 28, planar image regions whose activity is small (ie, have no image value) are treated as background images. For this background image, the spatial resolution is not improved. On the other hand, a background image may be extracted corresponding to the click data sent from the receiving device 2 in order to improve its spatial resolution.

In this case, the background image extraction section 1013 shown in FIG. 29 extracts a background image corresponding to the click data transmitted by the reception device 2 . The transmission processing section 1016 transmits the background image so that its spatial resolution is improved in the same manner as that of the object image is improved.

In this case, as shown in FIG. 47, the background image flag memory 74 shown in FIG. 13 is added to the combination processing section 1022 shown in FIG. The structure of the combination processing section 1022 shown in FIG. 47 is the same as that of the combination processing section 1022 shown in FIG. 35 except for the background image flag memory 74 . Like the structure shown in FIG. 13, in the structure shown in FIG. 47, when the background image writing section 17 writes a background image of an object with a high spatial resolution into the background image memory 73, corresponding to the constituent background image The background image flag stored at an address of the background image flag memory 74 changes from "0" to "1" for each pixel. In other words, when the background image writing section 71 writes background image data to the background image memory 73 , the background image writing section 71 checks the background image flag memory 74 . When the background image flag is "1" (that is, when the background image memory 73 has stored the background image data with high spatial resolution), the background image writing section 71 does not write the background image data with low spatial resolution into background image memory 73 . Whenever background image data is supplied to the background image writing section 71 , the background image data is written to the background image writing section 71 . However, the background image writing section 71 does not write the background image data having a low spatial resolution to the background image memory 73 when the background image memory 73 already stores background image data having a high spatial resolution. Thus, whenever the background image data with high spatial resolution is supplied to the background image writing section 71 , the number of background images with high spatial resolution is increased in the background image memory 73 .

In this instance, when the combination section 1077 receives click data from the click data input section 1024 shown in FIG. These data contain the coordinate position of a considered point contained in the click data, and the obtained data is supplied to the sub-window memory 79 .

A technique for determining (approving) changes in object-of-interest areas of the user of the receiving device 2 and classifying each object-of-interest area will be described below.

Various types of images are analyzed to determine whether object areas of interest to the user have changed and whether each object area of interest is classified. This analysis revealed the following results.

First, the interest object area of people (users) is a valuable area unit (eg, an object).

Second, when the user's interest object changes, each area unit with value changes.

Third, when the user's object of interest changes, the input time period required to specify the user's object of interest tends to become longer.

Fourth, when the user's object of interest changes, the spatial distance between the input positions where the user designates an object of interest area (for example, click operation) tends to become relatively longer.

Thus, FIG. 48 shows a structural example of the sending device 1, which obtains the input time interval and the input position distance corresponding to the click data input by the user of the receiving device 2, determines whether the user's interest object area of the receiving device 2 is due to consideration Changes to analysis results (1) to (4), and classify object-of-interest regions. The transmission device 1 shown in FIG. 48 is a modification of the transmission device 1 shown in FIG. 28 . For the sake of brevity, in Fig. 48, parts similar to those in Fig. 29 are denoted by like reference numerals, and descriptions thereof are omitted. In other words, the configuration of the transmission device 1 shown in FIG. 48 is basically the same as that of the transmission device 1 shown in FIG.

According to the embodiment shown in FIG. 48 , the variation determination and classification section 240 is provided in the preprocessing section 12 . On the other hand, the variation determination and classification section 240 may be provided in the object extraction section 1014 or the background image extraction section 1013 of the preprocessing section 12 . On the other hand, the change determination and classification part 240 can also be set independently from the preprocessing part 12 .

In this embodiment, a position indicated by the click data is a point of consideration that the user is considering. A point of consideration of the user may not be click data, but the point of consideration of the user may be recognized by detecting the viewing direction of the user.

FIG. 49 shows an example of the structure of the change determination and classification section 240 shown in FIG. 48 .

The input image storage section 231 temporarily stores image data output by the image input section 11 . The input image storage section 231 supplies image data to the peripheral area extraction section 233 and the still area and moving area determination section 234 as necessary.

The click data obtaining section 230 temporarily stores the data transmitted by the receiving device 2 via the transmission path 3, and supplies the stored click data to the surrounding area extracting section 233, the stationary area and the moving area determining section 234, the input time interval calculating section 237, and the input time interval calculating section 237. The location distance calculation part 238 .

In this embodiment, like the image memory 201 shown in FIG. 37, an input image storage section 231 may be provided. Furthermore, like the click data storage 202 shown in FIG. 37, a click data storage section 232 may be provided.

The click peripheral area extraction section 233 extracts an image area corresponding to the click data supplied from the click data storage section 232 from the image data supplied from the input image storage section 231 (the image area is, for example, a local small block near the click position; hereinafter, The image area is referred to as the click surrounding area). The data of the clicked surrounding area extracted by the clicked surrounding area extraction section 233 is sent to the clicked surrounding area storage section 235 . After the data is stored in the click peripheral area storage section 235 , the data is sent to the interest object area classification section 236 .

In addition, the still and moving area determining section 234 performs still using the difference between frames in the same manner as the embodiment shown in FIG. 37 corresponding to the image data supplied from the input image storage section 231 and the click data supplied from the click data storage section 232. Area and moving area determination processing.

Click surrounding area extraction processing and still area and moving area determination processing can be performed by the same processing as described in connection with FIG. 38 . Therefore, a detailed description of these processes is omitted. In the embodiment shown in FIG. 49, the result of the click data being determined as a stationary click or a moving click is output as a stationary click or moving click determination in the same manner as in the above-described embodiment. Also, for example, a result that the clicked peripheral area is determined to be a stationary area or a moving area may be output. According to the embodiment shown in FIG. 49, for the sake of brevity, a case where a stationary click or a moving click is output as a result of determination of a stationary area and a moving area will be described.

The still area and moving area determination results of the still area and moving area determining section 234 are sent to the input time interval calculating section 237 and the input position distance calculating section 238 .

When the still area and moving area determination results indicate that the click data is a still click, the input time interval calculation section 237 calculates the time interval between the input time of the previous click and the input time of the current click. In this case, the time interval is calculated regardless of whether, for example, a moving click occurs between the input time of the previous stationary click and the input time of the current stationary click. The data of the time interval calculated by the input time interval calculation section 237 is sent to the interest change determination section 239 .

When the still area and moving area determination results indicate that the click data is a still click, the input position distance calculating section 238 calculates the distance between the input click position (coordinate position) of the previous still click and the input click position (coordinate position) of the current still click. space distance. In this case, the spatial distance is calculated regardless of whether, for example, an input position of a moving click exists between the input position of the current stationary click and the input position of the previous stationary click. Data of the spatial distance calculated by the input position distance calculation section 238 is sent to the interest change determination section 239 .

When the static area and the moving area determination result click data are static clicks, the interest change determination section 239 determines whether the user's interest object changes corresponding to the time interval calculated by the input time interval calculation section 237 and the spatial distance calculated by the position distance calculation section 238. In other words, the interest change determination section 239 performs predetermined weighting processing of time intervals and spatial distances, determines whether the weighted time interval exceeds a predetermined threshold (time), and determines whether the weighted spatial distance exceeds a predetermined threshold (distance). When the weighted time interval exceeds a predetermined threshold and/or the weighted spatial distance exceeds a predetermined threshold, the interest change determination section 239 determines that the user's object of interest has changed. The object of interest determination section 239 sends the object of interest change determination result to the object of interest area classification section 236 .

When the determined result of the interest change determination section 239 shows that the user's object of interest has not changed, the interest object area classification section 236 determines that the clicked peripheral area of the current static click is contained in the clicked peripheral area corresponding to the previous (past) static click In the same image area as the object of interest of the current static click, the click surrounding area of the current static click is classified as the same interest object area as the click surrounding area of the previous static click (for example, the same category number is assigned to the click surrounding area of the current static click region), and output the classification results. In other words, when classifying object regions of interest for each object, the same object number is assigned in the same manner as in the above-described embodiment.

On the contrary, when the determined result of the interest change determination part 239 shows that the user's interest object has changed, the interest object area classification part 236 determines that the click surrounding area of the current static click is not included in the click surrounding area of the previous (past) static click. In the area of the object of interest corresponding to the area, the stored data of the clicked surrounding image currently clicked is output, and the data stored in the clicked surrounding area storage part 235 is reset. Thereafter, the interest object area classification part 236 classifies the clicked surrounding area of the current static click as an interest object area different from the clicked surrounding area of the previous static click (for example, assigning a different category number to the click of the current static click surrounding area). In other words, when classifying an object-of-interest area for each object, a new different object number is assigned in the same manner as in the above case.

When the still area and moving area determination results indicate that the click data is a moving click, the input time interval calculating section 237 also calculates the time interval between the input time of the previous moving click and the input time of the current moving click. In this case, the time interval is calculated regardless of whether, for example, a stationary click occurred between the input time of the current movement click and the input time of the previous movement click. The data of the time interval calculated by the input time interval calculation section 237 is sent to the interest change determination section 239 .

Conversely, when the still area and moving area determination results indicate that the click data is a moving click, the input position distance calculation section 238 also calculates the spatial distance between the input click position of the previous movement click and the input click position of the current movement click. In this case, the spatial distance is calculated regardless of, for example, whether there is an input position of a still click between the input position of the current movement click and the input position of the previous movement click. Data of the spatial distance calculated by the input position distance calculation section 238 is sent to the interest change determination section 239 .

In addition, in the case where the still area and moving area determination results indicate that the click data is a moving click, the interest change determination section 239 determines correspondingly to the time interval calculated by the input time interval calculation section 237 and the spatial distance calculated by the position distance calculation section 238 Whether the user's object of interest has changed. In other words, the interest change determination section 239 performs predetermined weighting processing of the time interval and the space distance, determines whether the weighted time interval exceeds a predetermined threshold (time), and determines whether the weighted space interval exceeds a predetermined threshold (distance). When the weighted time interval exceeds a predetermined threshold and/or the weighted spatial distance exceeds a predetermined threshold, the interest change determination section 239 determines that the user's object of interest has changed. The object of interest change determination result of the interest change determination section 239 is sent to the object of interest area classification section 236 .

When the determination result of the interest change determination part 239 shows that the interest object of the user has not changed, the interest object area classification part 236 determines that the clicked surrounding area of the current mobile click is contained in the clicked surrounding area corresponding to the previous (past) mobile click In the same image area as the object of interest area of the current mobile click, the clicked surrounding area of the current mobile click is classified as the same interest object area as the clicked surrounding area of the previous (old) mobile click (that is, the same category number is assigned to the current Click the surrounding area of the mobile click), and output the classification result. In other words, when classifying an object area of interest for each object, the same object number is assigned.

On the contrary, when the determined result of the interest change determination section 239 shows that the user's interest object changes, the interest object area classification section 236 determines that the clicked surrounding area of the current mobile click is not included in the clicked surrounding area of the previous (past) mobile click. In the corresponding interest object area, the stored data of the clicked surrounding image of the current movement click is output, and the data already stored in the clicked surrounding area storage section 235 is reset. Thereafter, the interest object area classification part 236 classifies the clicked surrounding area of the current mobile click as an interest object area different from the clicked surrounding area of the previous mobile click (for example, assigning a different category number to the clicked area of the current mobile click surrounding area). In other words, when classifying an object region of interest for each object, a new distinct object number is assigned.

The processing of the change determination and classification section 240 shown in FIG. 49 will be described below with reference to the flowchart shown in FIG. 50 .

In step S201 , the change determination and classification section 240 receives image data from the image input section 11 and click data input by the user of the receiving device 2 .

Thereafter, at step S202 , the image data supplied from the image input section 11 is stored in the input image storage section 231 . The click data obtained by the click data obtaining section 230 is stored in the click data storage section 232 .

Thereafter, at step S203 , the click peripheral area extracting section 233 extracts an image area (click peripheral area) corresponding to the click data from the click data of the image which has been stored and read out from the input image storage section 231 . At step S204, the clicked surrounding area storage section 235 stores the data of the clicked surrounding area which has been extracted.

Thereafter, in step S205, the still area and moving area determination area 234 performs still area and moving area determination processing using the inter-frame difference values in the above-described manner.

At step S205, when the determined result at step S205 indicates that the click data is a still click, the flow proceeds to step S206. In contrast, when the determined result at step S205 indicates that the click data is a movement click, the flow proceeds to step S212.

When the determined result at step S205 indicates that the click data is a still click, the flow proceeds to step S206. At step S206, the input time interval calculation section 237 calculates the time interval between the input time of the previous stationary click and the input time of the current stationary click. This time interval is calculated regardless of whether, for example, there is a moving click between the input time of the current stationary click and the input time of the previous stationary click.

Thereafter, at step S207, the input position distance calculation section 238 calculates the spatial distance between the input click position (coordinate position) of the previous stationary click and the input click position (coordinate position) of the current stationary click. In this case, the spatial distance is calculated without considering, for example, that an input position of a moving click exists between the input position of the current stationary click and the input position of the previous stationary click.

At step S208, the object of interest determination section 239 determines whether the object of interest of the user changes corresponding to the time interval calculated at step S206 and the spatial distance calculated at step S207. In other words, as described above, the interest change determination section 239 performs predetermined weighting processing of the time interval and the space interval, determines whether the weighted time interval exceeds a predetermined threshold (time), and determines whether the weighted space distance exceeds a predetermined threshold (distance) . When the weighted time interval exceeds a predetermined threshold and/or the weighted spatial distance exceeds a predetermined threshold, the interest change determination section 239 determines that the user's object of interest has changed. When the determined result at step S208 indicates that the object of interest has changed, the flow proceeds to step S209. When the determined result at step S208 indicates that the object of interest has not changed, the flow proceeds to step S211.

When the determined result at step S208 indicates that the object of interest has not changed, the flow advances to step S211. In step S211, the interest object area classification part 236 determines that the clicked surrounding area of the current still click is contained in the same image area as the interest object area corresponding to the clicked surrounding area of the previous (past) still click, and classifies the current still click The clicked surrounding area is classified as the same ROI as the clicked surrounding area of the previous static click (that is, the same category number is assigned to the clicked surrounding area of the current static click). In other words, when classifying an object-of-interest area for each object, the same object number is assigned in the same manner as in the above-mentioned embodiment. After step S211, the flow proceeds to step S218.

In contrast, when the determined result at step S208 indicates that the user's object of interest has changed, the flow advances to step S209. In step S209, the interest object area classification part 236 determines that the click surrounding area of the current static click is not included in the interest object area corresponding to the click surrounding area of the previous (old) static click, and outputs the current static click. Click the memory data of the peripheral image, and reset the memory data. Thereafter, at step S210, the object of interest area classification part 236 classifies the clicked surrounding area of the current static click into an object of interest area different from the clicked surrounding area of the previous static click (for example, assigning a different category number to The area around the click of the current static click). In other words, when classifying an object-of-interest area for each object, a new different object number is assigned in the same manner as in the above-mentioned embodiment. After step S211, the flow proceeds to step S218.

In contrast, when the determined result at step S205 indicates that the click data is a movement click, the flow proceeds to step S212. At step S212, the interest change determination section 239 calculates the time interval between the input time of the previous movement click and the input time of the current movement click. In this case, the time interval is calculated regardless of whether, for example, there is a still click between the input time of the current movement click and the input time of the previous movement click.

Thereafter, in step S213, the interest change determination section 239 calculates the spatial distance between the input click position of the previous movement click and the input click position of the current movement click. In this case, the spatial distance is calculated irrespective of, for example, whether there is an input position of a stationary click between the input position of the current movement click and the input position of the previous movement click.

Thereafter, at step S214, the interest change determination section 239 determines whether the user's object of interest has changed corresponding to the time interval calculated at step S212 and the spatial distance calculated at step S213. In other words, the interest change determination section 239 performs predetermined weighting processing of time intervals and spatial distances, determines whether the weighted time interval exceeds a predetermined threshold (time), and determines whether the weighted spatial distance exceeds a predetermined threshold (distance). When the weighted time interval exceeds a predetermined threshold and/or the weighted spatial distance exceeds a predetermined threshold, the interest change determination section 239 determines that the user's object of interest has changed. When the determined result at step S214 indicates that the object of interest has changed, the flow proceeds to step S215. When the determined result at step S214 indicates that the object of interest has not changed, the flow proceeds to step S217.

When the determined result at step S214 indicates that the user's object of interest has not changed, the flow proceeds to step S217. In step S217, as described above, the interest object area classification part 236 determines that the clicked surrounding area of the current mobile click is contained in the same image area as the interest object corresponding to the clicked surrounding area of the previous (past) mobile click, and The clicked surrounding area of the current mobile click is classified as the ROI corresponding to the clicked surrounding area of the previous mobile click (for example, the same category number is assigned to the clicked surrounding area of the current mobile click). In other words, when classifying an object-of-interest area for each object, the same object number is assigned in the same manner as in the above-mentioned embodiment. After step S217, the flow advances to step S218.

In contrast, when the determined result at step S214 indicates that the user's object of interest has changed, the flow proceeds to step S215. In step S215, the interest object area classification part 236 determines that the click surrounding area of the current mobile click is not included in the interest object area corresponding to the click surrounding area of the previous (past) mobile click, and outputs the click surrounding area of the current mobile click Store data of the image, and reset the stored data. Thereafter in step S216, the object of interest area classification part 236 classifies the clicked surrounding area of the current mobile click as an object of interest area different from the clicked surrounding area of the previous mobile click (for example, a different category number is assigned to the current mobile Click on the surrounding area of the click). In other words, when classifying an object-of-interest area for each object, a new different object number is assigned in the same manner as in the above-mentioned embodiment. After step S216, the flow proceeds to step S218.

After steps S210, S211, S216 and S217, the flow proceeds to step S218. In step S218, the change determination and classification section 240 determines whether all processing has been completed. When the determined result at step S218 is negative, the flow returns to step S201. When the determined result at step S218 is positive, the change determination and classification section 240 completes the processing shown in FIG. 50 .

Next, the object of interest change determination processing at steps S208 and S214 shown in FIG. 50 will be described with reference to the flowchart shown in FIG. 51 .

In step S221, the interest change determination section 239 obtains information of a time interval. Thereafter, in step S222, the interest change determination section 239 executes predetermined weighting processing of time intervals. In step S223, the interest change determination processing section 239 obtains information of the spatial distance. Thereafter, in step S224, the interest change determination section 239 performs predetermined weighting processing of the spatial distance. The order of steps S221 and S222 may be changed. Likewise, the order of steps S223 and S224 can also be changed. The weighting process of the time interval can be performed by compressing the time unit (for example, compression of ms/10). The weighting process of the spatial distance can be performed by compressing the pixel pitch in the horizontal direction and the vertical direction.

Thereafter, the flow advances to step S225. At step S225, the interest change determination section 239 generates a three-dimensional vector using the weighted time interval (t) and the weighted spatial distance (X coordinate and Y coordinate) in the horizontal and vertical directions, and obtains the magnitude of the three-dimensional vector. The size of this three-dimensional vector is obtained by calculating the distance between the current input point and the previous input point on the three-dimensional space that adds the time axis (t) to the X-coordinate axis and Y-coordinate axis of the input position of the click data. Euclidean distance. After step S225, the flow advances to step S226.

At step S226, the interest change determination section 239 determines whether the size of the three-dimensional vector obtained at step S225 exceeds a predetermined threshold. When the determined result at step S226 indicates that the size of the three-dimensional vector does not exceed the predetermined threshold, the flow proceeds to step S227. In step S227, the interest change determination section 239 determines that the object of interest of the user of the receiving device 2 has not changed. When the size of the three-dimensional vector exceeds the predetermined threshold, the flow proceeds to step S228. In step S228, the interest change determination processing section 239 determines that the user's object of interest has changed.

In the manner described above, the change determination and classification section 240 completes the change determination of the user's object of interest and its classification corresponding to the click data transmitted from the receiving device 2 .

Furthermore, when the object-of-interest areas of the user of the receiving device 2 are categorized, the categorized object-of-interest areas can be optimally processed. In other words, the amount of information assigned to each object of interest area of the user may vary. For example, a large amount of information can be assigned to a user's specific interest object area. On the other hand, the data of the user's interest object area may be sent preferentially.

In addition, an image area read out from the clicked peripheral area storage section 235 for the object of interest area 236 may be transmitted in order to improve the spatial resolution in the above-described manner.

In this case, even if the user of the receiving device 2 taps the non-interest object area by mistake, it is possible to avoid performing erroneous determination.

According to this embodiment, even if one object in the sense of an object area of interest is separated from another object in time or space, they can be classified as one object. In addition, it is also possible to extract a region having a specific meaning other than an object such as an entity.

Furthermore, the embodiments described in connection with FIGS. 37 to 46 and the embodiments described in connection with FIGS. 49 to 51 may be combined. In this case, in steps S206 and S207, a continuous click may be determined for the static click of the continuous click in the above-mentioned manner. Likewise, in steps S212 and S213, a continuous click may be determined for a continuous click of a moving click in the manner described above.

Fig. 52 shows a third configuration example of the image transmission device shown in Fig. 1 . In FIG. 52, the same parts as those in FIG. 28 are denoted by like reference numerals, and description thereof will be omitted. In other words, the structure of the image distribution system shown in FIG. 52 is basically the same as that of the image distribution system shown in FIG. 28, except that the exchange section 3-3 has the charging server 4.

Although such a service of transmitting click data (or control information) from the receiving device 2 to the transmitting device 1 and providing an image having an improved spatial resolution corresponding to one click data from the transmitting device 1 to the receiving device 2 (hereinafter, this Such a service is called a click service) may be a free service, but the service may be a paid service. When the click service is a charging service, the charging server 4 executes a charging process of collecting the service fee of the click service from the user.

In other words, FIG. 53 shows an example of the configuration of the charging server 4 shown in FIG. 52 .

The predetermined information is supplied to the communication link establishment detecting section 301 from the switching station 3-3. The communication link establishment detection section 301 checks the information supplied from the switching station 3-3. As a result, the communication link establishment detecting section 301 detects whether a communication link has been established between the terminal unit 1 as the transmitting device and the terminal unit 2 as the receiving device, and supplies the synthesized data to the terminal unit approving section 302 .

When the terminal unit approval section 302 receives information representing that a communication link has been established between terminal units such as the transmitting device 1 and the receiving device 2 (hereinafter, this information is referred to as communication link establishment information), the terminal unit The approval section 302 checks the information supplied from the switching station 3-3. As a result, the terminal unit approval section 302 authorizes the terminal unit that has been communicatively linked. Further, the terminal unit approval section 302 approves an ID assigned to a terminal unit (hereinafter, the ID is referred to as a terminal unit ID) and supplies the approved terminal unit ID to the click detection section 303 .

The click detection section 303 monitors the data received via the exchange station 3-3, detects the click data transmitted from the terminal unit using the terminal unit ID received from the terminal unit approval section 302, and supplies the detection result and the terminal unit ID to the charge processing section 304 .

In this case, for example, the reception device 2 transmits the transmission click data together with the local terminal unit ID. The click detection section 303 compares the terminal unit ID attached to the click data received via the switching station 3-3 with the terminal unit ID supplied from the terminal unit approval section 302, approves the click data sent from the terminal unit which has been communicated linked, and Detect this click data.

Hereinafter, a set of click data detection results and a terminal unit ID by the click detection section 303 is referred to as click detection information.

When the charging processing section 304 receives the click detection information from the click detecting section 303 , the charging processing section 304 updates the storage content of the charging database 305 . Furthermore, the billing processing section 304 performs billing processing, for example, periodically (for example, once a month) corresponding to the storage contents of the billing database 305 .

The billing database 305 stores information necessary for billing processing.

Next, the processing of the billing server 4 shown in FIG. 53 will be described with reference to the flowchart shown in FIG. 54 .

The communication link establishment detecting section 301 monitors whether or not the communication link between the terminal units has been established corresponding to the information supplied from the switching station 3-3. When communication link establishment detection section 301 detects that the communication link between transmission device 1 and reception device 2 has been established, communication link establishment detection section 301 supplies communication link establishment information to terminal unit approval section 302 .

When the terminal unit approval section 302 receives the communication link establishment information transmitted from the communication link establishment detection section 301, in step S301, the terminal unit approval section 302 checks the information supplied from the switching station 3-3. As a result, the terminal unit identification section 302 identifies the terminal unit IDs of the terminal units (for example, the transmitting device 1 and the receiving device 2 ) that have been communicatively linked, and supplies the terminal unit IDs to the click detecting section 303 .

When the click detection section 303 receives the terminal unit ID from the terminal unit approval section 302, the click detection section 303 starts to detect click data including the terminal unit ID.

Thereafter, the flow proceeds to step S302. In step S302, the charging processing section 304 determines whether click data has been detected from a terminal unit that has been communicatively linked. When the determined result on the step S302 shows that the click data has not been detected from the terminal unit of the communication link (that is, when the click detection section 303 does not supply the click detection information to the charge processing section 304), the flow proceeds to step S304, and jumps to Go through step S303.

In contrast, when the determined result at step S302 indicates that click data has been detected from the terminal unit that has been communicatively linked (ie, when the click detection section 303 supplies the click detection information to the charging processing section 304), the flow advances to step S303. In step S303, the charging processing section 304 updates the storage content of the charging database 305.

In other words, the charge database 305 stores click information such as the number of clicks and time between points (hereinafter, this information may be referred to as click information) and stores the communication time for a terminal unit to place a call and start a communication. The charging database 305 stores the click information and the terminal unit ID of the terminal unit in relation to each other. In step S303, the charging processing section 304 updates the click information corresponding to the terminal unit ID contained in the click detection information corresponding to the click detection information.

After step S303, the process proceeds to step S304. At step S304, the terminal unit approval section 302 determines whether or not the communication link corresponding to the communication link establishment information supplied from the communication link establishment detection section 301 has been interrupted.

In other words, the communication link establishment detecting section 301 monitors not only the establishment of the communication link between the terminal units but also the intermediate end of the link. When the communication link has been interrupted, the communication link establishment detection section 301 supplies result information as communication link interruption information to the terminal unit approval section 302 . In step S304, the terminal unit approval section 302 determines whether or not the communication link has been interrupted corresponding to the communication link interruption information.

When the determined result at step S304 indicates that the communication link is not interrupted, the flow proceeds to step S302. In step S302, the charging server 4 repeats similar processing.

In contrast, when the determined result at step S304 indicates that the communication link has been interrupted, the terminal unit approval section 302 controls the click detection section 303 to complete monitoring of the click data of the terminal unit that has been communication linked. Thereafter, the charging processing server 4 completes the processing.

Thereafter, the charge processing section 304 periodically checks the charge database 305, performs charge processing, calculates communication charges and click service charges, and transfers (transfers) from the user's bank account or the like.

As the click service fee, a charging unit for each click can be specified. Corresponding to the number of clicks, a click service fee may be calculated. On the other hand, as the click service fee, a charging unit per hour may be specified. Corresponding to the click time, a click service fee may be calculated. On the other hand, corresponding to the number of clicks and the click time, a click service fee may be calculated.

The above-described processing can be executed by hardware or software. When these processes are executed by software, a program constituting the software is installed to a computer provided in the transmitting device 1, the receiving device 2, a general-purpose computer, or the like, which are dedicated hardware devices.

Fig. 55 shows an example of the structure of a computer in which a program for executing the above-mentioned processing has been installed.

The program can be recorded in advance on the hard disk 405 or ROM 403 which is one recording medium built in the computer.

On the other hand, the program may be temporarily or permanently stored (recorded) on a removable recording medium such as a floppy disk, CD-ROM (Compact Disk Read Only Memory), MO (Magneto Optical) disk, DVD (Digital Versatile Disk), magnetic disk or semiconductor on memory. The removable recording medium 411 can be set as a so-called software package.

This program can be installed on a computer from the above-mentioned removable recording medium 411 . Alternatively, the program can be wirelessly transmitted to the computer from a download point via a digital satellite broadcasting satellite. On the other hand, the program can be transmitted to the computer via a network such as LAN (Local Area Network) or the Internet. In this computer, the program is received by the communication section 408 and installed on the hard disk 405 .

This computer contains a CPU (Central Processing Unit) 402 . The input/output interface 410 is connected to the CPU 402 via the bus 401 . When a user inputs a command to CPU 402 via input/output interface 410 using input section 407 such as a keyboard, mouse, microphone, etc., programs stored in ROM (Read Only Memory) 403 start executing. On the other hand, CPU 402 loads a program stored in hard disk 405 onto RAM (Random Access Memory) 404 and executes the program using RAM 404 . On the other hand, the CPU 402 transmits from a satellite or network, receives and installs a program on the hard disk 405 by the communication section 408, or reads a program from a removable recording medium 411 connected to the drive 409 and installs it on the hard disk 405, Load it on RAM404, and use ARM404 to execute the program. Thus, the CPU 402 executes processes corresponding to the flow charts shown in FIGS. deal with. And the CPU 402 executes the processing shown in the block diagrams corresponding to FIGS. The CPU 402 outputs the processed result from an output section 406 including an LCD (Liquid Crystal Display), a speaker, etc. via an input/output interface 410 as necessary. On the other hand, the CPU 402 transmits the processed result from the communication section 408 or recorded to the hard disk 405 .

Specifically, the processing steps of the programs that cause the computer to execute various processing are not always executed in the order of the flowcharts. On the other hand, processing steps may be performed in parallel or separately (eg, parallel processing or object processing may be performed).

The program can be processed by a computer. On the other hand, the program can be distributed by multiple computers. On the other hand. The program can be transmitted to a remote computer where processing is performed.

According to an embodiment of the present invention, the transmitting device 1 performs layered coding. According to the transmitted hierarchical data, the temporal resolution or spatial resolution of the image displayed by the receiving device 2 is changed. On the other hand, the temporal resolution and spatial resolution of the image displayed by the receiving device 2 can be changed according to the discrete cosine transform coefficient or quantizing step of the transmitted image.

On the other hand, temporal resolution and spatial resolution can be changed according to the encoding method of the transmitting device 1 . In other words, when an image is displayed with a conventional time resolution, the encoding section 31 of the transmitting device 1 continuously encodes the outline of its object, and obtains the average value of pixel values (colors) constituting an object as a typical value. The receiving device 2 displays the area of the colored object according to this typical value. Hierarchical coding can be used as described above when displaying images with improved spatial resolution at the expense of temporal resolution.

According to this embodiment, the improvement in spatial resolution comes at the expense of temporal resolution. Instead, temporal resolution can be improved at the expense of spatial resolution. Information representing the choice of sacrificing (improving) resolution may be included in control information such as click data, and transmitted from the receiving device 2 to the transmitting device 1 .

According to an embodiment of the invention, temporal resolution and spatial resolution are handled. However, according to the present invention, resolution in the level direction (hereinafter, the resolution is referred to as level resolution) can also be handled. In other words, when the number of bits allocated to data increases or decreases, temporal resolution and spatial resolution can improve or deteriorate. In this case, as the temporal resolution and spatial resolution of the image vary, its lightness and darkness also vary. The level resolution can be changed by changing the quantization scale described above.

Furthermore, according to this embodiment, the improvement of the spatial resolution of the partial region of the image (priority range) is at the expense of the temporal resolution. On the other hand, the spatial resolution of the entire image can also be improved.

On the other hand, the improvement of the spatial resolution of a certain part of the image can be at the expense of the spatial resolution of the rest of the image, rather than at the expense of the temporal resolution (ie, maintaining the temporal resolution).

Also, according to the present embodiment, an image is processed in such a way that it is divided into a background image and an object. It is also possible to process images without separating them.

Furthermore, the present invention is applicable not only to image data but also to audio data. In the case of audio data, the sampling frequency corresponds to time resolution, and the number of bits allocated to audio data corresponds to level resolution.

In addition, the processing of the change determination and classification section 240 shown in FIG. 49 is applicable to the case of extracting sound feature quantities such as pitch, voice, or a desired part of a musical instrument.

In addition, the processes of the object extraction sections 14 and 1014 and the variation determination and classification section 240 are applicable to so-called object encoding. In other words, for the processing of the object extraction sections 14 and 1014 and the change determination and classification section 240, since an object can be extracted, it is possible to correspond to the object encoding process of acquiring information representing the movement of the object, and the extracted object and Encodes information representing object contours or regions. This encoded data can be transmitted or recorded.

The reception device 2 can perform processing similar to that of the object extraction section 1014 using the click data to extract an object. In this case, when an object extracted by the receiving device 2 is stored, a database is formed.

According to the first transmitting device, the first transmitting method, the first recording medium and the first signal of the present invention, control information transmitted from the receiving device is received. Corresponding to the control information, resolutions in at least two directions of time direction, space direction and level direction are controlled. Data having controlled resolution in at least two directions corresponding to the control information is transmitted to the receiving device. Thus, for example, the spatial direction resolution of images displayed by the receiving device can be further improved.

According to the receiving device, the receiving method, the second recording medium, and the second signal of the present invention, the control information is sent to the sending device so that the control information in at least two directions of the time direction, the space direction and the level direction is controlled corresponding to the control information. resolution. Furthermore, data having controlled resolution in at least two directions corresponding to the control information is transmitted from the transmitting device and then received and output. Thus, for example, the spatial resolution of the output image can be further improved.

According to the transmitting and receiving device, the transmitting and receiving method, the third recording medium, and the third signal of the present invention, the transmitting device receives control information transmitted from the receiving device. Corresponding to the control information, resolutions in at least two directions of a time direction, a space direction and a level direction are controlled. Data having controlled resolution in at least two directions corresponding to the control information is transmitted to the receiving device. In addition, the receiving device sends control information to the sending device. Data having controlled resolution in at least two directions corresponding to the control information is transmitted from the transmitting device, then received and output. Thus, for example, the spatial direction resolution of images displayed by the receiving device can be further improved.

According to the second transmitting device, the second transmitting method, the fourth recording medium, and the fourth signal of the present invention, the control signal transmitted from the receiving device is received. Corresponding to the control information, the data is categorized. Corresponding to the classification result of the data, the data is transmitted to the receiving device. Thus, for example, the region of the image that the user is considering may be sent to the receiving device, irrespective of whether the region is moving or stationary.

Label description

1 terminal unit (sending device)

2 terminal unit (receiving equipment)

31,3-2 Wireless base station

4 Fee server

13 background image extraction part

14 object extraction part

22 Combined processing part

31 Encoding part

34 Data amount calculation part

41B, 41F difference calculation part

42B, 42F layered coding part

44B, 44F local decoder

51 Receive part

62B, 62F storage part

71 Background image writing part

73 background image memory

75 object storage

77 Composite parts

142 CPUs

146 display part

201 Image memory

203 Stationary area and moving area determination part

210 Mobile object connection processing part

211 static object connection processing part

214 object extraction result storage

231 input image storage part

234 Stationary area and moving area determination part

236 Classification of target area of interest

237 Enter the time interval calculation part

238 Input position distance calculation part

239 Change of Interest Determination Section

301 Communication link establishment detection part

303 click detection part

304 Charge processing part

402 CPU

408 Communication part

411 Removable recording media

1013 background image extraction part

1014 object extraction part

1022 combined processing part

1024 Click on the data entry section

1025 Click on the data sending part

Claims (20)

1. A sending device for sending data to a receiving device, comprising: a receiving device, configured to receive control information sent from the receiving device; classification means for classifying the data corresponding to the control information; and The sending means sends the data to the receiving device corresponding to the classification result of the data. 2. The sending device according to claim 1, wherein said data is image data, wherein the receiving device displays the image data sent from the sending device, wherein the control information contains considerations of the image data displayed by the receiving device, and The classifying means described therein classifies the image data corresponding to the considered area including the considered point of the image data. 3. The sending device according to claim 2, further comprising: stationary area and moving area determining means for determining whether the image data considered point is stationary or moving, and continuity determining means for determining whether the considered point is continuous in the time direction and in the space direction, Wherein said classifying means classifies the image data corresponding to the determination results of said still area and moving area determining means and continuous determining means. 4. The sending device according to claim 3, further comprising: Consideration point storage means for storing the consideration points contained in the stationary and continuous consideration area in the time and space direction, and storing the consideration contained in the moving and continuous consideration area in the time and space direction points; and category identifier adding means for acquiring a category identifier added to a consideration point stored in the consideration point storage means, and adding the identifier to the consideration point. 5. The sending device according to claim 4, Wherein in the case where the current consideration point is in a stationary and continuous consideration area in the time and space direction, when the previous consideration point stored in the consideration point storage means is included in the stationary and in the time and space direction When within the continuous consideration area, corresponding to the spatial position relationship between the current consideration point and the area containing the previous consideration point, the category identifier addition means obtains the category identifier added to the current consideration point. 6. The sending device according to claim 4, Wherein in the case that the current consideration point is in the moving and continuous consideration area in the time and space direction, when the previous consideration point stored in the consideration point storage device is included in the moving and in the time and space direction When within the continuous consideration area, corresponding to the similarity between the predetermined feature quantity of the consideration area containing the current consideration point and the predetermined feature quantity of the consideration area containing the previous consideration point, the category identifier adding means is added to The class identifier for the current consideration point. 7. The sending device according to claim 4, Wherein said classifying means classifies a predetermined area of the image data as an object corresponding to the density of the considered points stored in said considered point storage means. 8. The sending device according to claim 7, Wherein said categorization means is corresponding to the density of the consideration points assigned with the same category identifier stored in the consideration point storage means and included in the static consideration area, and performs a predetermined region of the image data as an object. categorized. 9. The sending device according to claim 7, Wherein said categorization means is corresponding to the density of the consideration points that are included in the moving consideration area stored in the consideration point storage means, assigned the same category identifier and performed motion compensation, for an image as an object A predetermined area of data is categorized. 10. The sending device according to claim 3, Wherein said static area and moving area determination means are corresponding to the difference between the considered area containing the considered point of the current frame and the considered area containing the considered point of the past frame, determine whether the considered area containing the current considered point is stationary or mobile. 11. The sending device according to claim 3, Wherein said continuity determining means determines whether the current consideration point is continuous in time and space direction corresponding to the time difference between the current consideration point and the past consideration point. 12. The sending device according to claim 7, The classification device described therein improves the resolution of the object-classified regions. 13. The sending device according to claim 3, Wherein said continuity determination means determines whether the current consideration point is continuous corresponding to the distance between the current consideration point and the past consideration point in the time and space direction, and obtains the same static area and moving area determination as the consideration area containing the current consideration point result. 14. Sending device according to claim 13, The sorting device described therein sorts the image data according to the weighting values for the distances in the time direction and in the space direction. 15. The sending device according to claim 13, further comprising: Image data storage means for storing image data contained in the consideration area of the consideration points continuous in the time direction and the space direction. 16. The sending device according to claim 15, Wherein when the current consideration point is discontinuous in the time direction and the space direction, after the content of the image data storage device is read out, its content is erased, and the image data in the consideration area containing the current consideration point is stored to the image data storage device. 17. The sending device according to claim 16, Wherein said sorting device improves the resolution of the image data read out from said image data storage device. 18. The sending device according to claim 1, Wherein the control information is used for a charging process. 19. The sending device according to claim 2, The image data is the object of encoding. 20. A sending method for sending data to a receiving device, comprising the following steps: receiving control information sent from the receiving device; classify the data corresponding to the control information; and The data is transmitted to the receiving device corresponding to the classification result of the data.

Images