JP2010160576A - Image reproduction apparatus and image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To suitably perform an operation on an image even if a pointer operation is performed on the transformed image for which a pointer's relative position is different from that for the original image. <P>SOLUTION: Image data included in the content reproduced by an image reproduction means 220 and operation image data displayed in the image reproduction means 220 together with the image data and adapted for instructing a user to perform a variety of process operations on the content are sampled integrally, and when a pointer event for display image data to be processed by an image processing means 240 is detected, the pointer event's coordinate position information is transformed and output to the image reproduction means 220. If the coordinate position of the pointer event is a part of the operation image data, the image reproduction means 220 performs a process associated with the operation image data. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image reproduction device and an image processing device.

  In recent years, browsing video content (AV content) published on a video distribution site on the Internet using a browser has been widely performed (see, for example, Patent Document 1).

  When the browser browses the web page of the video distribution site, the browser starts a video reception / playback program incorporated as plug-in software. When an operation for instructing the start of AV content playback is performed by the user, the video reception / playback program downloads the AV content from the video distribution site via the Internet, decodes it, and separates it into video data and audio data. The moving image reception / playback program outputs the separated video data and audio data to a DLL (Dynamic Link Library) belonging to the operating system. Via this DLL, video data and audio data are delivered to the kernel unit of the operating system, and display output, speaker output, and the like are performed.

JP 2007-280199 A

  By the way, recently, there is an increasing demand for viewing AV content reproduced through the above-described procedure as a higher quality image. In addition, there is a strong demand for the realization of existing modules as they are.

  However, when AV content played back through the above-described procedure is played back as a higher quality image, there is a demand to view the moving image received by the moving image reception / playback program and the high quality moving image in separate windows. is there. In such a case, when the image to be operated by the user with a pointing device or the like is included in the original image, and the relative coordinates on the image are different between the original image and the converted image (for example, When the resolutions of both images are different), even if the user operates the pointing device at a certain position of the converted image, the coordinates are different from the original image, so the operation on the original image is preferable. There is a possibility that it will not be done.

  The present invention has been made in view of the above, and even when a pointer operation is performed on a converted image in which the relative position of the pointer is different from that of the original image, the operation on the image can be suitably performed. An object of the present invention is to provide an image reproducing apparatus and an image processing apparatus.

  In order to solve the above-described problems and achieve the object, an image reproducing apparatus according to the present invention includes moving image data obtained by decoding encoded data by an image reproducing unit, and the image reproducing unit together with the moving image data. First display image data including image data to be output and operation image data for providing a user with various processing operations on the moving image data can be displayed as second display image data by performing image processing. Image processing means for outputting to the display, event detection means for detecting a pointer event for the second display image data output by the image processing means, and the coordinate position included in the pointer event when the pointer event is detected Coordinate position information converting means for converting information, and outputting the pointer event after the coordinate position information conversion to the image reproducing means Comprising an event output unit that, the.

  In addition, the image processing apparatus of the present invention includes an image reproducing unit that decodes encoded data into moving image data and outputs operation image data for providing a user with various operation functions for the moving image data, and the image reproducing device. Image processing means for performing image processing on the first display image data including the moving image data and the operation image data output by the means and outputting the second display image data so as to be displayable, and the image processing means Event detecting means for detecting a pointer event for the second display image data to be output; coordinate position information converting means for converting the coordinate position information included in the pointer event when the pointer event is detected; Event output means for outputting the pointer event after the position information conversion to the image reproduction means, and It means, when the coordinate position on the first display image data of the pointer event output from the event output means is on the operation image data, executes processing associated with the operation image data.

  According to the present invention, even when a pointer operation is performed on a converted image in which the relative position of the pointer is different from that of the original image, it is possible to perform an operation on the image suitably.

  Exemplary embodiments of an image reproducing apparatus and an image processing apparatus according to the present invention are explained in detail below with reference to the accompanying drawings.

  An embodiment of the present invention will be described with reference to FIGS. In this embodiment, a notebook type personal computer (PC) is applied as an image processing apparatus.

  FIG. 1 is a configuration diagram showing one usage pattern of a computer 1 according to an embodiment of the present invention. In the present embodiment, as shown in FIG. 1, it is assumed that the computer 1 is used to download and view AV content provided by the video distribution site 2 via the Internet 3.

  Here, FIG. 2 is a block diagram showing a hardware configuration of the computer 1. As shown in FIG. 2, the computer 1 includes a CPU 11, a north bridge 12, a main memory 13, a display controller 14, a video memory (VRAM) 14A, an LCD (Liquid Crystal Display) 15, a south bridge 16, a sound controller 17, and a speaker 18. BIOS-ROM 19, LAN controller 20, HDD (hard disk drive) 21, ODD (optical disc drive) 22, wireless LAN controller 23, USB controller 24, embedded controller / keyboard controller (EC / KBC) 25, keyboard (KB) 26, for example, a pointing device 27 such as a mouse.

  The CPU II is a processor that controls the operation of the computer 1 and executes an operating system (OS) loaded from the HDD 21 or ODD 22 to the main memory 13 and various application programs operating under the OS. The CPU 11 also executes a basic input / output system (BIOS) stored in the BIOS-ROM 19. The BIOS is a program for hardware control.

  The north bridge 12 is a bridge device that connects the local bus of the CPU 11 and the south bridge 16. The north bridge 12 also includes a memory controller that controls access to the main memory 13. The north bridge 12 also has a function of executing communication with the display controller 14.

  The display controller 14 is a device that controls the LCD 15 used as a display monitor of the computer 1. A display signal is generated by the display controller 14 and sent to the LCD 15.

  The south bridge 16 controls devices on a peripheral component interconnect (PCI) bus and a low pin count (LPC) bus. The south bridge 16 includes an IDE (Integrated Drive Electronics) controller for controlling the HDD 21 and the ODD 22 and a memory controller for controlling access to the BIOS-ROM 19. Further, the south bridge 16 has a function of executing communication with the sound controller 17 and the LAN controller 20.

  The sound controller 17 is a sound source device and outputs audio data to be reproduced to the speaker 18.

  The LAN controller 20 is, for example, a wired communication device that executes wired communication based on the Ethernet (registered trademark) standard, and the wireless LAN controller 23 is a wireless communication device that executes wireless communication based on the IEEE 802.11 standard, for example. The USB controller 24 executes communication with an external device via, for example, a USB 2.0 standard cable.

  The EC / KBC 25 is a one-chip microcomputer in which an embedded controller for performing power management and a keyboard controller for controlling the keyboard (KB) 26 and the pointing device 27 are integrated. The EC / KBC 25 has a function of powering on / off the computer 1 in accordance with a user operation.

  The computer 1 having the configuration as described above receives AV content provided by the video distribution site 2 shown in FIG. 1 by a program group (OS and various applications) loaded from the HDD 21 to the main memory 13 and executed by the CPU 11. It operates to download and play back via the Internet 3.

  More specifically, AV content is input by communication processing by the LAN controller 20, the wireless LAN controller 23, or the USB controller 24, and the AV content is decoded by the CPU 11 and separated into video data and audio data. The data is supplied to the display controller 14 and output from the LCD 15 as an image, and the audio data is supplied to the sound controller 17 and output from the speaker 18 as sound.

  Next, referring to FIG. 3, an outline of a group of programs for operating the computer 1 to download and play back the AV content provided by the video distribution site 2 shown in FIG. The configuration will be described.

  As shown in FIG. 3, an OS 100, a browser 210, a moving image reception / playback program 220, and an effect engine 230 are installed in the computer 1. The moving image reception / playback program 220 and the effect engine 230 are incorporated in the browser 210 as plug-in software.

  The OS 100 that centrally manages resource management of the computer 1 includes a kernel unit 101 and a DLL unit 102. The kernel unit 101 is a module that drives and controls each unit (hardware) of the computer 1 illustrated in FIG. 2, and the DLL unit 102 is a module that provides an interface with the kernel unit 101. The hierarchy up to the stage where various application programs issue various requests to the kernel unit 101 to the DLL unit 102 is the user mode, and thereafter, that is, from the stage where the DLL unit 102 transmits these requests to the kernel unit 101. This hierarchy is called kernel mode.

  When the web page of the video distribution site 2 is browsed by the browser 210, the browser 210 detects that the site has AV contents that can be distributed by the tag information in the web page, and is installed as plug-in software. The moving image receiving / reproducing program 220 is activated. When the user performs an operation to instruct the start of AV content playback while browsing the Web page, the video reception / playback program 220 starts downloading the AV content from the video distribution site 2 via the Internet 3. .

  The moving image reception / playback program 220 functions as an image playback means, and performs decoding of the AV content in parallel while downloading the AV content. By this decoding, the moving image reception / playback program 220 separates and extracts video data and audio data included in the AV content. The moving image reception / playback program 220 delivers the extracted video data and audio data to the DLL unit 102 of the OS 100 (a1, b1) to output an image (by the LCD 15) and output a sound (by the speaker 18).

  Normally, the video data a1 and audio data b1 passed to the DLL unit 102 are subjected to processing such as a format check in the DLL unit 102 and supplied to the kernel unit 101. The kernel unit 101 outputs the data from the LCD 15 to the DLL unit 102. Image output and audio output from the speaker 18 are performed.

  On the other hand, the computer 1 is equipped with an effect engine 230 in order to meet the user's request to view the AV content downloaded from the video distribution site 2 with high image quality. Correction is performed on the video data a1. Hereinafter, this point will be described in detail.

  The effect engine 230 is a program incorporated as resident plug-in software that is activated when the browser 210 is activated, and functions as an image playback device. As shown in FIG. 3, the effect engine 230 functions as image processing means. An image processing unit 240, an engine activation unit 235, and an operation instruction synchronization unit 236. The image processing unit 240 includes a capture setting unit 231, a time stamp unit 232, an effect unit 233, and an AV synchronization unit 234.

  The engine activation unit 235 is a module for activating the image processing unit 240 and the operation instruction synchronization unit 236 only when it is determined that execution of main processing in the effect engine 230 is instructed by the user.

  The capture setting unit 231 is a module that rewrites a routine in the DLL unit 102 in order to acquire the video data a1 and the audio data b1 output to the DLL unit 102 by the moving image reception / playback program 220. Although an example of the routine rewriting process in the DLL unit 102 will be described later, simply speaking, the video data a1 and the audio data b1 received from the moving image reception / playback program 220 are delivered to the time stamp unit 232 of the effect engine 230. (A2, b2). When the effect engine 230 is activated, the capture setting unit 231 requests the browser 210 to notify the user when the moving image reception / reproduction program 220 is activated. When receiving this notification, the capture setting unit 231 rewrites the routine in the DLL unit 102.

  The time stamp unit 232 that receives the video data a2 and the audio data b2 from the DLL unit 102 by the operation of the capture setting unit 231 is a module that assigns time information indicating the received timing to each. The time information may be, for example, the system time of the computer 1 as long as it is a value that can be discriminated in time, or time data that starts counting up after the effect engine 230 is started. The video data a2 and audio data b2 to which time information is given by the time stamp unit 232 are guided to the effect unit 233.

  The effect unit 233 is a module that performs correction for improving the image quality of the video data a2. Here, an example in which the effect unit 233 corrects only the video data a2 will be described, but it is not limited at all to perform correction for improving the sound quality of the audio data b2. Naturally, it is possible to correct each of the video data a2 and the audio data b2 for high image quality and high sound quality, and conversely, only the audio data b2 (high sound quality improvement). It is also possible to perform correction. These image quality enhancement and image quality enhancement processing may be performed by software processing on the CPU 11 or may be performed on a dedicated hardware chip that is not shown in FIG. Anyway.

  The effect unit 233 performs image processing (hereinafter referred to as “super-resolution conversion processing (sharpening processing)”) for increasing the resolution of the input medium-resolution or low-resolution frame, and converts the high-resolution video data. A frame (hereinafter referred to as “high resolution frame”) is generated. Here, the super-resolution conversion process means that the original pixel value is estimated from the low-resolution or medium-resolution image signal (low-resolution frame or medium-resolution frame) that is the first resolution, and the number of pixels is increased. Means a sharpening process for restoring a high-resolution sharpened video signal (high-resolution frame). Here, the original pixel value is, for example, an image of the same subject as that obtained when a low-resolution (first resolution) image signal is obtained with a camera having the number of pixels of the high-resolution (second resolution) image signal. It refers to the value of each pixel of the image signal that is sometimes obtained. “Estimate and increase pixels” means that the original pixel value is estimated from the surrounding (intra-frame or inter-frame) image by capturing the image characteristics and using the image characteristics that are correlated. Means increasing the number of pixels.

  For the super-resolution conversion processing, it is possible to use a publicly known / public technique disclosed in Japanese Unexamined Patent Application Publication No. 2007-310837, Japanese Unexamined Patent Application Publication No. 2008-98803, Japanese Unexamined Patent Application Publication No. 2000-188680, and the like. As a technique for the super-resolution conversion processing of the present embodiment, for example, a technique for restoring an image having a frequency component higher than the Nyquist frequency determined by the sampling period of the input image is used.

  For example, when using the super-resolution conversion process disclosed in Japanese Patent Application Laid-Open No. 2007-310837, the change pattern of the pixel value in the target image area including the target pixel in the frame for each of a plurality of medium resolution frames. Select a plurality of corresponding points corresponding to a plurality of target image areas closest to the reference frame from the reference frame, set the sample value of the luminance at the corresponding point to the pixel value of the target pixel corresponding to the corresponding point, and Calculating a pixel value of a high-resolution frame corresponding to the reference frame, which is a high-resolution frame having a number of pixels larger than the number of pixels of the reference frame, based on the size of the sample value and the arrangement of a plurality of corresponding points. Thus, the original pixel value is estimated from the medium resolution frame and the number of pixels is increased to restore the high resolution frame.

  In addition, when using the super-resolution conversion process using self-congruent position search in the same frame image disclosed in Japanese Patent Application Laid-Open No. 2008-98803, the error of each pixel in the search area of the medium resolution frame is compared. The minimum first pixel position is calculated, and based on the first pixel position and the first error, the second pixel position around the first pixel, and the second error, The position where the error is minimized is calculated with decimal precision. Then, a decimal precision vector with this position as the end point and the target pixel as the start point is calculated, and an extrapolation vector of the decimal precision vector with the pixel on the screen not included in the search region as the end point is calculated using the decimal precision vector. calculate. Then, based on the decimal precision vector, the extrapolation vector, and the pixel value acquired from the image data, the pixel value of the high-resolution image having a larger number of pixels than the number of pixels included in the image data is calculated. By performing such processing, the effect unit 233 restores the high resolution frame by estimating the original pixel value from the medium resolution frame and increasing the number of pixels.

  Also, a super-resolution conversion process using mapping between a plurality of frame images disclosed in Japanese Patent Laid-Open No. 2000-188680 can be used.

  However, the super resolution conversion process (sharpening process) method in the effect unit 233 is not limited to the above, but by estimating the original pixel value from the low-resolution or medium-resolution image signal and increasing the number of pixels. Any method can be applied as long as it restores a high-resolution image signal.

  The video data a2 and the audio data b2 (here, only the video data a2) corrected for at least one by the effect unit 233 are subsequently led to the AV synchronization unit 234. The AV synchronization unit 234 matches the video data a2 and the audio data b2 received from the effect unit 233 in time to absorb the difference in delay time associated with the correction based on the time information given by the time stamp unit 232. This is a module that outputs to the DLL unit 102 (a3, b3). The output of video data a2 and audio data b2 based on this time information will also be described later.

  The video data a3 and audio data b3 output from the AV synchronization unit 234 to the DLL unit 102 are passed to the kernel unit 101 via the DLL unit 102 (a4, b4), and high image quality and high sound quality are achieved. Image output and audio output are performed.

  Therefore, the user can appreciate the AV content downloaded from the moving image distribution site 2 with high image quality and high sound quality while using the existing moving image reception / playback program 220 as it is.

  FIG. 4 is a conceptual diagram for explaining an example of a routine rewriting process in the DLL unit 102 by the capture setting unit 231.

  As described above, the moving image reception / playback program 220 delivers the video data and audio data (AV data) separated and extracted from the AV content to the DLL unit 102 of the OS 100. Therefore, the browser 210 notifies that the moving image reception / playback program 220 has started, and the engine startup unit 235 determines that the execution of the main processing in the effect engine 230 has been instructed by the user, and started the effect engine. The capture setting unit 231 of 230 transfers the AV data to the effect engine 230 separately from the routine (the “original processing” portion in the figure) in the DLL unit 102 that performs processing such as a format check on the input AV data. The calling procedure ("call" in the figure) for rewriting is replaced with the routine arranged at the head thereof. Delivery of AV data is performed by transmitting address information on the main memory 13 in which AV data to be corrected is stored. This rewriting is executed by the capture setting unit 231.

  FIG. 5 is a conceptual diagram for explaining the output principle of video data and audio data corrected by the effect unit 233 based on the time information given by the time stamp unit 232 by the AV synchronization unit 234.

  In FIG. 5, (A) is a timing chart showing how the video reception / playback program 220 downloads AV content, and (B) is a video obtained by decoding and separating / obtaining the AV content downloaded by the video reception / playback program 220. A timing chart showing how data and audio data are output, (C) is a timing chart showing how the effect unit 233 of the effect engine 230 outputs corrected video data and audio data, and (D) is an effect engine. 23 is a timing chart showing how the AV synchronization unit 234 of 230 outputs video data and audio data received from the effect unit 233 to the DLL unit 102.

  Since data communication executed between the computer 1 and the moving image distribution site 2 via the Internet 3 can be affected by changes in the communication environment, AV contents (D1, D2,... ) May be irregular in time, as shown in FIG. On the other hand, the moving image reception / playback program 220 for decoding AV content displays video data (V1, V2,...) And audio data (A1, A2,. As shown in FIG. 5B, it has a function of outputting to the DLL unit 102 of the OS 100 in time synchronization. Therefore, the time stamp unit 232 of the effect engine 230 gives the same time information to the video data and audio data that are synchronized in time almost at the timing shown in FIG.

  However, when the effect unit 233 of the effect engine 230 executes correction for improving the image quality of the video data and correction for improving the sound quality of the audio data (depending on whether correction is made or the content differs) As shown in FIG. 5C, a time difference occurs between the video data and the audio data. Therefore, the AV synchronization unit 234 of the effect engine 230 outputs video data and audio data when, for example, a parameter input period (x) elapses from the time point specified by the time information given by the time stamp unit 232. By doing so, as shown in FIG. 5D, the video data and the audio data are again matched in time.

  This period (x) is sufficiently smaller than a required period until the AV content starts to be transferred from the video distribution site 2 to the video reception / playback program 220 after the user performs an operation to instruct the AV content playback start. Therefore, the user does not feel uncomfortable.

  Since the AV synchronization unit 234 includes a mechanism for absorbing a delay time difference associated with correction, various correction methods can be applied to the effect unit 233. For example, when applying an advanced correction method, the time (x) may be set to a large value, and the presence or absence of correction and the correction method are individually determined without considering the relationship between video data and audio data. It becomes possible to decide.

  When the engine starting unit 235 determines that the execution of the main process in the effect engine 230 is instructed by the user through the above processing, the video data and audio data included in the AV content by the moving image reception / playback program 220 are determined. Image output and audio output, and image output and audio output for video data and audio data that have been corrected for high image quality by the effect engine 230 are executed. In this case, in the present embodiment, playback of AV content by the video reception / playback program 220 is started, and thereafter, the effect engine 230 corrects the output screen of the video reception / playback program 220 for high image quality. The recorded moving image is reproduced on the entire screen of the LCD 15. At this time, the AV content playback window W1 (see FIG. 10) by the moving image reception / playback program 220 is a background (a full-screen window W2 (see FIG. 10) on which a high-quality moving image is displayed by the effect engine 230). Hiding under)).

  As will be described in detail later, the operation instruction synchronization unit 236 synchronizes the user operation on the moving image by the moving image reception / playback program 220 and the user operation on the moving image that has been corrected for high image quality by the effect engine 230. Module.

  Next, a determination process of an execution instruction for main processing in the effect engine 230 from the user in the engine starting unit 235 will be described. Here, FIG. 6 is a flowchart showing the flow of the main process execution instruction determination process in the effect engine 230. While the image (video) of the video data included in the AV content by the video reception / playback program 220 is output to the LCD 15 (see FIG. 7), the mouse pointer is moved over the video (on the playback window W1) by the user operation of the pointing device 27. ) (Mouse over) is notified to the engine activation unit 235 via the EC / KBC 25 and the OS 100, the engine activation unit 235 determines that the mouse pointer of the pointing device 27 is on the moving image (step The engine activation unit 235 obtains moving image information (moving image size, URL (Uniform Resource Locator) of the AV content download source, version of the moving image reception / playback program 220, etc.) via the OS 100 (step S2). Note that the moving image information (moving image size, URL (Uniform Resource Locator) of the AV content download source, version of the moving image reception / playback program 220, etc.) may be directly acquired from the moving image reception / playback program 220.

  Next, the engine activation unit 235 determines whether or not the moving image matches the support target condition for the image quality enhancement processing by the effect unit 233 of the effect engine 230 based on the moving image information acquired in step S2 (step S3). The moving image size acquired as the moving image information, the URL of the AV content download source, the version of the moving image reception / playback program 220, and the like are information that has various effects on the image quality improvement processing by the effect unit 233 of the effect engine 230. For example, when the moving image size does not match the specified size, the image quality improvement processing becomes complicated. Therefore, the moving image having a size other than the specified size is not supported by the effect unit 233. . When the URL of the AV content download source is a predetermined harmful site, the AV content downloaded from the site does not support the image quality improvement processing by the effect unit 233. Furthermore, depending on the version of the video reception / playback program 220, there is a possibility that the image quality improvement processing routine may be different. Therefore, the video by the video reception / playback program 220 of a version other than the specified version is processed by the effect unit 233. It is not supported.

  When the engine activation unit 235 determines that the moving image matches the support target condition for the image quality enhancement processing by the effect unit 233 (Yes in step S3), the engine activation unit 235 is illustrated in FIG. 8 via the OS 100 or the display controller 14. As described above, the button B for high image quality processing is displayed on the moving image displayed on the LCD 15 (step S4).

  When the engine start unit 235 is notified via the EC / KBC 25 and the OS 100 that the button B displayed on the moving image in the state shown in FIG. When the button B is clicked by the pointing device 27, it is determined that execution of the main process in the effect engine 230 is instructed by the user (Yes in step S5), the capture setting unit 231, the time stamp unit 232, The effect unit 233, the AV synchronization unit 234, and the operation instruction synchronization unit 236 are each activated (step S6).

  If the engine activation unit 235 determines that the moving image does not match the support target condition for the image quality enhancement processing by the effect unit 233 (No in step S3), the engine activation unit 235 returns to step S1 and performs a mouse operation by the user of the pointing device 27. Wait for over.

  If several seconds (for example, 5 seconds) have passed without the button B being clicked (Yes in step S7), the button B for high image quality display operation displayed on the moving image is not displayed (step S8). ), The process returns to step S1 and waits for the mouse over by the user operation of the pointing device 27.

  Thereby, the user can perform optional function processing such as high image quality processing without considering whether the AV content is a moving image that matches the support target condition of the optional function processing such as high image quality processing by the effect unit 233. It can be activated smartly.

  Next, a user operation on a moving image (image data included in AV content) by the moving image reception / playback program 220 in the operation instruction synchronization unit 236 and a moving image (captured) that has been corrected by the effect engine 230 for high image quality. A synchronization process for synchronizing the image data with the user operation will be described.

  As described above, in the present embodiment, the playback of AV content by the moving image reception / playback program 220 is started, and thereafter, the effect engine 230 performs correction for improving the image quality on the output screen of the moving image reception / playback program 220. Is reproduced on the entire screen of the LCD 15. At that time, the AV content playback window W1 by the video reception / playback program 220 is continued in the background (hidden under the full-screen window W2 on which the high-quality moving image is displayed by the effect engine 230). ing.

  The problem here is that the user operation on the full-screen window W2 on which a moving image with high image quality is displayed by the effect engine 230 is displayed in the background (the image quality is improved by the effect engine 230 (for example, high resolution If the moving image is reproduced by the moving image receiving / reproducing program 220 that is being reproduced in (the hidden image is hidden under the full-screen window W2 on which the moving image is displayed) It is a point that cannot be said.

  In the present embodiment, the effect unit 233 in the effect engine 230 applies to the display image data in which the moving image data obtained by decoding with the moving image reception / playback program 220 and the operation image data for user operation are integrated. Processing such as higher resolution is performed. Here, since the image actually viewed by the user is an image processed and output by the effect engine 230, the user gives an operation instruction to the display image data after processing. On the other hand, processing for the operation (for example, stop / restart of playback) is actually performed by the moving image reception / playback program 220. Therefore, it is necessary to pass a pointer event obtained by operating a pointer to an image on the effect engine 230 to the moving image reception / playback program 220.

  Here, since the resolution is different between the original display image data and the display image data after the high resolution processing, the relative position (coordinate position) of the pointer with respect to the image is different between the two. . That is, even if a pointer event (including coordinate position information) obtained by operating a pointer on the effect engine 230 is passed to the moving image reception / playback program 220 as it is, an operation on an image at a coordinate position intended by the user can be performed. It will not be.

  Therefore, in the present embodiment, by providing the operation instruction synchronization unit 236, the operation instruction synchronization unit 236 has a window (the image quality is improved by the effect engine 230) that is different from the moving image reproduced by the moving image reception reproduction program 220. When the user's operation on the full-screen window W2) on which the moving image is displayed is received, the user's mouse operation (pointer event) is analyzed, the coordinate position information is converted, and the video reception / playback program 220 is appropriate. By sending a message, a behavior equivalent to a pointer event on the moving image reproduction window W1 by the moving image reception / reproduction program 220 can be performed.

  FIG. 9 is a block diagram illustrating a functional configuration of the operation instruction synchronization unit 236. As shown in FIG. 9, the operation instruction synchronization unit 236 is an event detection unit that detects a pointer event for a full-screen window W2 on which a high-quality moving image that is reproduced by the effect unit 233 of the effect engine 230 is displayed. When an event detection unit 236a detects a pointer event, an event output that transfers (outputs) the pointer event (including coordinate position information that is information on the coordinate position where the pointer event occurred) to the moving image reception / playback program 220 Moving image reproduction by the full-screen window W2 on which the moving image with high image quality (resolution is converted) reproduced by the event transferring unit 236b and the effect unit 233 of the effect engine 230 and the moving image receiving / reproducing program 220 are displayed. Coordinate system with window W1 (pointer to image Coordinate system conversion unit which is coordinate position information conversion means for converting the coordinate position information on the window W2 of the entire screen where the event is detected into the coordinate position information on the moving image reproduction window W1 236c.

  Here, FIG. 10 is a schematic diagram showing an outline of the operation of the synchronization processing by the operation instruction synchronization unit 236, and FIG. 11 is a flowchart showing the flow of the synchronization processing by the operation instruction synchronization unit 236. As shown in FIG. 10, the full-screen window W2 on which the high-quality moving image is displayed by the effect engine 230 is instructed to the user for various processing operations for the AV content realized by the moving image reception / playback program 220. Various operation buttons b1 to b5 (operation image data) of the player, which are operation instruction members for causing the operation, are also captured. As illustrated in FIG. 10, the operation instruction synchronization unit 236 is a pointing device for any of the operation buttons b1 to b5 displayed in the full-screen window W2 on which the high-quality moving image is displayed by the effect engine 230. When an event due to a click by 27, mouse over, or key operation on the keyboard (KB) 26 is detected (Yes in step S11: event detection unit 236a), the process proceeds to step S12.

  Subsequently, the operation instruction synchronization unit 236 acquires the coordinate position on the entire screen where the click or mouse over by the pointing device 27 or the key operation / pointing operation by the keyboard (KB) 26 is performed (step S12), and captures the acquired coordinates. Movie reception / playback program that is played back in the background (hidden under the full-screen window W2 on which the high-quality moving image is displayed by the effect engine 230) by converting to correspond to the size The message is converted into a coordinate system on the display image data output by 220 (step S13: coordinate system conversion unit 236c), and the message (pointer event) is obtained by replacing the coordinate position information portion of the obtained pointer event with the converted coordinate position information. Convert (step S14) and point A menu M (for selecting an instruction for a function that the effect engine 230 has, as will be described later, is the position (acquired coordinates) on the entire screen where the key 27 is clicked by the vice 27, the mouse over, or the keyboard (KB) 26. If not (see step S15, No), the converted message is transferred to the moving image reception / playback program 220 (step S16: event transfer unit 236b). Thereby, when the acquired coordinate position in the whole screen is on the various operation buttons b1 to b5, it is possible to realize the operation on the various operation buttons b1 to b5 of the player realized by the moving image reception / playback program 220.

  Here, the effect engine 230 may not only pass the pointer event to the moving image reception / playback program 220 but also process it by itself. For example, when an event for moving the pointing device 27 occurs, the effect engine 235 performs display control so that the menu M is overlaid on the moving image at the top of the window W2.

  It should be noted that the operation instruction synchronization unit 236 has a position (acquired coordinates) on the entire screen where the key is operated by the click or mouse over by the pointing device 27 or the keyboard (KB) 26 for the function of the effect engine 230. If the area is on the menu M for selecting an instruction (see FIG. 12C) (Yes in step S15), a pointer event is passed only to the capture setting unit 231 (step S17), and processing for the effect engine 230 is performed. Is executed. Here, as described above, when the “moving pointing device 27” event occurs on the full-screen window W2, the menu M appears regardless of the location (coordinate position) (FIGS. 12B and 12C). )reference). Thereafter, when the pointing device 27 stops, the menu M disappears unless the pointer position of the pointing device 27 is on the menu M. When the menu M option is clicked with the pointing device 27, the function of the effect engine 230 associated with the menu M option is executed. At this time, no pointer event is output to the moving image reception / playback program 220.

  In this way, the full-screen W2 on which the high-quality moving image is displayed by the effect engine 230 is displayed by the mouse operation area on the full-screen window W2 on which the high-quality moving image is displayed by the effect engine 230. And the operation processing for the various operation buttons b1 to b5 of the player realized by the moving image reception / playback program 220 can be made different.

  As a result, the result of operating the full-screen window W2 on which the high-quality moving image is displayed by the effect engine 230 is reflected in the operations on the various operation buttons b1 to b5 of the player realized by the moving image reception / playback program 220. Then, a moving image by the moving image reception / playback program 220 reflecting this operation is further captured by the capture setting unit 231 of the effect engine 230, and as a result, a moving image whose quality is improved by the effect engine 230 is displayed on the full screen. The playback state also changes. Therefore, the user is not aware of the positional relationship between the full-screen window W2 on which the moving image with high image quality is displayed by the effect engine 230 and the moving image playback window W1 by the moving image reception / playback program 220, and the effect engine 230 Thus, the full-screen playback window W2 on which the high-quality moving image is displayed can be recognized and operated as the full-screen state of the moving image reception / playback program 220.

  As described above, according to the present embodiment, the image data included in the content reproduced by the image reproduction means and the image reproduction means are displayed together with the image data, and the user is instructed to perform various processing operations on the content. When a pointer event is detected with respect to display image data processed by the image processing means by fetching the operation image data, the coordinate position information of the pointer event is converted and output to the image reproduction means. When the coordinate position of the pointer event is the operation image data part, the result of operating the display image data reproduced by the image processing means by executing the process associated with the operation image data is It is reflected in the operation for the operation image data that it has, and the image playback means reflecting this operation is restarted. The image data to be processed is further captured by the image processing means, and as a result, the display state of the display image data output from the image processing means also changes, so that the user reproduces the display image data displayed by the image processing means and the image reproduction means. The image data reproduced by the image reproduction means and the display image data reproduced by the image processing means can be operated simultaneously without being aware of the positional relationship with the display image data.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, you may combine a component suitably in different embodiment.

It is a block diagram which shows the one usage pattern of the computer concerning one Embodiment of this invention. It is a block diagram which shows the hardware constitutions of a computer. It is a block diagram which shows schematic structure of the program group for operating a computer. It is a conceptual diagram for demonstrating an example of the rewriting process of the routine in a DLL part by a capture setting part. It is a conceptual diagram for demonstrating the output principle of video data and audio data. It is a flowchart which shows the flow of the determination process of the execution instruction of the main process in an effect engine. It is a front view which shows the state in which the image (moving image) of the video data contained in AV content by a moving image reception reproduction program is output. It is a front view which shows the state in which the button for operation of an image quality improvement process is displayed. It is a block diagram which shows the function structure of an operation instruction synchronizer. It is a schematic diagram which shows the operation | movement outline | summary of the synchronous process by an operation instruction | indication synchronizing part. It is a flowchart which shows the flow of the synchronous process by the operation instruction | indication synchronous part. It is a front view which shows the state in which the menu is displayed.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 220 Image reproduction means 230 Image reproduction apparatus 236a Event detection means 236b Event output means 236c Coordinate position information conversion means 240 Image processing means b1-b5 Operation image data

Claims (10)

Moving image data obtained by decoding the encoded data by the image reproducing means, and image data output by the image reproducing means together with the moving image data, for providing the user with various processing operations on the moving image data Image processing means for performing image processing on the first display image data including the operation image data so as to be displayed as second display image data; and
Event detection means for detecting a pointer event for the second display image data output by the image processing means;
When the pointer event is detected, coordinate position information conversion means for converting the coordinate position information included in the pointer event;
Event output means for outputting the pointer event after the coordinate position information conversion to the image reproduction means;
An image reproducing apparatus comprising: The coordinate position information conversion unit is configured to include coordinates included in the pointer event based on a coordinate relationship between a relative position of the pointer on the first display image data and a relative position of the pointer on the second display image data. Convert location information,
The image reproducing apparatus according to claim 1, wherein: The event output means outputs the pointer event including coordinate position information on the second display image data to the image processing means,
The image processing means executes processing according to a coordinate position on the second display image data in which the pointer event is detected;
The image reproducing apparatus according to claim 1, wherein: When the coordinate position on the second display image data where the pointer event is detected is within a specific range, the event output means reproduces the event and the pointer event where the event is detected in the image reproduction. Output to the image processing means without outputting to the means;
The image reproduction apparatus according to claim 3. The image processing means converts the resolution of the first display image data to generate the second display image data;
5. An image reproducing apparatus according to claim 4, wherein Image reproduction means for decoding encoded data into moving image data and outputting operation image data for providing a user with various operation functions for the moving image data;
Image processing means for performing image processing on the first display image data including the moving image data and the operation image data output by the image reproducing means and outputting the second display image data so as to be displayable;
Event detection means for detecting a pointer event for the second display image data output by the image processing means;
When the pointer event is detected, coordinate position information conversion means for converting the coordinate position information included in the pointer event;
Event output means for outputting the pointer event after the coordinate position information conversion to the image reproduction means;
With
The image reproduction means performs processing associated with the operation image data when the coordinate position on the first display image data of the pointer event output from the event output means is on the operation image data. Execute,
An image processing apparatus. The coordinate position information conversion unit is configured to include coordinates included in the pointer event based on a coordinate relationship between a relative position of the pointer on the first display image data and a relative position of the pointer on the second display image data. Convert location information,
The image processing apparatus according to claim 6. The event output means outputs the pointer event including coordinate position information on the second display image data to the image processing means,
The image processing means executes processing according to a coordinate position on the second display image data in which the pointer event is detected;
The image processing apparatus according to claim 6. When the coordinate position on the second display image data where the pointer event is detected is within a specific range, the event output means reproduces the event and the pointer event where the event is detected in the image reproduction. Output to the image processing means without outputting to the means;
The image processing apparatus according to claim 8. The image processing means converts the resolution of the first display image data to generate the second display image data;
The image processing apparatus according to claim 9.

Images