JP4113564B1 - Operation image reproduction apparatus and operation image reproduction program - Google Patents

Abstract

Provided is a technique for facilitating operation tracking when confirming or monitoring a noticed user operation while reproducing an operation image on a terminal.
An operation image reproducing apparatus that acquires an operation image in a terminal and an operation event in the terminal and reproduces the operation image. A transition implied image creation unit that creates a transition implied image that implies transition information when transitioning from a preceding operation event that is a preceding operation event to a subsequent operation event that is an operation event subsequent to this preceding operation event will be played back Replay information processing means for displaying on the monitor together with the replay operation image a transitional implied image corresponding to the replay operation image.
[Selection] Figure 1

Description

  The present invention relates to an operation image reproduction technique for acquiring an operation image obtained by capturing an operation screen on a terminal and an operation event log describing an operation event on the terminal and reproducing the operation image.

  In a system in which the remote control server and client terminal are connected via a communication network, the monitor screen displayed on the monitor is transmitted to the remote control server and received by the remote control server when the operator remotely operates the terminal. The captured monitor screen is displayed on the monitor, and the captured image recording means of the remote control server records the monitor screen of the client terminal as a captured image (operation image) in the operation event log at a predetermined interval. There is one in which a log reproduction processing unit of a remote control server reproduces a captured image recorded in an operation event log (see, for example, Patent Document 1). In this prior art, the operation image stored in the data storage unit can be reproduced on the monitor monitor to monitor the operation content of the terminal by the user. However, when the user wants to monitor by paying attention to a specific dangerous operation. Even if there are, other operations appear frequently, and the monitoring work becomes distracting. For this reason, it is difficult to reliably find an operation that is prohibited in some cases, such as copying a file and copying it to a portable memory.

  In addition to receiving information such as the water level and gate opening from each district river facility, the river management facilities that should be operated (monitored) based on the importance and safety based on these information are listed in the monitoring list display area in order. A wide area monitoring system that displays and displays an image of a CCTV selected in a facility monitoring area in a monitoring operation area is known (see, for example, Patent Document 2). In this wide-area monitoring system, information such as the water level and gate opening that describes the state of the river as a monitoring target and CCTV images that display the state of the river are sent from the monitoring target site to the monitoring center. While monitoring the state of the river. However, since this monitoring system is intended to monitor the state of rivers that change from moment to moment, it is naturally built for real-time river monitoring, and sent measurement information and monitoring images (TV images). It is not considered to monitor the river while recording and reproducing it, and there is no suggestion about facilitating monitoring when the surveillance image is reproduced.

  Furthermore, in order to visually confirm the user operation on the image, the time position and spatial coordinate data in the video when the user clicks on a specific subject in the video is saved as a click operation event log, and the video is being played back In addition, a technique for superimposing (combining) and displaying the click operation event log is known (for example, see Patent Document 3). With this technique, it is possible to visually confirm a click operation such as what scene the user clicked at the time of reproduction. However, this technique only displays the user's click operation on an image that was not visible in real time as an image at the playback time when the playback was clicked. It is not possible to contribute to a request that facilitates confirmation of specific operations.

JP 2006-108947 A (paragraph numbers 0094 to 0099, FIGS. 1 and 17) Japanese Patent Laying-Open No. 2006-41770 (paragraph numbers 0035 to 0051, FIGS. 1 and 2) Japanese Patent Laying-Open No. 2004-110096 (paragraph numbers [0022] to 0030, FIG. 3)

  In general, when a user operation on a computer screen (operation screen) using a pointing device such as a mouse is confirmed or monitored by playing an operation image obtained by capturing the operation screen, Since the operation event occurrence point, that is, the operation point position jumps in all directions, skill and experience are required to track the operation. In view of the above situation, an object of the present invention is to provide a technique for facilitating operation tracking when confirming or monitoring a noticeable user operation while reproducing an operation image on a terminal. .

In order to solve the above-mentioned problem, an operation image obtained by capturing an operation screen on a terminal and an operation event log describing an operation event on the terminal are acquired, and the operation image reproduction apparatus performs reproduction of the operation image. In the apparatus according to the present invention, a transition implied image is generated that implies the operation content of the subsequent operation event when shifting from the preceding operation event that is the preceding operation event to the subsequent operation event that is the operation event subsequent to the preceding operation event. A transitional implied image creation unit and a reproduction information processing means for displaying the transitional implied image corresponding to the reproduction operation image that is the operation image to be reproduced together with the reproduction operation image on a monitor are provided.

According to the above configuration, the operation content of the operation event (subsequent operation event) performed following the operation event (previous operation event) occurring in the operation image is implied on the operation image reproduced on the monitor. Since the transition implied image is displayed, even if the operation event occurrence point, that is, the operation point position jumps in all directions, the transition implied image makes it easier to predict subsequent operations, Confirmation is easy. Note that a plurality of subsequent operation events as operation events performed following the preceding operation event may be defined, and in that case, a plurality of transitional implied images are also displayed.
In addition, by adopting a configuration in which the transition implied image created by the transition implied image creation unit is assigned to the operation event of the transition source, the operation event and the transition implied image are reliably linked before the operation image is played back. It can be stored in a state and can contribute to speeding up of processing.

  For example, if the operation content of the subsequent operation event is displayed as the transitional implied image, the subsequent operation can be known in advance, so that monitoring can be performed with a margin. In addition, the movement direction of the operating point of a pointing device such as a mouse can be predicted just by displaying an arrow line from the operation position on the operation image of the preceding operation event to the operation position of the subsequent operation event as the transition implied image, Confirmation of the terminal operation through the reproduction operation image becomes easy. Also, since the time from one operation to the next operation is long and short, the observer has the frustration of waiting for the next operation that may start, but if the subsequent operation event is used as a transition suggestion image If a time lapse image representing a period until the occurrence of an error is included, the terminal monitoring can be performed without being frustrated. For example, this time lapse image may be expressed as an hourglass or a slide bar, but if it is a countdown time, it is possible to easily know the time until an operation event occurs.

Many of the user operations to be noticed when playing back operation images are, for example, “access shared folder” (preceding operation event), “select confidential file” (intermediate operation event) in the shared folder, and “confidential file” Is transferred to a removable disk ”(final operation event). Therefore, in a preferred embodiment of the present invention, a configuration in which a display of a transitional implicit image displayed together with an operation image is continued over a plurality of subsequent operation images (a plurality of continuous transitions to one operation image). A configuration in which an implied image is displayed is employed. At that time, if the transitional suggestion image is displayed continuously to visually differentiate the transitional implied image indicating the transition information of the past operation event and the transitional implied image indicating the transition information to the operation event that will occur in the future , It is advantageous if the display form is changed between the transition implied image implying the operation content of the subsequent operation event when transitioning to the operation event and the transition implied image displayed continuously .
In addition, even when a user operation that does not require much attention from the monitor is displayed, it may be annoying if the transitional implicit image is displayed. In this case, it is convenient that the type of user operation to be watched by the monitor, that is, the type of attention event, is specified in advance, and the transitional implied image is displayed only for the specific user operation. For this purpose, in one of the preferred embodiments of the present invention, an attention operation event type designation unit for designating an attention operation event type to be monitored with priority is provided, and the subsequent operation event is designated as an attention operation event type designation. The transition implied image corresponding only to the case of the attention operation event type specified in the section is displayed on the monitor monitor together with the reproduction operation image.

It is necessary to associate the operation image and the operation event that coincide or approximate in time with each other in terms of time. In this regard, the operation image and the operation event are associated with each other based on the time stamp assigned to the operation event. If the link unit that links the operation image to be operated and the operation event is provided, it is convenient that the operation image and the operation event can be stored in a linked state reliably. Further, when the operation image and the operation event are linked and stored for each terminal by a link code such as an index code in the link unit, a desired operation image and operation event can be searched and extracted at high speed. In addition, when the acquisition timing of the operation event and the capture timing of the operation image greatly deviate and it is difficult to take a time correspondence between them, it is particularly time for the operation event to be processed in the present invention. If an operation image corresponding to the operation event is missing, an interpolation operation image generation unit that generates an interpolation operation image representing the operation event is provided. Can be made.
On the contrary, if there is a possibility that there is no operation event corresponding to the operation image in time, the operation event is linked to the operation event linked to the preceding operation image and the subsequent operation event. An interpolation operation event creation unit that creates an interpolation operation event describing the contents of the operation process in the middle of the operation process may be provided. Of course, if necessary, both an interpolation operation image generation unit and an interpolation operation event creation unit may be provided.

In an operation image reproduction program for an operation image reproduction device that obtains an operation image obtained by capturing an operation screen on a terminal and an operation event log describing an operation event on the terminal and reproduces the operation image In order to solve the above-described problem, the terminal monitoring program according to the present invention is configured to detect a subsequent operation event when shifting from a preceding operation event that is a preceding operation event to a subsequent operation event that is an operation event subsequent to the preceding operation event. wherein to achieve the function of displaying the transition implied image on the monitor with the reproduction operation image on the computer corresponding to the operation is the image reproducing operation image ability to create a migration implied image to imply operation contents about to be reproduced . The operation image reproduction program configured in this way also has the operational effects described in the previous operation image reproduction device, and can have various additional characteristic configurations described above.

  The operation image on the terminal is associated with the operation event in time, and when the operation image is reproduced on the monitor, when the operation image is displayed, the operation event linked to the operation image and the operation image displayed following the operation image are displayed. First, the basic principle of the operation image reproduction technique according to the present invention capable of predicting the contents of the next user operation, in which a transition implied image implying the transition information between the linked operation events, is also described. FIG. 1 shows a terminal 1 composed of a general-purpose notebook computer or desktop personal computer, and an operation image reproduction device 3 for reproducing an operation screen of a user operation on the terminal 1, for example, in a network (an organization such as a LAN). Or a wide-area network that is open like the Internet) or the like, and a computer system connected through a data transmission line 2 is shown. In the terminal 1, an operation event log that sequentially describes operation events representing each user operation is created, and an image that creates a monitor screen (operation screen) at a predetermined interval (for example, every 1 second) is captured as an operation image. (# 01). The operation event log and the operation image are sent to the operation image reproduction device 3 through the data transmission line 2 (# 02). Before the operation event described in the operation event log sent to the terminal monitoring device 3 and the operation image similarly sent to the operation image reproduction device 3 are stored in a data storage unit such as a large-capacity hard disk, Alternatively, once stored, for example, a time code indicating an acquisition time such as a time stamp is referred to and matched with a matching time or an approximate time as a condition (# 03). This association may be performed when the operation image is reproduced. However, when there is no operation image corresponding to the operation event within a predetermined time range (for example, 1 second), the interpolation operation image having the operation content estimated that the operation event is performed from the content of the operation event. May be generated and linked to the corresponding operation event (# 04). Of course, conversely, when an operation event corresponding to time is missing, an interpolation operation event having contents that can be estimated from previous and subsequent operation events may be generated. Next, the operation events that are temporally related, here called the predecessor operation event and the subsequent operation event, are referred to as the operation event subsequent to the predecessor operation event (next operation event, next next operation event, and further. Transition implied images that imply transition information when transitioning to subsequent operation events that are operation events...) Are sequentially created (# 05). The sequentially generated transitional implied images are assigned to each operation event, or can be associated with each other in time using a time stamp possessed as an attribute value. (# 06). In FIG. 1, the preceding operation event and the subsequent operation event are described in a one-to-one correspondence. However, a plurality of subsequent operation events subsequent thereto may be associated with each preceding operation event. A plurality of transition implied images are associated with the event. In addition, in order to avoid the annoyance caused by the display of the transition implied image for each user operation that does not require much attention from the supervisor, the transition implied image is generated and linked only to a specific noticed operation event. You may do it. In order to check the operation screen of the specific terminal on the monitor, the operation image is reproduced on the monitoring monitor along the time axis, that is, in the order of capture (# 07). When the operation image is reproduced, the transitional implied image assigned to the operation event linked to the operation image is called up, combined with the operation image, and displayed on the monitor (# 08).

  The simplest example of a transition-implied image is a sign indicating the movement of the mouse for the next operation, an arrow line extending to the next operation position, an illustration showing a right-click or left-click of the mouse, or an open showing an application launch For example, a rectangle indicating the frame of the window to be displayed. Of course, a mode in which the operation content of the corresponding subsequent operation event is directly displayed in text may be adopted. In addition, the time from the preceding operation event to the subsequent operation event is calculated from the time stamps assigned to each, and the time period until this attention event occurs is visualized (time lapse image: count time and time A progress bar graph or the like) may be used as a transitional implicit image.

  FIG. 2 shows a configuration of a terminal monitoring system as an example of a terminal operation confirmation system that employs the basic principle of the operation image reproduction technique according to the present invention described above. In this terminal monitoring system, a large number of terminals 1 are connected to a terminal monitoring apparatus 3 as an operation image reproduction apparatus via a network 2. The terminal 1 is a general personal computer, and operations related to various data processing and data browsing, such as creation, browsing, copying, correction, deletion, printing, Internet browsing, mail transmission, and the like of files, etc. are performed by the user. A user name or a user code input through a login operation performed before the user uses the terminal 1 is sent to the terminal monitoring apparatus 3 as user attribute information together with the terminal ID, so that the terminal monitoring apparatus 3 The user who is using can be specified. Furthermore, from the terminal 1 to the terminal monitoring device 3, an operation event log describing an operation event performed between the time when the user logs in and the time when the user logs off, and an operation image obtained by capturing the operation screen each time are provided in a timely manner. Sent. The terminal monitoring device 3 can monitor the operation performed on the terminal by reproducing the transmitted operation image for each terminal 1 in real time or after recording.

  The terminal 1 includes a monitor 51 such as a liquid crystal display, a keyboard and a mouse as the input operation device 52, a stationary hard disk 53 for storing various data, and a removable recording medium (USB memory or the like) 54. Various operation commands input by the user through the operation screen displayed on the monitor 51 are executed by the GUI unit 10 created by the OS installed in the terminal 1 as an intermediary. Data processing based on user instructions including various file processing is performed by the data processing unit 11. At this time, in order to display various operation screens on the monitor 51, these display data are temporarily stored in the video memory 12, but the display data developed in the video memory 12, that is, operation images representing the operation screens. Is captured by the operation screen capture unit 13. In addition, all operations performed by the user through the input operation device 52 are performed by the operation event log generation unit 14 as operation events such as operation execution program names, operation target data names (file names), operation command names, operation screens (operation images). It is described in the form of enumerating the operation positions in the above, and these operation events are scripted as operation event logs. The operation event log generated by the operation event log generation unit 14 and the captured image captured by the operation screen capture unit 13 are sent to the terminal management device 3 connected by the network 2 through the network IF. In order to facilitate management, a common time stamp and user attribute information are assigned to the operation event log and the captured image.

  Normally, in order to use the terminal 1, the user must first log in by entering the user name and password. The user name obtained through the login process is used as it is or in the login / logoff control unit 15. Accordingly, it is converted into a user ID or the like, and becomes user (terminal) attribute information to be sent to the terminal monitoring device 3 together with a terminal ID for identifying the terminal 1.

The terminal monitoring apparatus 3 is also generally constituted by a general-purpose computer, and is attached with a monitoring monitor 55 such as a liquid crystal display, a data storage unit 57 including a keyboard, a mouse, a large-capacity hard disk, etc. as an input operation device 56. . A terminal that manages the terminal specified based on the user attribute information related to the terminal operating user sent from the GUI unit 30 or the terminal 1 by the OS installed in the terminal monitoring device 3 and the user operating this terminal Various basic functions such as the management unit 31 are created. Further, most of the functions related to the present invention in the terminal monitoring device 3 are created along with the execution of the terminal monitoring program installed in the terminal monitoring device 3, and have been sent from the terminal 1 as typical ones. An operation event log acquisition unit 32, an operation image acquisition unit 33, and an operation event log acquisition that receive an operation event log describing an operation event performed by the user and an operation image indicating an operation screen at that time via the network 2. The operation event storage unit 57a formed in the data storage unit 57 for storing the operation event for each terminal extracted by the unit 32, and the acquired operation image can be read out for each terminal and in time series. An operation image management unit 36 stored in the operation image storage unit 57b formed in the data storage unit 57 is provided. Further, as described with reference to FIG. 1, the operation event stored in the operation event storage unit 57a has a next operation event determination unit 38a that associates a preceding operation event with a subsequent operation event. An operation event management unit 38 is provided that generates transition information when shifting from each preceding operation event to each subsequent operation event, for example, “select a file and drag (move)”. Note that when the transition information is created by the operation event management unit 38 only for the transition to a specific type of subsequent operation event, the type of the specific operation event to be noted (subsequent operation event) is set. A noticed operation event type designation unit 39 that is designated and registered in the noticed operation event type registration table 39a is provided. The operation event type to be noted is input to the attention operation event type designation unit 39 through the GUI unit 30 and is referred to the operation event management unit 38 through the attention operation event registration table 39a. In this embodiment, in order to be able to search and extract the acquired operation event and the operation image at the time of occurrence of the operation event at high speed, a link code such as an index code is added, and the operation image and the operation event are classified for each terminal. And the like, “mouse click (selection)”, “mouse drag”, “keyboard character input”, and the like for the operation event for each terminal extracted by the operation event log acquisition unit 32. An operation event classifying unit 35 that assigns a type code for each operation type is provided. Furthermore, as described above, when there is no operation image that temporally corresponds to the operation event to be processed, the interpolation operation image generation unit 37 that generates an interpolation operation image representing the operation event is provided. ing. The link unit 34 may link the operation image and the operation event corresponding to each other based on the time stamp given to the acquired operation image and the operation event.

  As a reproduction information processing unit that creates a migration implied image that implies the content based on the migration information given to the operation event by the operation event management unit 38 and displays it on the monitor monitor 55 together with the operation image to be reproduced. The monitoring information processing means 40 receives the operation image (reproduction operation image) to be reproduced from the operation image management unit 36 and also receives, from the operation event management unit 38, the transition information given to the operation image. The monitoring information processing means 40 includes a transition implied image creation unit 41 that creates a transition implied image for implying the content based on the transition information given to the operation event, and an operation event to which the transition implied image is assigned. An image synthesizing unit 42 for synthesizing with the reproduction operation image linked to the reproduction operation image, a reproduction operation image synthesized with the transitional implied image, and, in some cases, a reproduction operation image without such synthesis is reproduced on the monitoring monitor 55. An operation image reproduction unit 43 to be processed is included.

  Here, user operations performed on the terminal 1 called X1, for example, “select shared folder”, “open (access) shared folder”, “select xxx.xls file”, “moving and copying xxx.xls file” ”,“ Right click on the selected xxx.xls file ”,“ Transfer (write) to USB memory ”,..., And FIG. 3 shows the operation image captured and sent in the operation process. An operation event log created and sent during the operation process is shown. Furthermore, for the operation event corresponding to each operation image in time, the operation event management unit 38 creates the operation event based on the content of the operation event and the content of the subsequent operation event having the operation event as the preceding operation event. The transition implied image created by the transition implied image creation unit 41 using the transition information is directly assigned to the operation event, or the time of the preceding operation event corresponding to the transition implied image is assigned. Give a stamp. FIG. 3 also schematically shows the continuous connection of the operation screen, the operation event, and the transitional implied image with time. In the explanatory diagram of FIG. 3, the preceding operation event (preceding operation event) is accompanied by only one piece of transition information and, as a result, only one piece of migration implied image. As an example, event ID: E021 “select shared folder”, event ID: E022 “open shared folder (access)”, event ID: E023 “select xxx.xls file”, event ID: E024 “xxx” “Move and copy of .xls file”, event ID: E025 “right click on selected xxx.xls file”, event ID: E026 “write to USB memory”. Then, each operation event is sequentially regarded as a preceding operation event and a subsequent operation event, and transition information is created. Based on this transition information, a transition suggestion image is created. As is apparent from the figure, here, a symbol mark that symbolically represents the content of the subsequent operation event is employed as the transitional implied image. Instead of creating such a transitional implied image each time, it is preferable to create and register in advance and create a transitional implied image by calling it.

  In this embodiment, a symbol mark of the operation content of the subsequent operation event, an arrow line indicating the moving direction of the mouse when moving from the previous operation event to the subsequent operation event, and the like are used as the transfer implicit image (see FIG. 3 (see S014 in FIG. 3), and the operation content of the operation event that has become the subsequent operation event, for example, the words “write xxx.xls file to USB memory” are drawn in a balloon frame, as shown in FIG. A simple implied image may be employed. In addition, the countdown variable inserted in parentheses in the wording is a variable for displaying the calculated value of the time from the occurrence of the preceding operation event to the occurrence of the subsequent operation event, and the preceding operation event as the starting point The contents of the variable, that is, the calculated value (countdown value) varies depending on the time of occurrence of. In addition, when the playback speed in the operation image playback is the actual speed, the countdown value may be the difference in time when each operation event occurs, but when the playback speed is not the actual speed, that is, in the case of slow playback or fast forward playback, The countdown value may be changed in accordance with the playback speed so that the actual time is increased so that it becomes longer during slow playback and shorter during fast forward playback. Accordingly, it is possible to grasp the time from the preceding operation event to the occurrence of the subsequent operation event with the actual time without being aware of whether the reproduction screen is slow reproduction or fast forward reproduction slow.

  As shown in FIG. 3, when the operation image to which the transitional implied image is assigned is continuously reproduced on the monitor monitor 55 by the operation of the operation image reproduction unit 43, a screen as shown in FIG. Is done. When an operation image linked to the first operation event (event ID: E021) is to be reproduced, the countdown variable is assigned a countdown value in which the time from the preceding operation event to the subsequent operation event is given to the countdown variable. Is displayed on the monitoring monitor 55 in a form synthesized with the operation image (see FIG. 5A). From the screen of the monitoring monitor 55, the supervisor can know that the shared folder will be opened after one second, and the concentration of monitoring can be enhanced. Further, the operation image linked to the next operation event (event ID: E022) is reproduced (see FIG. 5B). Here, the symbol mark of “file selection” and the countdown value until “file selection” are combined with the operation image and displayed as a transitional implied image. Similarly, as shown in FIG. 5C, FIG. 5D, and FIG. 5E, the symbol mark representing the transition information and the countdown value are combined with the operation image and displayed as a transition implied image. In particular, in FIG. 5 (e), a transitional implied image (an explosion mark is adopted as a symbol mark) is displayed indicating that “write file to USB memory”, which is a notable operation event, will be performed next. And the operation is shown to occur after 2 seconds. Then, the operation image linked to the notable operation event (event ID: E026) is reproduced (see FIG. 5 (f)), and the notable operation at this time (file writing to the USB memory) Can be understood at this point.

In the above-described reproduction of the transitional implied image synthesized with the operation image on the monitoring monitor 55, the one-to-one relationship among the operational image, the operation event, and the transitional implied image as shown in FIG. 3 is maintained as it is. Then, every time the operation image is switched, the transitional implied image is also switched. Instead of this method, in order to make it easier to understand the progress of the user operation, several operation images that precede the operation image when the operation image is played back, that is, the transitional implied image assigned to the operation image that has already been displayed, are also included together. It is also possible to synthesize. The screen of the monitoring monitor 55 when such a reproduction mode is selected is illustrated in FIG. In this example, the display of the countdown value is omitted. In the reproduction of the operation image linked to the first operation event (event ID: E021), only the shared implicit image (folder open image) to be opened is selected by combining the shared folder. (See FIG. 6A), when the next operation screen is reproduced, not only the transition image assigned to the operation event (event ID: E022) linked to the operation image to be reproduced, but also the previous image A transitional implied image that is a folder open image is also synthesized (see FIG. 5B). That is, with only the transitional implied image assigned to the operation event (event ID: E021) remaining, the next operational image and the transitional implied image assigned to the operation operational event (event ID: E022) linked to the next operational image. Since both are displayed, it becomes easy to observe the flow of user operations. However, both are displayed to distinguish between the transitional implied image of the upcoming operation event corresponding to the operation image that is actually played back and the transitional implied image of the operation event that has already occurred corresponding to the past operational image. The form is changing. In FIG. 6, the transition-implied image of the operation event that has already occurred is indicated by a dotted line, but an appropriate method for image processing such as semi-transparency may be used in practice. In the reproduction of the transition operation screen, as shown in FIG. 6C, FIG. 6D, and FIG. 6E, an operation image that also tries to reproduce the transition implied image representing the transition information going back in the past. Are combined with the operation image and displayed together with the transitional implied image corresponding to. Then, the operation image linked to the notable operation event (event ID: E026) is reproduced (see FIG. 6 (f)), and the notable operation (file writing to the USB memory) is performed. It can be easily grasped by the transitional implied image in which the operation procedure is continuous.

  FIG. 7 shows still another embodiment in which the transitional implied image is combined with the operation image to be reproduced. In this embodiment, when the operation image is played back, a plurality of transitional implied images are synthesized that imply that the operation event temporally corresponding to the operation image transitions to a plurality of subsequent operation events. it's shown. In other words, several previous operation images that continue to the operation image when the operation image is reproduced, that is, the transitional implied image assigned to the already displayed operation image are also synthesized together. FIG. 7 shows an example in which three implied images implying the transition to three subsequent operation events are assigned to one operation image to be reproduced. The transition implied image based on transition information between events is the one shown in FIG. As is apparent from the figure, the reproduction operation image: S011 is transferred to the operation event (event ID: E021) via the operation event (event ID: E021), which is the content of the next operation event (event ID: E022). An image, a migration implied image relating to “selection of xxx.xls file” which is the content of the next operation event (event ID: E023), and the content of the next operation event (event ID: E024) A migration implied image related to a certain “xxx.xls file moving duplication” is assigned. Similarly, in the replay operation image: S012, a transitional implied image regarding “selection of xxx.xls file” which is the content of the next operation event (event ID: E023) and the next next operation event (event ID: E024) are displayed. ) Is the migration implied image related to “moving duplication of xxx.xls file”, and the next next operation event content (event ID: E025) “right click on the selected xxx.xls file ”And a transitional implied image. Three transitional implied images are sequentially assigned to subsequent operation images. As a result, as schematically shown in FIG. 7, three transitional implied images implying an operation event to be generated are combined and displayed on the reproduction operation image finally displayed on the monitor monitor 55. Is done. Of course, in principle, a plurality of preset migration implied images will be combined and displayed, but if the contents of the operation event that will occur in the future are negligible, a combined display of the transition implied images related to this will be displayed. May be omitted to avoid complications on the monitor screen that may be caused by a plurality of transitional implied image displays.

  The terminal monitoring device 3 stores the operation image and the operation event log for each terminal 1 acquired by the operation image acquisition unit 33 and the operation event log acquisition unit 32 in the data storage unit 57 as described above. In addition to a mode for selectively reproducing, various display modes for terminal monitoring such as a mode for directly displaying operation images and operation event logs on the monitoring monitor 51 before storing them in the data storage unit 57 are provided. FIG. 8 shows a terminal monitoring basic screen 100 displayed on the monitoring monitor 55 for selecting a desired mode from such various display modes. On the terminal monitoring basic screen 100, an operation image playback button 101, a log browsing button 102, a PC information button 103, a remote control button 104, an alert setting button 105, and the like are arranged in the upper position in the form of icons. At the bottom of the basic screen 100, an operation image display area 110 for each terminal, a message display unit 111, and a frame 120 for displaying a real-time operation image of each terminal 1 are displayed in a matrix. A terminal identification display unit 121 indicating a terminal name or a terminal ID is arranged at the bottom.

  When the operation image playback button 101 is operated on the terminal monitoring basic screen 100, the operation image of the terminal 1 selected at that time is read from the operation image storage unit 57b as shown in FIG. A replay screen 140 is displayed. Before operating the operation image playback button 101, the terminal 1 displayed in the monitor screen display area 110 or the terminal list display unit 112 of the terminal monitoring basic screen 100 is selected. In this case, a dialog box (not shown) may be displayed, and information specifying the terminal 1 may be input from the input operation device 52. The playback screen 140 includes an attention operation event type designation field 141, a playback start date / time input unit 142, a playback end date / time input unit 143, an OK button 144, a playback area 145, a plurality of playback control buttons 146, A playback slider 147 is arranged.

  A typical method for displaying the operation image on the playback screen 140 is to input the playback start date and time from the keyboard or the like in the playback start date and time input unit 142 and input the playback end time to the playback end date and time input unit 143 as necessary. After that, the user clicks the OK button 144. As a result, the specified operation image of the terminal 1 is read from the operation image storage unit 57 b by the operation image management unit 36 and displayed in the reproduction area 145 in time series. At this time, as described above, the subsequent operation is performed only when the operation event belonging to the attention operation event type input through the attention operation event type designation column 141 and set by the attention operation event type designation unit 39 becomes the subsequent operation event. A transitional implied image indicating the contents of the event is created and displayed together with the operation image linked to the preceding operation event of the subsequent operation event.

  When the log browsing button 102 is operated on the basic screen 100 shown in FIG. 8, the operation event log list is displayed in a separate window. When the PC information button 103 is operated on the basic screen 100, a process of displaying PC information such as physical data of the selected terminal 1 and an installed application name is performed. When the remote control button 104 is operated on the basic screen 100, it is possible to remotely control individual terminals by operating icons or executing commands on the remote control screen.

  In the above-described embodiment, when the operation image playback button 101 is operated, the operation image of the terminal 1 selected at that time is read from the operation image storage unit 57b, that is, as recording / playback, as shown in FIG. The operation image with the transitional implied image is reproduced and displayed through the reproduction screen 140 as shown in FIG. However, on the one hand, the operation image acquired by the operation image acquisition unit 33 is sent from the link unit 34 directly to the image composition unit 42 via the operation image management 36, and on the other hand, the operation event is transmitted from the link unit 34 to the operation event classification unit. 35, if the transitional implied image sent to the operation event management unit 38 and assigned to the operational event is created by the transitional implied image creation unit 40, the operation image with the transitional implied image is monitored in real time. It is also possible to play back at 55. Therefore, in the present invention, the operation image with the transitional implied image is reproduced not only by recording and reproduction but also substantially in real time (although there is actually a delay of several seconds because it is necessary to wait for the acquisition of the subsequent operation event). It includes the reproduction of.

  Furthermore, in the above-described embodiment, when an operation event corresponding to the operation image in time is lacking, the interpolation operation event generation unit 37 generates an interpolation operation event from the preceding operation event and the subsequent operation event. The generated interpolation operation event was linked to the operation image. Instead, the transitional implied image creation unit 41 performs the preceding and following operations on an operation screen that lacks an operation event corresponding to time. An interpolated transitional implied image may be created and synthesized by inferring from the transitional implied image assigned to the image. In particular, when an arrow line from the operation position on the operation image of the preceding operation event to the operation position of the subsequent operation event is mainly used as the transition implied image that represents the movement of the pointing device such as a mouse, the interpolation transition is performed. Implicit images can be created easily.

  In the above-described embodiment, operation images and operation event logs in a large number of terminals 1 are sent to the terminal monitoring device 3 via the network 2, and the operation images are displayed on the monitoring monitor 55 of the terminal monitoring device 3 as necessary. Is displayed. However, the operation image reproduction function according to the present invention is modularized to be an application that can be executed on the terminal 1 and installed in each terminal 1, so that the terminal 1 itself becomes a stand-alone terminal with an operation image reproduction function (that is, a terminal). A terminal incorporating a monitoring device). Compared to the server / client type terminal monitoring system described above, there is a difference that the operation event log and the operation image are transmitted via the network 2 via the personal computer data bus. The content is the same.

  The term operation image used in this specification refers to a still image file obtained by capturing an operation screen, a collection of still image files, and a moving image obtained by converting continuous still images into animations. A collection of files and video files. Therefore, the operation image transferred from the terminal 1 to the terminal monitoring device 3 may be a still image or a moving image. Further, when the operation image is displayed on the monitoring monitor 55, a continuous display of a still image (style like a slide show) or a moving image display may be used, and an appropriate still image-moving program or moving image- A still image program is implemented. Furthermore, various types of operation images can be selected when recorded in the operation image storage unit 57b. For example, JPEG or GIF is used for still images, and MPEG or AVI is used for moving images.

  In the above-described embodiment, the terminal monitoring device 3 has been described as being configured by a single computer. However, the functions of the terminal monitoring device 3 may be divided and distributed to a plurality of computers. For example, the data storage unit 57 in the terminal monitoring device 3 is allocated to a dedicated data storage device such as a file server, and an operation image or an operation event is read from the dedicated data storage device, and the above-described processing is performed and displayed on the monitoring monitor. A monitoring terminal may be arranged. By connecting this monitoring terminal to a network incorporating a dedicated data storage device, the terminal can be monitored anytime and anywhere.

Explanatory drawing explaining the basic principle of operation screen reproduction by this invention The system block diagram which shows an example of the terminal monitoring system by this invention Explanatory drawing explaining the flow which synthesize | combines the transition implicit image regarding a specific operation event on an operation screen Another explanatory diagram when synthesizing the migration implied image on the operation screen Screen view showing the operation image with the transition implied image combined The screen figure which shows the operation image by which the suggestion image in another embodiment was synthesize | combined Schematic diagram showing a speech balloon as a form of transition implied image Screen diagram showing the terminal monitoring basic screen Screen diagram showing the playback screen

Explanation of symbols

1: Terminal 3: Operation image playback device (terminal monitoring device)
13: Operation screen capture unit 14: Operation event log generation unit 32: Operation event log acquisition unit 33: Operation image acquisition unit 34: Link unit 37: Interpolation operation image generation unit 36: Operation image management unit 38: Operation event management unit 38a : Next operation event determination unit 39: Attention operation event type designation unit 39 a: Attention operation event type registration table 40: Reproduction information processing means (monitoring information processing means)
41: Implicit image creation unit 42: Image composition unit 43: Operation image reproduction unit 57: Data storage unit 57a: Operation event storage unit 57b: Operation image storage unit 55: Monitoring monitor

Claims (9)

In the operation image reproduction device that acquires the operation image obtained by capturing the operation screen in the terminal and the operation event log describing the operation event in the terminal, and reproduces the operation image,
A transition implied image creation unit that creates a transition implied image that implies the operation content of the subsequent operation event when transitioning from the preceding operation event that is the preceding operation event to the subsequent operation event that is the operation event subsequent to the preceding operation event; ,
Reproduction information processing means for displaying on the monitor together with the reproduction operation image the transition implied image corresponding to the reproduction operation image that is the operation image to be reproduced;
An operation image reproduction apparatus provided with   The operation image reproduction device according to claim 1, wherein the migration implied image created by the migration implied image creation unit is assigned to a migration source operation event.   The link part which links the said operation image and the said operation event which respond | correspond temporally based on the said operation image and the time stamp provided to the said operation event is provided. The operation image reproducing device described in 1.   A noticed operation event type designation unit for designating a noticed operation event type to be monitored with priority is provided, and the transitional implied image is reproduced only when the subsequent operation event belongs to the designated noticed operation event type. The operation image reproduction device according to claim 1, wherein the operation image reproduction device is displayed on the monitoring monitor together with the operation image.   The operation image reproduction device according to any one of claims 1 to 4, wherein the display of the transitional implied image displayed together with the operation image is continued over a plurality of subsequent operation images. When the transitional implied image is continuously displayed , the display form is changed between the transitional implied image implying the operation content of the subsequent operation event when transitioning to the subsequent operational event and the transitional implied image continuously displayed. The operation image reproducing device according to claim 5, wherein:   The interpolated operation image generation unit that generates an interpolated operation image representing the operation event when an operation image corresponding to the operation event is missing is provided. The operation image reproduction device according to one item.   The operation image reproduction device according to claim 1, wherein the implied image includes a time lapse image that represents a period until the subsequent operation event occurs. In an operation image reproduction program for an operation image reproduction device that obtains an operation image obtained by capturing an operation screen on a terminal and an operation event log describing an operation event on the terminal and reproduces the operation image ,
A function for creating a migration implied image that implies the operation content of the subsequent operation event when moving from the preceding operation event that is the preceding operation event to the subsequent operation event that is the operation event subsequent to the preceding operation event;
A function for displaying the transitional implied image corresponding to the reproduction operation image that is the operation image to be reproduced together with the reproduction operation image on a monitor;
An operation image reproduction program for causing a computer to realize the above.

Images