CN102841789A - Method and device for recording and playing back user operations in browser - Google Patents

Abstract

The invention provides a method and a device for recording and playing back user operations in a browser. The method comprises the following steps of: starting to record the user operations; storing the configuration information of the current browser; acquiring the operation message of keyboard operations, and/or the operation message of mouse operation, and generating operation records; and converting the configuration information and the operation records of the browser into a single operation program block, and storing the single operation program block. The method and the device can be used for accurately and completely recording the operation steps of a program used by a user, and also can play back the operation steps. The stability and the reliability in the recording process and the playback process are improved, and the user operations are simplified; and functions can be conveniently integrated in the browser.

Description

A method and device for recording and playing back user operations in a browser

technical field

The present invention relates to the technical field of program intelligent operation, in particular to a method for recording and replaying user operations in a browser, and a device for recording and replaying user operations in a browser.

Background technique

Today's computer applications are becoming more and more widespread, and many things can be solved through the Internet, but for some elderly or special groups (such as the disabled), complex computer operations make them more and more powerless. Therefore, some program intelligence is needed to provide convenience. For example, you can record a program operation and generate a shortcut, and you can just click the shortcut when you need to use it next time, and the system will automatically complete the program operation.

Some existing solutions are: QuickTest Professional (QTP for short) is an automatic testing tool developed by Mercury. When the user operates, the recording mechanism of QTP can record every operation step of the user and the object being operated. Automatically generate test script statements for users to play back operations at any time. QTP provides 3 recording methods: normal recording (Normal Recording), analog recording (Analog Recording), low-level recording (LowLevel Recording).

However, there are many deficiencies in the recording method of QTP.

1. The normal recording mode is a direct operation on standard controls, and cannot recognize non-standard controls;

2. The analog recording mode is to simulate the trajectory and button operation of the mouse and keyboard. There are many invalid operations, and when the window coordinates are moved, it is easy to misuse;

3. The low-level recording mode recognizes all objects as WinObjects, and has stricter requirements on the position and cannot move the window. The x/y coordinates need to be set during recording, and the process is too complicated.

Moreover, the QTP software is too large (hundreds of MB) and is an independent program, which cannot be easily integrated in the browser.

Therefore, an urgent problem to be solved by those skilled in the art is to provide a method and device for recording and replaying user operations in the browser, so as to accurately and completely record the operation steps of the program used by the user, and be able to replay . Improve the stability and reliability of the recording and playback process, simplify the user's operation; and can easily integrate the functions in the browser.

Contents of the invention

The technical problem to be solved by the present invention is to provide a method for recording and replaying user operations in a browser, so as to accurately and completely record the operation steps of the program used by the user and replay them. Improve the stability and reliability of the recording and playback process, simplify the user's operation; and can easily integrate the functions in the browser.

Correspondingly, the present invention also provides a device for recording and playing back user operations in the browser, so as to ensure the practical application of the above method.

In order to solve the above problems, the present invention discloses a method for recording user operations in a browser, which is characterized in that it includes:

Start the recording of user actions;

Save the configuration information of the current browser;

Collect operation messages during keyboard operations and/or operation messages during mouse operations to generate operation records;

The configuration information and operation records of the browser are converted into single operation program blocks for storage.

Preferably, the step of saving the configuration information of the current browser may include:

Save the window size and position information of the current browser;

Save the setting information of interface toolbar, status bar and favorite bar.

Preferably, the step of collecting operation messages during keyboard operations, and/or operation messages during mouse operations, and generating operation records may include:

Collect the input operation and switching operation of the keyboard;

And/or collect mouse movement events and click events;

Save the lParam parameter of the HookProc function when obtaining the input operation and switching operation of the keyboard, and/or the movement event and click event of the mouse;

An operation record corresponding to the keyboard and/or mouse operation is generated according to the lParam parameter information.

Preferably, the step of collecting the input operation and switching operation of the keyboard may further include:

Start the text editing tool and enter the login information into the text editing tool;

The entered login information in text format is encrypted and written into the lParam.

Preferably, the movement event and click event of the collection mouse may include:

Monitor the mouse movement event and click event, and when the movement event is triggered, obtain the current mouse coordinates and color;

When a mouse click event occurs, save the coordinates and color saved in the previous movement event to the click color parameter clickcolor.

Preferably, the step of converting the configuration information and operation records of the browser into single operation program blocks for saving may include:

Save the corresponding browser window size and position information and the setting information of the interface toolbar, status bar and favorite bar when performing each recording operation;

The configuration information and operation records of the browser are stored in the same operation program block as a single operation program block.

Preferably, the method may further include:

Analyze the lParam parameter information in the generated operation record;

deleting an operation in which the time interval of the mouse movement event is lower than a preset invalid time threshold;

Modifying the time interval before the keyboard operation to a preset time;

The input operation and switching operation of the keyboard, and/or the movement event and click event of the mouse are recorded in sequence to generate an operation record.

Preferably, the preset invalid time threshold may be 20 milliseconds, and the preset time may be 5 milliseconds.

Preferably, the step of starting the recording of user operations may include:

Set shortcuts or shortcut keys for preset recording operations;

Click to activate the shortcut or shortcut key.

Preferably, the method may further include:

Click the preset shortcut or shortcut key to end the recording operation;

In the generated operation program block, delete the operation record of clicking the preset shortcut or the shortcut key to end the recording.

The present invention also discloses a method for replaying the login operation in the browser based on the recorded operation program block in claim 1, which may specifically include:

Loading the browser configuration information in the single operation program block;

Read the operation record, restore it to the operation message when the keyboard is operated, and/or the operation message when the mouse is operated;

According to the operation message when the keyboard is operated, and/or the operation message when the mouse is operated, a corresponding operation is performed at the target position of the browser.

Preferably, the method may also include:

Through the DocumentComplete function of WebBrowser, it is judged whether the operation playback is completed normally;

If so, then end the operation recording and playback program;

If not, the user is prompted to re-record.

Preferably, the step of judging whether the operation playback is completed normally may further include:

Check whether the current browser configuration information has been loaded;

Check whether the operation message in the operation record has been returned.

Preferably, the step of reading the operation record and restoring it to the operation message when the keyboard is operated, and/or the operation message when the mouse is operated may include:

Return the operation record to the location pointed to by lParam;

The operation record at the position pointed to by the lParam is parsed, and restored to the operation message when the keyboard is operated, and/or the operation message when the mouse is operated.

Preferably, the step of loading the browser configuration information in the single operation program block may include:

parsing the browser configuration information in the one-time operation block;

Get the window size and position information of the browser when the operation in each operation record is in progress, as well as the setting information of the interface toolbar, status bar and favorite bar.

Preferably, the step of performing a corresponding operation at the browser target position according to the operation message during keyboard operation and/or the operation message during mouse operation may include:

By calling the message playback function of PlayProc, the operation message during the keyboard operation and/or the operation message during the mouse operation is parsed;

Obtaining the input operation and switching operation of the keyboard, and/or the movement event and click event of the mouse in order;

According to the window size and position information of the browser when the operation in the obtained operation record is in progress, and the setting information of the interface toolbar, status bar and favorite bar, the input operation and switching operation of the keyboard are correspondingly executed in sequence, and/or Or mouse move events and click events.

Preferably, the step of performing a corresponding operation at the browser target position according to the operation message during the mouse operation may include:

When the mouse moves to the target position, obtain the color value at the target position, compare it with the click color parameter clickcolor saved in the operation program block, and if they are equal, execute the corresponding mouse operation record at the target position.

The invention also discloses a device for recording user operations in a browser, which may specifically include:

a recording initiating unit configured to initiate recording of user operations;

a configuration information saving unit configured to save the configuration information of the current browser;

An operation record generation module configured to collect operation messages during keyboard operations and/or operation messages during mouse operations to generate operation records;

The operation program block storage module is configured to convert the configuration information and operation records of the browser into single operation program blocks for storage.

Preferably, the configuration information storage unit may include:

Spatial information saving sub-module, which is configured to save the window size and position information of the current browser;

The function information saving sub-module is configured to save the setting information of the interface toolbar, status bar and favorite bar.

Preferably, the operation record generating module may include:

The keyboard operation collection sub-module is configured to collect the entry operation and switching operation of the keyboard;

A mouse event collection sub-module configured to collect mouse movement events and click events;

Operation and event saving submodule, which is configured to obtain the input operation and switching operation of the keyboard, and/or the lParam parameter saved in the HookProc function during the movement event and click event of the mouse;

The operation record generation sub-module is configured to generate an operation record corresponding to the keyboard and/or mouse operation according to the lParam parameter information.

Preferably, the operation program block preservation module may include:

Setting information saving sub-module, which is configured to save the corresponding browser window size and position information and the setting information of the interface tool bar, status bar and favorite bar when carrying out each said recording operation;

The program block saving sub-module is configured to save the configuration information and operation records of the browser in the same operation program block as a single operation program block.

Preferably, the device may further include:

An operation record analysis submodule configured to analyze the lParam parameter information in the generated operation record;

An invalid operation deletion sub-module configured to delete an operation in which the time interval of the mouse movement event is lower than a preset invalid time threshold;

A time interval modifying submodule configured to modify the time interval before the keyboard operation to a preset time;

The input operation and switching operation of the keyboard, and/or the movement event and click event of the mouse are recorded in sequence to generate an operation record.

The present invention also discloses a device for replaying the login operation in the browser based on the recorded operation program block in claim 18, which may specifically include:

a configuration information loading module configured to load the browser configuration information in the single operation program block;

An operation record reading module, which is configured to read the operation record and restore it to the operation message when the keyboard is operated, and/or the operation message when the mouse is operated;

The corresponding operation executing module is configured to execute corresponding operations at the target position of the browser according to the operation message when the keyboard is operated and/or the operation message when the mouse is operated.

Preferably, said device may also include:

Playback operation judging module, it is configured to pass the DocumentComplete function of WebBrowser, judges whether described operation playback is completed normally;

A program end module configured to end the operation recording and playback program;

A rerecording prompt module configured to prompt the user for rerecording.

Preferably, the playback operation judging module may include:

The environment setting check submodule is configured to check whether the current system environment setting has been loaded;

The operation message checking submodule is configured to check whether the operation messages in the operation record have been returned.

Compared with the prior art, the present invention has the following advantages:

The invention generates a message recording subroutine and a message replaying subroutine by configuring the operation recording and replaying program. The operation message of keyboard and mouse operation is collected by the message recording subroutine, and an operation record is generated; a corresponding group of operation records is packaged and converted into a single operation program block. Click the single operation program block, the message playback subroutine starts, and the operation message in the operation record is returned to the area pointed to by lParam; the system executes the corresponding keyboard and mouse operation of the operation message, and completes the corresponding single operation program block Playback of keyboard and mouse operations.

And, by setting the HookProc function as the message recording function of the recording operation, generate the message recording subroutine; Set the storage location corresponding to lParam of the HookProc function, generate the recording operation storage location; Set PlayProc as the message playback function of the recording operation, generate the message playback subroutine program. This facilitates the integration of action recording and playback programs in the browser.

Description of drawings

Fig. 1 is a flow chart of an embodiment of a method for recording user operations in a browser according to the present invention;

Fig. 2 is a sub-step flowchart of step 101 in a method embodiment of recording user operations in a browser according to the present invention;

Fig. 3 is a sub-step flowchart of step 102 in a method embodiment of recording user operations in a browser according to the present invention;

Fig. 4 is a sub-step flowchart of step 103 in a method embodiment of recording user operations in a browser according to the present invention;

Fig. 5 is a sub-step flowchart of step 104 in a method embodiment of recording user operations in a browser according to the present invention;

Fig. 6 is a flowchart of a method embodiment of replaying the login operation in the browser based on the recorded operation program block of the present invention;

Fig. 7 is a sub-step flow chart of step 201 in the embodiment of the method for replaying the login operation in the browser based on the recorded operation program block of the present invention;

Fig. 8 is a sub-step flow chart of step 202 in the method embodiment of replaying the login operation in the browser based on the recorded operation program block of the present invention;

Fig. 9 is a flow chart of the sub-steps of step 203 in a method embodiment of replaying the login operation in the browser based on the recorded operation program block of the present invention;

Fig. 10 is a flow chart of the sub-steps of step 204 in a method embodiment of replaying the login operation in the browser based on the recorded operation program block of the present invention;

FIG. 11 is a structural block diagram of an embodiment of a device for recording user operations in a browser according to the present invention;

FIG. 12 is a structural block diagram of submodules of the recording generation module 303 in an embodiment of a device for recording user operations in a browser according to the present invention;

Fig. 13 is a block diagram of the submodule structure of the operating program block saving module 304 in an embodiment of a device for recording user operations in a browser of the present invention;

Fig. 14 is a structural block diagram of an embodiment of an apparatus for replaying a login operation in a browser based on a recorded operation program block according to the present invention.

Detailed ways

In order to make the above objects, features and advantages of the present invention more comprehensible, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

One of the core concepts of the embodiments of the present invention is to generate a message recording subroutine and a message playback subroutine by configuring the operation recording and playback program. The operation message of keyboard and mouse operation is collected by the message recording subroutine, and an operation record is generated; a corresponding group of operation records is packaged and converted into a single operation program block. Click the single operation program block, the message playback subroutine starts, and the operation message in the operation record is returned to the area pointed to by lParam; the system executes the corresponding keyboard and mouse operation of the operation message, and completes the corresponding single operation program block Playback of keyboard and mouse operations.

And, by setting HookProc function as the message recording function of recording operation, generate message recording subroutine; Set the storage location corresponding to the lParam of HookProc function, generate recording operation storage location; Set PlayProc as the message playback function of recording operation, generate message playback subroutine program. This facilitates the integration of action recording and playback programs in the browser.

Referring to FIG. 1 , it shows a flow chart of an embodiment of a method for recording user operations in a browser, which may specifically include the following steps:

Step 101, start recording user operations;

In a preferred embodiment of the present invention, the step 101 may specifically include the following sub-steps:

Sub-step S01, setting a shortcut or shortcut key for a preset recording operation;

Sub-step S02, click to activate the shortcut or shortcut key.

Sub-step S03, click the preset shortcut or shortcut key to end the recording operation;

Sub-step S04, delete the operation record of clicking the preset shortcut or shortcut key to end the recording in the generated operation program block.

Step 102, saving the configuration information of the current browser;

In a preferred embodiment of the present invention, the step 102 may specifically include the following sub-steps:

Sub-step S11, saving the window size and position information of the current browser;

Sub-step S12, saving the setting information of the interface tool bar, status bar and favorite bar.

In the specific implementation, the window size and position of the browser, as well as the setting information of the toolbar, status bar and favorite bar in the browser interface can affect the accuracy of the coordinate position of the mouse during recording and playback, so when recording and playback It is necessary to ensure the consistency of the size and position of the browser window, as well as the setting information of the toolbar, status bar and favorite bar in the browser interface.

In a specific implementation, saving the current system configuration information may include saving the configuration information of the Windows system and saving the configuration information of the current browser.

For example: 360 Extreme Browser as an example, the Preferences file records information such as window size and position, whether to activate the favorite bar, status bar, etc.

Step 103, collect operation messages during keyboard operations and/or operation messages during mouse operations, and generate operation records;

In a preferred embodiment of the present invention, the step 103 may specifically include the following sub-steps:

Sub-step S21, collecting the input operation and switching operation of the keyboard;

In a preferred embodiment of the present invention, the sub-step S21 may specifically include the following units:

Start the text editing tool and enter the login information into the text editing tool;

The entered login information in text format is encrypted and written into the lParam.

In the specific implementation, the recording function can be used to record the operation message, such as:

SetWindowsHookEx(idHook: WH_JOURNALRECORD; lpfn: HookProc; hmod: HInstance; dwThreadId: 0) where WH_JOURNALRECORD is the calling method of "recording", HookProc is the message function of recording operation, whenever a mouse and keyboard message occurs, the system will call this function , the message information is stored in lParam of the HookProc function, and the message information in lParam can generate the current operation record.

Sub-step S22, and/or collecting mouse movement events and click events;

In a preferred embodiment of the present invention, the sub-step S22 may specifically include the following units:

Monitor the mouse movement event and click event, and when the movement event is triggered, obtain the current mouse coordinates and color;

When a mouse click event occurs, save the coordinates and color saved in the previous movement event to the click color parameter clickcolor.

In a specific implementation, the operation of saving the mouse color judgment point can be:

Monitor mouse movement events and click events (use the WH_MOUSE method of SetWindowsHookEx to hook the global hook), and when the movement event is triggered, obtain the current mouse coordinates and color. When a mouse click event occurs, save the coordinates and colors saved in the previous movement event to clickcolor.

Sub-step S23, obtaining the input operation and switching operation of the keyboard, and/or the lParam parameter stored in the HookProc function during the movement event and click event of the mouse;

Sub-step S24, generating an operation record corresponding to the keyboard and/or mouse operation according to the lParam parameter information.

In the specific implementation, Windows provides the API function SetwindowsHookEx to create a Hook. Through this function, a program can be added to the Hook chain to monitor Windows messages. The function syntax is: SetWindowsHookEx(idHook: Integer; lpfn: TFNHookProc; hmod: HINST; dwThreadId: DWORD) Among them, the parameter idHook specifies the monitoring function type established. WH_JOURNALRECORD and WH_JOURNALPLAYBACK are configured to monitor and playback keyboard and mouse operations.

In specific implementation, in the Windows system, the document (Document) object is a part of the Windows object, which can be accessed through the window.document property. Every HTML (Hypertext Markup Language, Hypertext Markup Language) document loaded into the browser becomes a Document object. Use the Document object to access all elements in the HTML page from the script, such as: click (), find the element with the specified id getElementById () and other user operations. It is configured to integrate the action recording and playback program in the browser.

In the specific implementation, when you need to enter the bank account and password, press the [*] key to open the notepad program, enter the user name and password into the notepad, and use the code to analyze the text information to interpret it as a code language and integrate it into the recorded function lParam. Close notepad and continue online banking operation. In this way, operations such as online banking involving passwords and authentication can be generated into operation records.

In a preferred embodiment of the present invention, the step 103 may further include the following sub-steps:

Sub-step S25, analyzing the lParam parameter information in the generated operation record;

Sub-step S26, deleting the operation that the time interval of the mouse movement event is lower than the preset invalid time threshold;

Sub-step S27, modifying the time interval before the keyboard operation to a preset time;

Sub-step S28, record the input operation and switch operation of the keyboard, and/or the movement event and click event of the mouse in order to generate an operation record.

In a preferred embodiment of the present invention, the preset invalid time threshold may be 20 milliseconds, and the preset time may be 5 milliseconds.

In a specific implementation, the operation record is analyzed before packaging. When the time interval of the mouse movement event is within 20ms (milliseconds), the waiting time and a mouse movement event before it may be deleted. The purpose is to remove invalid mouse movement operations, allowing playback to perform faster. When there is a time interval before keyboard operation, the waiting time can be modified to 5ms; the waiting time interval can be set according to customer needs, but in order to avoid downtime and stuck situations, the minimum setting can be controlled to 5ms.

Step 104: Convert the configuration information and operation records of the browser into single operation program blocks for storage.

In a preferred embodiment of the present invention, the step 104 may specifically include the following sub-steps:

Sub-step S31, saving the corresponding browser window size and position information and the setting information of the interface tool bar, status bar and favorite bar when performing each recording operation;

Sub-step S32, saving the configuration information of the browser and the operation record in the same operation program block as a single operation program block.

In a specific implementation, after the recording ends, the last operation of clicking the icon may be deleted. The operation recording and playback program integrated in the browser can be implemented in the form of a plug-in, which will display an icon on the browser. Both the start and stop of the message recording subroutine need to be completed by clicking the icon, so the click operation to end the recording is not a valid operation of the user, so it is deleted.

In a specific implementation, a corresponding operation program block may be generated by recording a completed operation. For the convenience of customers, the operation program block can be created as a shortcut on the desktop or in a folder, and the user can click on the shortcut to play back the operation records in the operation program block.

Referring to FIG. 6 , it shows a flow chart of an embodiment of a method for replaying a login operation in a browser based on a recorded operation program block, which may specifically include the following steps:

Step 201, loading the browser configuration information in the single operation program block;

In a preferred embodiment of the present invention, the step 201 may specifically include the following sub-steps:

Sub-step S41, analyzing the browser configuration information in the single operation program block;

Sub-step S42, obtaining the window size and position information of the browser when the operation in each operation record is in progress, and the setting information of the interface tool bar, status bar and favorite bar.

Step 202, read the operation record, restore it to the operation message when the keyboard is operated, and/or the operation message when the mouse is operated;

In a preferred embodiment of the present invention, the step 202 may specifically include the following sub-steps:

Substep S51, return the operation record to the pointed position of lParam;

Sub-step S52, analyzing the operation record at the position pointed to by the lParam, and restoring it to the operation message during keyboard operation and/or the operation message during mouse operation.

Step 203, perform a corresponding operation at the browser target position according to the operation message when the keyboard is operated and/or the operation message when the mouse is operated;

In a preferred embodiment of the present invention, the step 203 may specifically include the following sub-steps:

Sub-step S61, by calling the message playback function of PlayProc, analyzing the operation message during the keyboard operation and/or the operation message during the mouse operation;

Sub-step S62, acquiring the input operation and switching operation of the keyboard, and/or the movement event and click event of the mouse in order;

Sub-step S63, according to the window size and position information of the browser when the operation in the obtained operation record is in progress, and the setting information of the interface toolbar, status bar and favorite bar, correspondingly execute the input operation and switching of the keyboard in order Actions, and/or mouse move events and click events;

Sub-step S64, when the mouse moves to the target position, obtain the color value at the target position, compare it with the click color parameter clickcolor saved in the operation program block, if they are equal, execute the corresponding mouse operation record at the target position.

In a specific implementation, the system environment setting may include configuration information of various elements of the recording process. Wherein, for the method of judging whether the element has been displayed, whether the operation message is the same as the configuration information at the corresponding position where the operation message is generated during the recording process can be determined.

For example: when the mouse moves to the target position, get the color of the mouse and compare it with the previously inserted color clickcolor. If they are equal, the desired action element is clickable. If the mouse is not in the area of the browser, no need to judge.

Take a browser as an example to determine whether the mouse is inside the browser page:

Enumerate all windows and save the handle of the class named Internet Explorer_Server to MainHwnd.

Then use the GetWindowRect function to get the coordinates of the page.

Step 204, through the DocumentComplete function of the WebBrowser, judge whether the operation playback is completed normally; if yes, end the operation recording and playback procedure; if not, prompt the user to re-record.

In a specific implementation, it may be judged whether the operation message is executed smoothly according to whether the page executed by the operation message is loaded. For example: use WebBrowser's DocumentComplete to determine whether the page is loaded.

In a preferred embodiment of the present invention, the step 204 may specifically include the following sub-steps:

Sub-step S71, check whether the current browser configuration information has been loaded;

Sub-step S72, check whether the operation message in the operation record has been returned.

In a specific implementation, the playback operation record can open the operation recording and playback program and select the playback operation; it can also click the corresponding single operation block to directly start the message playback subroutine.

The method of playback mouse button operation can be:

SetWindowsHookEx(idHook: WH_JOURNALPLAYBACK; lpfn: PlayProc; hmod: HInstance; dwThreadId: 0)

Among them, WH_JOURNALPLAYBACK is the calling method of "playback", PlayProc is the message playback function, when the system can perform operation message playback, call this function, the program will return the previously recorded operation message value to the area pointed to by lParam, and the system will execute this function Manipulate the message, thus enabling playback.

In order to facilitate those skilled in the art to better understand the present invention, the present invention is further described by a complete example below:

1. In a browser, use the API function SetwindowsHookEx to establish a message recording function for recording operations, and generate a message recording subroutine. And the storage location corresponding to lParam of the function is used as the storage location of the record operation. Use PlayProc to establish the message playback function of the recording operation, and generate the message playback subroutine.

2. Open a browser, start the operation recording and playback program, and prepare to record the operation of handing in water and electricity. First save the configuration information of the current browser.

3. You can use the WH_MOUSE method of SetWindowsHookEx to hang a global hook to monitor mouse movement events and click events. When the movement event is triggered, obtain the current mouse coordinates and color. When a mouse click event occurs, save the coordinates and colors saved in the previous movement event to clickcolor. At the same time, the HookProc message function can be used to obtain the operation messages of the mouse click operation and keyboard movement operation and save them in the lParam of the HookProc function to generate the current operation record.

4. When it is necessary to enter the online banking page and enter the bank account number and password, open the notepad program, enter the user name and password into the notepad, and interpret the text information into a code language by means of code analysis, and integrate it into the recorded function lParam. Generate operation records for online banking operations involving passwords and authentication.

5. After all the operations of paying water and electricity are completed, sort out the operation records recorded in lParam of the function, delete the last operation record of clicking the icon, convert other operation records into operation message values, and package them in sequence as the water and electricity fees submitted by online banking Operation block. Create a corresponding shortcut on the desktop.

6. Click the corresponding shortcut generated on the desktop to open the operation program block submitted by online banking for utility bills. Start the message playback subroutine, which can be loaded; you can use PlayProc as a message playback function to return the operation message value recorded in the previous operation to the area pointed to by lParam, and the system will execute the corresponding operation message. The corresponding browser configuration information and the corresponding page are loaded according to the corresponding operation message.

7. If it is found that the corresponding browser configuration information and the corresponding page cannot be loaded, a prompt message will be sent to the user; the user will be informed that some pages, browser configuration and other elements have changed in the operation submitted by the online banking for utility bills, and the corresponding complete page needs to be re-recorded operate.

It should be noted that, for the method embodiment, for the sake of simple description, it is expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action order, because according to this According to the invention, certain steps may be performed in other order or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification belong to preferred embodiments, and the actions involved are not necessarily required by the present invention.

Referring to FIG. 11 , it shows a structural block diagram of an embodiment of an apparatus for recording user operations in a browser, which may specifically include:

A recording start unit 301 configured to start recording user operations;

A configuration information saving unit 302 configured to save the configuration information of the current browser;

In a preferred embodiment of the present invention, the configuration information storage unit 302 may specifically include the following submodules:

Spatial information saving sub-module, which is configured to save the window size and position information of the current browser;

The function information saving sub-module is configured to save the setting information of the interface toolbar, status bar and favorite bar.

An operation record generation module 303 configured to collect operation messages during keyboard operations and/or operation messages during mouse operations to generate operation records;

In a preferred embodiment of the present invention, the operation record generating module 303 may specifically include the following submodules:

The keyboard operation collection sub-module S81 is configured to collect keyboard entry operations and switching operations;

Mouse event collection sub-module S82, which is configured to collect mouse movement events and click events;

Operation and event saving sub-module S83, which is configured to obtain the input operation and switching operation of the keyboard, and/or the lParam parameter stored in the HookProc function during the movement event and click event of the mouse;

Operation record generation sub-module S84, which is configured to generate an operation record corresponding to the keyboard and/or mouse operation according to the lParam parameter information;

An operation record analysis submodule S85 configured to analyze the lParam parameter information in the generated operation record;

Invalid operation deletion sub-module S86, which is configured to delete the operation in which the time interval of the mouse movement event is lower than the preset invalid time threshold;

The time interval modification submodule S87 is configured to modify the time interval before the keyboard operation to a preset time;

The operation record conversion sub-module S88 is configured to sequentially record the input operation and switch operation of the keyboard, and/or the movement event and click event of the mouse to generate an operation record.

The operation program block saving module 304 is configured to convert the configuration information and operation records of the browser into a single operation program block for storage.

In a preferred embodiment of the present invention, the operation program block saving module 304 may specifically include the following submodules:

Setting information saving sub-module S91, which is configured to save the corresponding browser window size and position information and the setting information of the interface tool bar, status bar and favorite bar when performing each recording operation;

The program block saving sub-module S92 is configured to save the browser configuration information and operation records in the same operation program block as a single operation program block.

Referring to FIG. 14 , it shows a structural block diagram of an embodiment of an apparatus for replaying a login operation in a browser based on a recorded operation program block, which may specifically include:

A configuration information loading module 401 configured to load the browser configuration information in the single operation program block;

An operation record reading module 402, which is configured to read the operation record and restore it to an operation message during a keyboard operation, and/or an operation message during a mouse operation;

The corresponding operation execution module 403 is configured to perform corresponding operations at the browser target position according to the operation message during the keyboard operation and/or the operation message during the mouse operation;

Playback operation judging module 404, it is configured to pass the DocumentComplete function of WebBrowser, judges whether described operation playback is completed normally;

A program end module 405, which is configured to end the operation recording and playback program;

A re-recording prompt module 406 configured to prompt the user to re-record.

Since the device embodiment basically corresponds to the foregoing method embodiment, for details not detailed in the description of this embodiment, reference may be made to the relevant description in the foregoing embodiment, and details are not repeated here.

Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Accordingly, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and combinations of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a Means for realizing the functions specified in one or more steps of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart flow or flows and/or block diagram block or blocks.

While preferred embodiments of the present invention have been described, additional changes and modifications can be made to these embodiments by those skilled in the art once the basic inventive concept is appreciated. Therefore, it is intended that the appended claims be construed to cover the preferred embodiment as well as all changes and modifications which fall within the scope of the invention.

The methods, devices and servers in the above embodiments of the present invention are applicable to various network or client environments, for example, they can be implemented in computer equipment such as personal computer equipment, or can be implemented in computer equipment such as mobile phones, mobile communication equipment, In other portable electronic devices such as personal digital assistants (PDAs) or non-portable electronic devices. Therefore, it should be clear to those skilled in the art that the scope of protection of the present invention is not limited to the operation recording function and operation playback function in the browser running on the PC, and is only used in the embodiments of the present invention for simplicity and convenience of description. The operation recording function and operation playback function in the browser running on the PC are described.

The above is a method for recording user operations in the browser provided by the present invention, a method for replaying login operations in the browser based on recorded operating program blocks; and, a method for recording user operations in the browser A device for recording by operation, a device for replaying the login operation in the browser based on the recorded operation program block is introduced in detail. In this paper, specific examples are used to illustrate the principle and implementation of the present invention. The above implementation The description of the example is only used to help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation and scope of application. In summary As stated above, the content of this specification should not be construed as limiting the present invention.

Claims (25)

1. A method for recording user operations in a browser, comprising: Start the recording of user actions; Save the configuration information of the current browser; Collect operation messages during keyboard operations and/or operation messages during mouse operations to generate operation records; The configuration information and operation records of the browser are converted into single operation program blocks for storage. 2. The method according to claim 1, wherein the step of saving the configuration information of the current browser comprises: Save the window size and position information of the current browser; Save the setting information of interface toolbar, status bar and favorite bar. 3. The method according to claim 1, wherein the operation message during said collection of keyboard operation, and/or the operation message during mouse operation, and the step of generating an operation record comprises: Collect the input operation and switching operation of the keyboard; And/or collect mouse movement events and click events; Save the lParam parameter of the HookProc function when obtaining the input operation and switching operation of the keyboard, and/or the movement event and click event of the mouse; An operation record corresponding to the keyboard and/or mouse operation is generated according to the lParam parameter information. 4. method as claimed in claim 3, is characterized in that, the step of input operation and switching operation of described acquisition keyboard further comprises: Start the text editing tool and enter the login information into the text editing tool; The entered login information in text format is encrypted and written into the lParam. 5. The method according to claim 3, wherein the moving event and the click event of the collection mouse comprise: Monitor the mouse movement event and click event, and when the movement event is triggered, obtain the current mouse coordinates and color; When a mouse click event occurs, save the coordinates and color saved in the previous movement event to the click color parameter clickcolor. 6. The method according to claim 2, wherein the step of converting the configuration information and operation records of the browser into single operation program blocks for saving comprises: Save the corresponding browser window size and position information and the setting information of the interface toolbar, status bar and favorite bar when performing each recording operation; The configuration information and operation records of the browser are stored in the same operation program block as a single operation program block. 7. The method of claim 3, further comprising: Analyze the lParam parameter information in the generated operation record; deleting an operation in which the time interval of the mouse movement event is lower than a preset invalid time threshold; Modifying the time interval before the keyboard operation to a preset time; The input operation and switching operation of the keyboard, and/or the movement event and click event of the mouse are recorded in sequence to generate an operation record. 8. The method according to claim 7, wherein the preset invalid time threshold is 20 milliseconds, and the preset time is 5 milliseconds. 9. The method according to claim 1, wherein the step of starting recording user operations comprises: Set shortcuts or shortcut keys for preset recording operations; Click to activate the shortcut or shortcut key. 10. The method of claim 9, further comprising: Click the preset shortcut or shortcut key to end the recording operation; In the generated operation program block, delete the operation record of clicking the preset shortcut or the shortcut key to end the recording. 11. A method for replaying the login operation in the browser based on the recorded operation program block according to claim 1, comprising: Loading the browser configuration information in the single operation program block; Read the operation record, restore it to the operation message when the keyboard is operated, and/or the operation message when the mouse is operated; According to the operation message when the keyboard is operated, and/or the operation message when the mouse is operated, a corresponding operation is performed at the target position of the browser. 12. The method of claim 11, further comprising: Through the DocumentComplete function of WebBrowser, it is judged whether the operation playback is completed normally; If so, then end the operation recording and playback program; If not, the user is prompted to re-record. 13. The method according to claim 12, wherein the step of judging whether the operation playback is completed normally further comprises: Check whether the current browser configuration information has been loaded; Check whether the operation message in the operation record has been returned. 14. The method according to claim 11, wherein the step of reading the operation record and restoring it to the operation message when the keyboard is operated, and/or the operation message when the mouse is operated comprises: Return the operation record to the location pointed to by lParam; The operation record at the position pointed to by the lParam is parsed, and restored to the operation message when the keyboard is operated, and/or the operation message when the mouse is operated. 15. The method according to claim 11, wherein the step of loading the browser configuration information in the single operation program block comprises: parsing the browser configuration information in the one-time operation block; Get the window size and position information of the browser when the operation in each operation record is in progress, as well as the setting information of the interface toolbar, status bar and favorite bar. 16. The method according to claim 15, wherein the step of performing a corresponding operation at the browser target position according to the operation message during keyboard operation and/or the operation message during mouse operation comprises: By calling the message playback function of PlayProc, the operation message during the keyboard operation and/or the operation message during the mouse operation is parsed; Obtaining the input operation and switching operation of the keyboard, and/or the movement event and click event of the mouse in order; According to the window size and position information of the browser when the operation in the obtained operation record is in progress, and the setting information of the interface toolbar, status bar and favorite bar, the input operation and switching operation of the keyboard are correspondingly executed in sequence, and/or Or mouse move events and click events. 17. The method according to claim 11, wherein the step of performing a corresponding operation at the browser target position according to the operation message during the mouse operation comprises: When the mouse moves to the target position, obtain the color value at the target position, compare it with the click color parameter clickcolor saved in the operation program block, and if they are equal, execute the corresponding mouse operation record at the target position. 18. A device for recording user operations in a browser, comprising: a recording initiating unit configured to initiate recording of user operations; a configuration information saving unit configured to save the configuration information of the current browser; An operation record generation module configured to collect operation messages during keyboard operations and/or operation messages during mouse operations to generate operation records; The operation program block storage module is configured to convert the configuration information and operation records of the browser into single operation program blocks for storage. 19. The device according to claim 18, wherein the configuration information storage unit comprises: Spatial information saving sub-module, which is configured to save the window size and position information of the current browser; The function information saving sub-module is configured to save the setting information of the interface toolbar, status bar and favorite bar. 20. The device according to claim 18, wherein the operation record generating module comprises: The keyboard operation collection sub-module is configured to collect the entry operation and switching operation of the keyboard; A mouse event collection sub-module configured to collect mouse movement events and click events; Operation and event saving submodule, which is configured to obtain the input operation and switching operation of the keyboard, and/or the lParam parameter saved in the HookProc function during the movement event and click event of the mouse; The operation record generation sub-module is configured to generate an operation record corresponding to the keyboard and/or mouse operation according to the lParam parameter information. 21. The device according to claim 19, wherein the operating program block preservation module comprises: Setting information saving sub-module, which is configured to save the corresponding browser window size and position information and the setting information of the interface tool bar, status bar and favorite bar when carrying out each said recording operation; The program block saving sub-module is configured to save the configuration information and operation records of the browser in the same operation program block as a single operation program block. 22. The apparatus of claim 20, further comprising: An operation record analysis submodule configured to analyze the lParam parameter information in the generated operation record; An invalid operation deletion sub-module configured to delete an operation in which the time interval of the mouse movement event is lower than a preset invalid time threshold; A time interval modifying submodule configured to modify the time interval before the keyboard operation to a preset time; The operation record conversion sub-module is configured to sequentially record the input operation and switching operation of the keyboard, and/or the movement event and click event of the mouse to generate an operation record. 23. A device for replaying the login operation in the browser based on the recorded operation program block according to claim 18, characterized in that it comprises: a configuration information loading module configured to load the browser configuration information in the single operation program block; An operation record reading module, which is configured to read the operation record and restore it to the operation message when the keyboard is operated, and/or the operation message when the mouse is operated; The corresponding operation executing module is configured to execute corresponding operations at the target position of the browser according to the operation message when the keyboard is operated and/or the operation message when the mouse is operated. 24. The apparatus of claim 23, further comprising: Playback operation judging module, it is configured to pass the DocumentComplete function of WebBrowser, judges whether described operation playback is completed normally; A program end module configured to end the operation recording and playback program; A rerecording prompt module configured to prompt the user for rerecording. 25. The device according to claim 24, wherein the playback operation judging module comprises: The environment setting check submodule is configured to check whether the current system environment setting has been loaded; The operation message checking submodule is configured to check whether the operation messages in the operation record have been returned.

Images