CN114971539A - Simulated manual operation method based on image matching - Google Patents

Abstract

The invention discloses a simulated manual operation method based on image matching, which comprises the following steps: step 1: selecting a positioning image of an operation interface; the positioning image is a screenshot of the component; the characteristic elements of the positioning image are unique; step 2: providing a sequence of operational events; the operation event sequence comprises operation behaviors on the positioning image; and step 3: repeating the step 1 and the step 2 to form a set of task scripts; and 4, step 4: and running the task script to automatically finish the office action. The invention provides a simulated manual operation method based on image matching. Since the operation area of each step is positioned by image matching, the positional area variation of the operation interface cannot affect the positioning.

Description

A Method of Simulating Manual Operation Based on Image Matching

technical field

The invention relates to the technical field of office automation, in particular, to a method for simulating manual operation based on image matching.

Background technique

With the development of modern information technology, office automation technology, which is a combination of office and computer technology, is a comprehensive technology that is emerging. It can not only realize the automatic processing of office affairs, but also greatly improve the efficiency of individual or team office affairs processing. At present, whether it is a government, an enterprise or a self-employed person, there may be a lot of process and repetitive document work. Manual operation is not only boring, but also prone to errors. Automatic operation software can save people from tedious repetitive labor by simulating the behavior of manual operation and replacing manual processing of such work by computers. At the same time, machine operation can greatly reduce the error rate of manual operation.

At present, a mainstream technical idea of simulating manual operation is to record the relevant events (mouse click events, keyboard input events, etc.) generated by the operating system during manual operation through codes, and then send messages that have been processed to the operating system in sequence during the process of simulating manual operations. recorded events. This technique generally records the corresponding operation area when the event occurs, and changes in the operation area may cause the behavior of the machine simulation to fail. Another problem is that for some scenarios with time requirements for the previous and subsequent steps, when performing the demonstration operation manually, it is necessary to carefully consider the time interval and other factors to avoid the phenomenon that the object to be operated has not yet appeared but the simulated event has occurred, which will eventually lead to the failure of the task. .

SUMMARY OF THE INVENTION

In order to solve the problem that the computer simulation manual operation process is affected by the change of the operation area and the time difference between the preceding and following steps, the embodiment of the present invention provides a method for simulating manual operation based on image matching. The technical solution is as follows:

In one aspect, a method for simulating manual operation based on image matching is provided, including:

Step 1: select the positioning image of the operation interface; the positioning image is a screenshot of the component; the feature element of the positioning image is unique;

Step 2: providing an operation event sequence; the operation event sequence includes an operation behavior on the positioning image;

Step 3: Repeat steps 1 and 2 to form a set of task scripts;

Step 4: Run the task script to automate the office behavior.

Further, in step 1, the specific steps of selecting the positioning image of the operation interface include:

Get the image of the operation interface form;

Then, use an image comparison algorithm to calculate whether there are multiple points in the window image that meet the characteristics of the positioning image; if there are multiple points, the feature elements of the positioning image cannot be uniquely positioned and need to be re-selected Position the element.

Further, in step 2, the specific steps of providing the operation event sequence include:

Action events are selected before input is provided.

Further, in step 3, the specific steps of forming a set of task scripts include:

For the operation steps of decomposing the preset task, step 1 and step 2 are performed in sequence according to the decomposition sequence, and the task script is saved after all processing is completed. .

The beneficial effects brought by the technical solutions provided in the embodiments of the present invention are:

The invention provides a method for simulating manual operation based on image matching, which determines each operation step through image matching, and then determines how to simulate manual operation for each step according to an instruction file. Since the operation area of each step is positioned according to the image matching, the position area change of the operation interface cannot affect the positioning. In addition, if the software does not detect the positioning image of a certain step, it means that the current step has not yet started, and will enter the waiting mode to wait for the next step to start. .

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 is a schematic diagram of a method for simulating manual operation based on image matching according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

The present embodiment provides a method for simulating manual operation based on image matching, including:

Step 1: Select the positioning image A; the positioning image A is usually a screenshot of a button, text box and other components, and it is necessary to ensure that the image features of A are unique and obvious as much as possible.

Specifically, the positioning inspection needs to first obtain the entire operation interface window image called S, and the positioning element is called A, and then calculate whether there are multiple points in S that meet the characteristics of A through the image comparison algorithm. If there are multiple points, it means that the A feature element cannot be uniquely positioned, and the positioning element needs to be reselected.

Step 2: Provide an operation event sequence B; the content included in the event sequence B is the operation behavior on the positioning image A, such as button click and text box input.

Specifically, select the operation event first, and then provide the input content; for example, the keyboard input event may need to provide the text to be input.

Step 3: A set of task scripts is formed by repeating steps 1 and 2, and the content of the task script is a series of operations to complete the preset task;

Specifically, for the operation steps of decomposing a certain task, step 1 and step 2 are performed in sequence according to the decomposing order, and the task script file is saved after all processing is completed.

In this embodiment, taking accessing a web page as an example, accessing a web page requires:

1) Click on the browser;

2) Enter the URL in the address bar;

3) Click the Access button.

The corresponding script is:

1) Select the browser icon on the desktop as the positioning image A1, and the mouse click event as the content of the event sequence B1, and this step is called T1 as a part of the task script.

2) Select the address bar image as the positioning image A2, the text input event and the website address as the content of the event sequence B2, and this step is called T2 as a part of the task script.

3) Select the access button image as the positioning image A3, and the mouse click event as the content of the event sequence B3. This step is called T3 as a part of the task script.

T1, T2, and T3 form a task script T for accessing the webpage in sequence.

Step 4: When the task script T is executed, the desktop restores the initial image P0, and executes according to the operation sequence in the task script T:

1) Execute the task script T1, take out the positioning image A1 in the task script T1, match and search the positioning image A1 in the image P0, when finding the area where the image features meet the characteristics of the positioning image A1, determine the position coordinates of this area in the operating system ( x1, y1), simulate a mouse move event to move the mouse to this position (x1, y1). The event sequence B1 is entered at this location through the interface provided by the operating system. After all is completed, the execution of the task script T1 is completed.

2) After the task script T1 is executed, a new image P1 will be formed on the desktop. Execute the task script T2, take out the positioning image A2 in the task script T2, match and search the positioning image A2 in the image P1, and input the event sequence B2 after finding the positioning image A2. After all is completed, T2 execution is completed.

3) After the task script T2 is executed, the desktop forms a new image P2. Execute the task script T3, take out the positioning image A3 in the task script T3, match and search for the positioning image A3 in the image P2, find the positioning image A3 and input the event sequence B3. After all is completed, the task script T3 is executed.

After the task scripts T1, T2, and T3 are all executed, the task of the task script T is completed, that is, the task of accessing a certain webpage is completed.

For specific operations, see Figure 1:

1. The user opens the software, creates a new task file, determines the current office task, and decomposes each step of the task.

2. Go to the single-step operation interface, use the software to select the positioning image, and the software will check the positioning reliability, so that the software can determine a certain step when the task file is executed. After the inspection is passed, the positioning image will be temporarily stored in the memory; if the inspection fails, the user will be prompted to select again.

3. Prompt the user to select an operation event, and store the operation event and content provided by the user in the memory.

4. Prompt the user whether to continue writing the script operation. If yes, go to step 2;

5. The user uses the software to run the task file, and the software automatically completes the office task pointed to by the task file according to the positioning image of each step saved in the task file and the event content.

Since the operation area of each step is positioned according to the image matching, the position area change of the operation interface cannot affect the positioning. In addition, if the software does not detect the positioning image of a certain step, it means that the current step has not yet started, and will enter the waiting mode to wait for the next step to start. .

The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention. Inside.

Claims (4)

1. a simulation manual operation method based on image matching, is characterized in that, comprises: Step 1: select the positioning image of the operation interface; the positioning image is a screenshot of the component; the feature element of the positioning image is unique; Step 2: providing an operation event sequence; the operation event sequence includes an operation behavior on the positioning image; Step 3: Repeat steps 1 and 2 to form a set of task scripts; Step 4: Run the task script to automate the office behavior. 2. a kind of simulated manual operation method based on image matching as claimed in claim 1, is characterized in that, in step 1, the concrete step of selecting the positioning image of operation interface comprises: Get the image of the operation interface form; Then, use an image comparison algorithm to calculate whether there are multiple points in the window image that meet the characteristics of the positioning image; if there are multiple points, the feature elements of the positioning image cannot be uniquely positioned and need to be re-selected Position the element. 3. a kind of simulated manual operation method based on image matching as claimed in claim 2, is characterized in that, in step 2, the concrete step that provides operation event sequence comprises: Action events are selected before input is provided. 4. a kind of simulated manual operation method based on image matching as claimed in claim 3, is characterized in that, in step 3, the concrete steps of forming a set of task script comprise: For the operation steps of decomposing the preset task, step 1 and step 2 are performed in sequence according to the decomposition sequence, and the task script is saved after all processing is completed.

Images