CN114670194B - Positioning method and device for manipulator system - Google Patents

Abstract

The application provides a mechanical arm system positioning method and a mechanical arm system positioning device, wherein the method obtains a first mapping relation between a mechanical coordinate system and a camera coordinate system; acquiring a second mapping relation between a camera coordinate system and a device coordinate system of the electronic device; obtaining a third mapping relation between the equipment coordinate system and the mechanical coordinate system based on the first mapping relation and the second mapping relation; and acquiring the equipment coordinates of the target position of the electronic equipment in the equipment coordinate system, and converting the equipment coordinates to obtain the mechanical coordinates of the target position based on the third mapping relation. According to the scheme, the mechanical position of the target position is not required to be obtained according to the mechanical arm equipment, the development and debugging process of the automatic script can be decoupled from the mechanical arm equipment, and the development and debugging efficiency of the automatic script is improved. Meanwhile, the mechanical coordinates corresponding to the target position do not need to be obtained by a camera, so that the time consumption of the mechanical coordinate obtaining process of the target position is saved, and the testing efficiency of the whole testing process is improved.

Description

Positioning method and device for manipulator system

technical field

The present application relates to the technical field of machine vision systems, in particular to a positioning method and device for a manipulator system.

Background technique

The automatic testing and automatic assembly of equipment can be realized by using the manipulator system, which is widely used in industrial production. The machine vision system is an important part of the manipulator system. The machine vision system is equivalent to the eyes of the manipulator system. The position coordinates of the tested items on the device are obtained through the machine vision system. Further, control the manipulator to move to the position coordinates to perform corresponding operations, such as sliding, clicking, and long-pressing the screen of the electronic device, such as realizing automated testing of the target electronic device, such as stress testing and performance testing. But current robotic systems are inefficient.

Contents of the invention

In view of this, this application provides a positioning method and device for a manipulator system to solve the above-mentioned technical problems. The disclosed technical solutions are as follows:

In a first aspect, the present application provides a positioning method for a manipulator system, the method comprising: obtaining a first mapping relationship between a mechanical coordinate system and a camera coordinate system; obtaining a first mapping relationship between a camera coordinate system and a device coordinate system of an electronic device Two mapping relationships; based on the first mapping relationship and the second mapping relationship, obtain a third mapping relationship between the device coordinate system and the mechanical coordinate system; obtain the device coordinates of the target position of the electronic device in the device coordinate system, and based on the third The mapping relationship is transformed to obtain the mechanical coordinates of the target position. It can be seen that the robot system positioning method provided by this solution can directly convert the device coordinates corresponding to the target position of the electronic device into mechanical coordinates, and further control the robot device to move to the target position and perform corresponding operations based on the mechanical coordinates. This solution can obtain the equipment coordinates of the target position through tools, or directly write the equipment coordinates of the target position into the automation script, so that there is no need to write the mechanical position of the target position according to the manipulator device, so the development and debugging process of the automation script can be compared with The decoupling of manipulator equipment improves the development and debugging efficiency of automation scripts. At the same time, this solution does not need to use the camera to obtain the camera coordinates of the target position, and then convert the camera coordinates into mechanical coordinates. Therefore, the time-consuming process of obtaining the mechanical coordinates of the target position is saved, and the test efficiency of the entire test process is improved. .

In a possible implementation of the first aspect, the target position is a target control on the display interface of the electronic device; obtaining the device coordinates of the target position of the electronic device in the device coordinate system includes: obtaining a control identifier corresponding to the target control; Sending a control coordinate request for obtaining the device coordinates of the target control to the electronic device, where the control coordinate request includes the control identifier of the target control; and receiving the device coordinates corresponding to the target control sent by the electronic device. It can be seen that in this scenario, the device coordinates corresponding to any control can be obtained directly through the tools installed on the electronic device. It is not necessary to write the device coordinates of the target position in the automation script, but only need to write the corresponding identification of the control in the automation script. For different types of electronic devices with different physical sizes but the same control identification, there is no need to re-develop automation scripts. Therefore, the automation scripts developed by using this scheme can be compatible with electronic devices of different physical sizes, which improves the compatibility of automation scripts sex.

In another possible implementation manner of the first aspect, the target position is a non-control position of the display interface of the electronic device; the acquiring the device coordinates of the target position of the electronic device in the device coordinate system includes : receiving the device coordinates corresponding to the target position sent by the electronic device, and the device coordinates are obtained after the electronic device receives a click operation on the target position. It can be seen that this scene needs to obtain the device coordinates of the target position in advance and write them into the automation script.

In yet another possible implementation of the first aspect, obtaining the first mapping relationship between the mechanical coordinate system and the camera coordinate system includes: obtaining the mechanical coordinates corresponding to at least two manipulator calibration points on the manipulator device; The camera coordinates of each manipulator calibration point in the camera coordinate system; based on the machine coordinates and camera coordinates corresponding to at least two manipulator calibration points, a first mapping relationship between the machine coordinate system and the camera coordinate system is obtained. It can be seen that this scheme directly sets the calibration point on the manipulator device, so that it does not need to rely on the manipulator to obtain the mechanical coordinates of the mechanical calibration point, realizes decoupling from the manipulator device, and improves the calibration efficiency at the same time.

In another possible implementation of the first aspect, obtaining the mechanical coordinates corresponding to at least two manipulator calibration points on the manipulator device includes: at least two calibration points are set on the fixture platform of the manipulator device, and the equipment based on the manipulator device information to obtain the machine coordinates of each calibration point.

In yet another possible implementation of the first aspect, obtaining the second mapping relationship between the camera coordinate system and the device coordinate system of the electronic device includes: obtaining device coordinates corresponding to at least two device calibration points on the electronic device; The camera obtains the camera coordinates of each device calibration point in the camera coordinate system; based on the device coordinates and camera coordinates corresponding to the at least two device calibration points, obtain the coordinates between the camera coordinate system and the device coordinate system Second mapping relationship.

In yet another possible implementation of the first aspect, the device calibration point is a calibration control on the display interface of the electronic device; obtaining the device coordinates corresponding to at least two device calibration points on the electronic device includes: obtaining the coordinates corresponding to the calibration control Control identification; sending a control coordinate request to the electronic device, the control coordinate request including the control identification; receiving the device coordinate corresponding to the calibration control sent by the electronic device.

In another possible implementation of the first aspect, the device calibration point is a calibration point in a preset picture displayed by the electronic device; obtaining device coordinates corresponding to at least two device calibration points on the electronic device includes: Preset the coordinates in the image to obtain the device coordinates of the calibration point in the device coordinate system.

In yet another possible implementation manner of the first aspect, obtaining the camera coordinates of each device calibration point in the camera coordinate system through the camera includes: receiving an image of the electronic device captured by the camera, the image including the device calibration point; The technology identifies the device calibration points contained in the image, and determines the position coordinates of each device calibration point in the image, and obtains the camera coordinates of the device calibration points.

In another possible implementation of the first aspect, the target position is a non-control position of the display interface of the electronic device; obtaining the device coordinates of the target position of the electronic device in the device coordinate system includes: receiving the target position sent by the electronic device Corresponding device coordinates, the device coordinates are obtained after the electronic device receives a click operation on the target position.

In yet another possible implementation manner of the first aspect, converting to obtain the mechanical coordinates of the target position based on the third mapping relationship includes: inputting the device coordinates of the target position into the mapping relationship, and calculating the mechanical coordinates corresponding to the target position.

In yet another possible implementation manner of the first aspect, the method further includes: controlling the manipulator to move to the target position of the electronic device based on the mechanical coordinates of the target position; and controlling the manipulator to perform corresponding operations at the target position.

In another possible implementation manner of the first aspect, controlling the manipulator device to move to the target position of the electronic device based on the mechanical coordinates of the target position includes: based on the mechanical coordinates of the target position, and the The mechanical coordinates of the current position of the manipulator device to generate a motion control command; send the motion control command to the manipulator device, so that the manipulator device moves from the current position to the position in response to the motion control command the target location.

In a second aspect, the present application also provides a computer device, the electronic device includes: one or more processors, memory and touch screen; the memory is used to store program code; the processor is used to run the program code, so that the electronic device realizes the first The robot system positioning method described in any possible implementation manner of the aspect.

In the third aspect, the present application also provides a computer-readable storage medium, on which instructions are stored, and when the instructions are run on the computer equipment, the electronic equipment executes the manipulator system positioning method described in any one of the first aspect .

It should be understood that descriptions of technical features, technical solutions, beneficial effects or similar language in this application do not imply that all features and advantages can be realized in any single embodiment. On the contrary, it can be understood that the description of features or beneficial effects means that specific technical features, technical solutions or beneficial effects are included in at least one embodiment. Therefore, descriptions of technical features, technical solutions or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in this embodiment may also be combined in any appropriate manner. Those skilled in the art will understand that the embodiments can be implemented without one or more specific technical features, technical solutions or advantageous effects of the specific embodiments. In other embodiments, additional technical features and beneficial effects may also be identified in certain embodiments that do not embody all embodiments.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are For some embodiments of the present invention, those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is a schematic diagram of a manipulator clamp platform provided by an embodiment of the present application;

Fig. 2 is a schematic diagram of the corresponding image when the target electronic device is placed on the manipulator fixture platform provided by the embodiment of the present application;

Fig. 3 is a schematic structural diagram of a manipulator system provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of a device coordinate system of a target electronic device provided in an embodiment of the present application;

Fig. 5 is a schematic diagram of a machine coordinate system, a camera coordinate system and a device coordinate system provided by the embodiment of the present application;

Fig. 6 is a flow chart of a positioning method for a manipulator system provided by an embodiment of the present application;

Fig. 7 is a schematic diagram of a calibration point on a fixture platform provided by an embodiment of the present application;

Fig. 8 is a schematic diagram of camera coordinates of a calibration point on a fixture platform provided by an embodiment of the present application;

FIG. 9 is a schematic diagram of device coordinates of a calibration point of a target electronic device provided in an embodiment of the present application;

FIG. 10 is a schematic diagram of camera coordinates of a calibration point of a target electronic device provided in an embodiment of the present application;

FIG. 11 is a schematic diagram of a fixture platform provided in an embodiment of the present application and camera coordinates corresponding to calibration points on a target electronic device.

Detailed ways

The terms "first", "second" and "third" in the specification, claims and description of the drawings of this application are used to distinguish different objects, rather than to limit a specific order.

In the embodiments of the present application, words such as "exemplary" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application shall not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner.

In order to make the description of the following embodiments clear and concise, a brief introduction of related technologies is first given:

The machine vision system uses machines instead of human eyes to make measurements and judgments, converts the captured target into an image signal through an image capture device (such as a camera), and further processes the image signal to obtain certain characteristics of the target, and then controls the scene based on the characteristics. Device action.

Coordinate system calibration refers to the calibration of the conversion relationship between two different coordinate systems. For example, in this application, it involves the visual coordinate system (also called the camera coordinate system), the mechanical coordinate system (that is, the coordinate system of the manipulator fixture platform) coordinate system) and the device coordinate system to calibrate the conversion relationship between the three coordinate systems, and help the robot (ie, the manipulator system) to convert the obtained visual information through the coordinate system calibration, so as to complete the subsequent control work, such as visual grasping, etc.

The following is an example of a scene where a manipulator system is used to automatically test the screen of an electronic device. Among them, a manipulator system positioning method provided by the related art is as follows:

In the stage of developing the automation script, first place the electronic device on the fixture platform of the manipulator, move the manipulator to the specified measured position, record the mechanical coordinates corresponding to these measured positions, and write these mechanical coordinates into the test script .

In the testing phase of electronic equipment, the automation script is executed, and the manipulator is controlled to move to the corresponding measured position according to the mechanical coordinates of the measured position in the test script, and the corresponding test operation is performed on the measured position.

It can be seen that in the stage of developing the automation script, the process relies on the manipulator to obtain the mechanical coordinates corresponding to the measured position, resulting in low development efficiency of the automation script. In addition, if the physical size of the test equipment changes, it is necessary to rely on the manipulator again to obtain the mechanical coordinates of the measured position and modify the coordinates of the test points in the automation script. It can be seen that the automation script developed in this way is suitable for electronic devices of different physical sizes. Poor compatibility.

Another positioning method of the manipulator system in the related art is: obtain the mechanical coordinates of the measured position in the electronic device through the mapping relationship between the camera coordinate system and the mechanical coordinate system, and then control the movement of the manipulator according to the mechanical coordinates corresponding to the measured position.

For example, in order to test whether the basic functions of the application program installed in the electronic device meet the requirements, it is necessary to control the corresponding controls (such as setting controls, pop-up window deletion controls, confirmation controls, etc.) to perform corresponding operations. In this scenario, the icon of the control is easier to identify through image recognition technology. Therefore, the machine coordinates of the corresponding position of the icon can be obtained by establishing the mapping relationship between the machine coordinate system and the camera coordinate system. The specific process is as follows:

In the stage of developing the automated test script, a camera is used to take a picture of the screen of the electronic device containing the icon to be tested, and the picture of the icon to be recognized is further obtained from the picture and saved as a template picture.

When executing automated test scripts for automated testing of electronic devices, as shown in Figure 1, target electronic devices (such as mobile phones, tablet computers, watches, etc.) are placed on the fixture platform of the manipulator. Then, the image of the target electronic device as shown in FIG. 2 is obtained by shooting with a camera. After identifying the icon object contained in the image that matches the template picture based on the image recognition technology, the camera coordinates corresponding to the icon object (such as point P in FIG. 2 ) are obtained.

Further, according to the mapping relationship between the machine coordinate system and the camera coordinate system, the camera coordinates of point P are converted into coordinates in the machine coordinate system, that is, the machine coordinates of point P are obtained. Finally, based on the mechanical coordinates of point P, the manipulator is controlled to perform corresponding operations.

It can be seen that in this way, when obtaining the mechanical coordinates of the device under test, a camera needs to be used to capture corresponding images, resulting in low test efficiency.

In order to solve the above technical problems, the inventors of the present application provide a positioning method for the manipulator system. The solution is to set at least two calibration points on the gripper platform of the manipulator, that is, the mechanical coordinates of the calibration points are known. The image of the fixture platform is captured by the camera, and the coordinates of the calibration point on the fixture platform in the camera coordinate system are obtained, that is, the camera coordinates of the calibration point. Through the machine coordinates and camera coordinates corresponding to the calibration points on the fixture platform, the mapping relationship between the machine coordinate system and the camera coordinate system is obtained. And, at least two calibration points are determined on the electronic device under test, that is, the coordinates of the calibration points on the electronic device under test are known in the device coordinate system, that is, the device coordinates. Then, the camera is used to capture the image of the electronic device under test, and the camera coordinates of the calibration points on the electronic device under test are obtained. Further, the mapping relationship between the device coordinate system and the camera coordinate system is obtained. Further, according to the above-mentioned mapping relationship between the machine coordinate system and the camera coordinate system, and the mapping relationship between the camera coordinate system and the device coordinate system, the mapping relationship between the device coordinate system and the machine coordinate system is obtained. Based on the mapping relationship between the mechanical coordinate system and the device coordinate system, the device coordinates of any point on the electronic device under test can be directly converted into mechanical coordinates. Finally, based on the mechanical coordinates of this point, the manipulator is controlled to move to the corresponding position and perform corresponding operations.

It can be seen that the manipulator positioning method provided by this scheme can directly control the manipulator based on the equipment coordinates of the measured position of the electronic device under test, and the automation script obtained by this scheme does not need to be written into the mechanical coordinates of the measured position. In other words, the automation of this scheme The development and debugging process of the script is decoupled from the manipulator device, thereby improving the development and debugging efficiency of the automation script.

At the same time, this solution can directly obtain the equipment coordinates of the measured position without first using the camera to obtain the camera coordinates of the measured position, which reduces the time-consuming process of obtaining the mechanical coordinates of the measured position and improves the efficiency of the entire testing process. Test efficiency.

Further, in the scenario of performing automated testing on the tested control of the device under test, the tested location is the location of the tested control. In this scenario, the automation script of this solution does not need to write the mechanical coordinates of the measured position of the electronic device, but writes the control ID corresponding to the tested control. For the same type of tested electronic device, such as two different models The physical size of mobile phones (or other electronic devices) is different, but the control ID corresponding to the same control under test does not change. Therefore, there is no need to modify the automation script for this type of electronic device under test, that is, the automation script obtained by this scheme can be compatible with different The physical size of the device under test improves the compatibility of automation scripts with different physical sizes.

In order to make the description of the following embodiments clear and concise, the working process of the manipulator system is illustrated by taking the automatic test of the mobile phone by the manipulator system as an example.

Please refer to FIG. 3 , which shows a schematic structural diagram of a manipulator system provided by an embodiment of the present application. As shown in FIG. 3 , the manipulator system includes: a manipulator device, at least one camera (or may be called a video camera), and a computer. Among them, the manipulator

Including manipulator equipment includes manipulator (or can be called operating mechanism) and fixture platform (or can be called support platform).

It can be understood that the structure of the automatic assembly equipment shown in this embodiment does not constitute a specific limitation on the equipment. In other embodiments, the automated assembly equipment may include more or fewer components than shown, or combine certain components, or separate certain components, or a different arrangement of components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

In the automated testing scenario, the manipulator is used to control the electronic equipment for automated testing, and the fixture platform is used to fix the electronic equipment. For example, the mobile phone to be tested is fixed on the fixture platform, so that automated tests such as stability test, function test and power consumption test can be performed on the mobile phone.

In this application, the target electronic device refers to the automatic testing by the manipulator system, and the target electronic device can be a mobile phone, a tablet computer, a desktop, a laptop, a notebook computer, an ultra-mobile personal computer (Ultra-mobile Personal Computer, UMPC), Handheld computers, netbooks, personal digital assistants (Personal Digital Assistant, PDA), wearable electronic devices, smart watches and other devices, this application does not specifically limit the specific form of electronic devices.

The camera is used to take images of the target electronic device and determine the position coordinates of the target electronic device. Any one of a CCD camera and a CMOS camera can be used as the camera.

The computer is used to control the motion state of the manipulator, control the working state of the camera, and perform image analysis on the image captured by the camera to obtain the coordinates of the target position in the image. Wherein, the positioning method of the manipulator system provided by the present application runs on the computer shown in FIG. 3 .

The computer sends a shooting instruction to the camera, and the camera executes the shooting instruction to perform corresponding shooting operations. The camera sends the captured image to the computer, so that the computer can process the image to obtain the coordinates of the target position in the image, and convert the coordinates of the target position into mechanical coordinates. Further, the computer generates motion control instructions based on the mechanical coordinates of the target position and sends them to the manipulator to control the manipulator to move to the target position and perform corresponding operations.

Taking the target electronic device as a mobile phone as an example, FIG. 4 shows the coordinate system of the mobile phone, that is, the device coordinate system.

Assuming that the manipulator can ensure that the above three coordinate systems (ie camera coordinate system, machine coordinate system and device coordinate system) are on the same plane by improving the accuracy during production, as shown in Figure 5, which shows the camera coordinate system on the same plane , machine coordinate system and device coordinate system.

As shown in Figure 5, the xoy coordinate system is the camera coordinate system, the x'o'y' coordinate system is the mechanical coordinate system, and the x"o"y" coordinate system is the device coordinate system. The origin of different coordinate systems and the positive coordinates of the coordinate axes The direction is different.

As mentioned above, the machine vision system is established based on the camera coordinate system, and the operation of the manipulator is based on the mechanical coordinate system. Therefore, the mapping relationship between the camera coordinate system and the mechanical coordinate system needs to be established. After obtaining the mapping relationship between the two coordinate systems, for the same location point, if the coordinates of the location point in any coordinate system are known, according to the mapping relationship between the two coordinates, it can be converted to the location point in another Coordinates in the coordinate system. For example, after obtaining the camera coordinates corresponding to the position point, according to the mapping relationship between the camera coordinates and the machine coordinates, the machine coordinates corresponding to the position point can be converted to obtain.

The flow of the positioning method for the manipulator system provided by the embodiment of the present application will be described below with reference to FIG. 6 , and the method runs on the computer in FIG. 3 . As shown in Figure 6, the manipulator system positioning method includes the following steps:

S110. Obtain mechanical coordinates of at least two calibration points A and B on the fixture platform of the manipulator.

In one example, when the manipulator is produced, two points can be fixed on the fixture platform of the manipulator as calibration points, and the mechanical coordinates of at least two calibration points can be marked when the manipulator leaves the factory. For example, the mechanical coordinates of each calibration point can be marked directly on the fixture platform.

Wherein, the number of calibration points on the fixture platform may be two or more, which is not limited in this application.

In another example, the party using the manipulator may select two fixed points on the fixture platform of the manipulator as calibration points, and determine the mechanical coordinates corresponding to the two calibration points according to the mechanical coordinate system of the manipulator.

For example, as shown in FIG. 7 , the mechanical coordinates corresponding to the two calibration points on the fixture platform are A(x′ ₁ , y′ ₁ ), B(x′ ₂ , y′ ₂ ), respectively.

In an exemplary embodiment, the image recognition accuracy can be improved by modifying the color of the calibration points on the fixture platform or adding fluorescence.

S120. Obtain camera coordinates corresponding to each calibration point on the fixture platform.

The image of the fixture platform can be taken by a high-definition camera, as shown in FIG. 8 , which is a schematic diagram of the image of the fixture platform captured by the camera. The image includes the fixture platform and calibration points A and B on the fixture platform.

Further obtain the coordinates of the two calibration points in the image in the camera coordinate system through image recognition technology, that is, obtain the camera coordinates corresponding to the calibration points A and B in the image, such as A(x ₁ , y ₁ ), B(x ₂ , y ₂ ).

S130. Obtain a mapping relationship between the machine coordinate system and the camera coordinate system, or called a first mapping relationship, according to the machine coordinates and camera coordinates corresponding to the calibration points on the fixture platform.

If the camera coordinates and machine coordinates corresponding to a certain point are known, the mapping relationship between the camera coordinate system and the machine coordinate system can be established. For example, the camera coordinates of a certain point are (x, y), and the point is The coordinates in the coordinate system are (x', y'), then the mapping relationship between the camera coordinate system and the machine coordinate system can be expressed by the following formula (1):

a, b, c, d in formula (1) are unknown parameters.

The camera coordinates (x ₁ , y ₁ ), (x ₂ , y ₂ ) corresponding to the two calibration points of A and B are known, and the mechanical coordinates (x′ ₁ , y′ ₁ ) and ( x′ ₂ , y′ ₂ ), respectively substituting the camera coordinates and machine coordinates corresponding to points A and B into formula (1) to obtain the equation group shown in formula (2):

According to the equations shown in formula (2), the specific values of the unknown parameters a, b, c, and d can be solved, that is, if the camera coordinates of any point are known, the corresponding mechanical coordinates of the point can be converted by using the mapping relationship . Similarly, if the machine coordinates corresponding to a certain point are known, the camera coordinates corresponding to the point can be converted.

The mapping relationship between the machine coordinate system and the camera coordinate system only needs to be recalculated after the camera position changes. On the premise that the camera position does not change, the processes described in S110-S130 only need to be performed once, and the first mapping relationship can be directly used subsequently.

For example, the position of the camera may change in scenarios such as camera inspection or replacement. After the camera position is changed, recalculate the mapping relationship between the machine coordinate system and the camera coordinate system according to the above-mentioned process of S110-S130.

S140. Acquire coordinates (ie, device coordinates) of at least two calibration points on the target electronic device in the device coordinate system.

In this embodiment of the present application, the target electronic device is a mobile phone as an example for description. The number of calibration points marked on the target electronic device may be two or more, which is not limited in this application.

In the scenario of automatically testing the screen of the mobile phone, two points on the mobile phone can be selected as calibration points, for example, the locations of two controls on the display interface of the mobile phone can be selected as the calibration points. Alternatively, a picture containing the calibration points can be displayed on the mobile phone. The resolution of the picture is the same as that of the screen of the mobile phone. The picture can be a solid color picture, and the color of the calibration points is different from the color of the picture. For example, the picture can be White, the color of the calibration point can be black.

For the scene where the calibration point is a control displayed on the display interface of the mobile phone, the UI automation test tool can be used to directly obtain the device coordinates corresponding to each control. For example, the device coordinates corresponding to the controls can be obtained by using a UI automated testing tool (such as UIAutomator) installed in the electronic device.

For the scene where the calibration point is the calibration point in the picture, the picture containing the calibration point can be generated according to the screen resolution of the device under test, that is, the device coordinates of the calibration point in the picture are known.

For example, as shown in FIG. 9 , the device coordinates corresponding to the two calibration points on the mobile phone are C(x″ ₃ , y″ ₃ ) and D(x″ ₄ , y″ ₄ ), respectively.

S150. Obtain camera coordinates corresponding to the calibration points on the target electronic device.

Similar to the manner of obtaining the camera coordinates of the calibration point on the fixture platform, the target electronic device is fixed at a fixed position on the fixture platform of the manipulator, and then the image of the target electronic device is captured by a high-definition camera.

Further, the camera coordinates corresponding to the calibration points C and D in the image are obtained through image recognition technology, for example, as shown in Figure 10, the coordinates of the calibration points C and D in the camera coordinate system are C(x′ ₃ , y′ ₃ ) and D(x′ ₄ , y′ ₄ ).

S160. Obtain a mapping relationship between the camera coordinate system and the device coordinate system, or called a second mapping relationship, according to the device coordinates and camera coordinates corresponding to the calibration point on the target electronic device.

For the same location point, if the coordinates of the point in the device coordinate system and the coordinates of the point in the camera coordinate are known, the mapping relationship between the device coordinate system and the camera coordinate system can be established. For example, the mapping relationship can be represented by the following formula (3):

In formula (3), a ₁ , b ₁ , c ₁ , and d ₁ are all unknown parameters.

Substituting the device coordinates and camera coordinates corresponding to points C and D on the target electronic device into the formula (3), the equation group shown in the following formula (4) is obtained:

According to the equations shown in formula (4), the specific values of the unknown parameters a ₁ , b ₁ , c ₁ , and d ₁ can be obtained by solving them, that is, if the camera coordinates corresponding to any point on the target electronic device are known, use the mapping The relationship can be converted to obtain the device coordinates corresponding to the point. Similarly, if the device coordinates corresponding to any point on the target electronic device are known, the camera coordinates corresponding to the point can be obtained through conversion.

After obtaining the second mapping relationship between the camera coordinate system and the device coordinate system through the process described in S140-S160, the second mapping relationship can be directly used subsequently, that is, S140-S160, provided that the physical size of the electronic device does not change It only needs to be done once.

When the physical size of the electronic device changes, the second mapping relationship will be updated following the change of the electronic device, for example, different types of test equipment (such as different models of mobile phones), or different types of test equipment (such as, The physical size of the mobile phone and tablet computer, smart watch, etc.) may be different, and the physical size of the test device is different, and the corresponding device coordinate system is also different. Therefore, it is necessary to establish a device coordinate system and a camera coordinate system for test devices with different physical sizes. mapping relationship between them. That is, after the test device is replaced, the second mapping relationship needs to be re-determined according to S140-S160.

Through the above steps, the parameters of the mapping relationship between the machine coordinate system and the camera coordinate system, that is, the specific values of a, b, c, and d, and the parameters of the mapping relationship between the device coordinate system and the camera coordinate system, Such as the specific values of a ₁ , b ₁ , c ₁ , and d ₁ . Further, the mapping relationship between the machine coordinate system and the device coordinate system may be obtained according to the parameter values of the first mapping relationship and the second mapping relationship, or may be called a third mapping relationship.

In another example, calibration points A and B are marked on the fixture platform, and the target electronic device displayed with calibration points C and D can be directly fixed on the fixed position of the fixture platform. Then, a camera is used to take an image including the fixture platform and the target electronic device, that is, an image as shown in FIG. 11 is obtained.

As shown in FIG. 11 , the image includes the jig platform and the mobile phone placed on the jig platform, and includes calibration points A and B on the jig platform, and calibration points C and D on the mobile phone.

By analyzing an image shown in Figure 11, the camera coordinates corresponding to the calibration points A and B on the fixture platform and the camera coordinates corresponding to the calibration points C and D on the mobile phone can be obtained at the same time. There is no need for a camera to take images of the fixture platform and test equipment separately, and perform image analysis to obtain the camera coordinates of calibration points A, B, and calibration points C and D. Improved efficiency of the coordinate system calibration process.

S170. Obtain a third mapping relationship between the machine coordinate system and the device coordinate system based on the first mapping relationship between the machine coordinate system and the camera coordinate system and the second mapping relationship between the device coordinate system and the camera coordinate system.

Substituting formula (1) into formula (3), the third mapping relationship between the machine coordinate system and the device coordinate system can be obtained. For example, the third mapping relationship is shown in formula (5):

Among them, a, b, c, d and a ₁ , b ₁ , c ₁ , d ₁ are all known, therefore, the parameters aa ₁ -bb ₁ , a ₁ b+ab ₁ , -ab ₁ -a ₁ b, a _{Both 1} c+b ₁ d+c ₁ and a ₁ db ₁ c+d ₁ can be calculated to obtain specific values.

It should be noted that, when the position of the camera and the physical size of the electronic device remain unchanged, the process of obtaining the third mapping relationship described in S170 only needs to be performed once. Three mapping relationships. For example, when testing electronic devices with the same physical size, the process described in S110-S170 above only needs to be performed once to obtain the third mapping relationship. When testing multiple electronic devices with the same physical size, the first mapping can be directly used. Three mapping relationships.

S180. Acquire device coordinates corresponding to the target position on the target electronic device, and convert the device coordinates of the target position into mechanical coordinates based on the third mapping relationship.

In the scenario where a manipulator is used to operate the screen of a mobile phone to realize automated testing, the manipulator needs to move to the position to be tested for corresponding test operations. For example, when testing the touch performance of a mobile phone screen, the manipulator needs to move to the test point and touch the screen; another example, when performing a stress test on the mobile phone screen, the manipulator also needs to move to the test point and press the screen.

After obtaining the device coordinates corresponding to the measured position on the screen, the device coordinates of the measured position can be directly converted into mechanical coordinates according to the third mapping relationship. Further, the manipulator is controlled to move to the corresponding measured position based on the mechanical coordinates corresponding to the measured position. In one scenario, the location to be tested is a control on the display interface of the mobile phone. In this scenario, a UI automated testing tool, such as UI Automator, can be installed in the target electronic device, and the device of the tested control can be obtained through the UI automated testing tool coordinate. Moreover, after replacing other electronic devices of the same type, such as after testing some controls of a certain model of mobile phone, continue to test these controls of another model of mobile phone. In this scenario, the control IDs of these tested controls remain unchanged. The device coordinates of the corresponding controls of these tested controls can be quickly obtained by using the UI automated test tool, that is, the coordinates of the tested controls can be quickly obtained after replacing the electronic device, without modifying the automation script, and the compatibility of the automation script is improved.

In another scenario, if the measured position is a position without controls, such as a blank position on the display interface of the mobile phone, in this scenario, position the finger at the measured position of the mobile phone, and use the "pointer position" of the electronic device to Function, you can get the device coordinates of the touched position, and write the device coordinates of the measured position into the automation script. In the stage of testing the tested equipment, based on the mapping relationship between the equipment coordinate system and the mechanical coordinate system, the equipment coordinates of the measured position are converted into mechanical coordinates, and the manipulator is controlled to move to the measured position based on the mechanical coordinates. Execute the corresponding test operation. It can be seen that this kind of scenario does not need to rely on the manipulator to develop automation scripts.

S190. Generate a motion control command based on the mechanical coordinates of the target position, and send the motion control command to the manipulator, so that the manipulator moves to the target position.

In one example, computer-generated motion control instructions can directly control manipulator movement. For example, based on the current position of the manipulator and the mechanical coordinates corresponding to the target position, the computer generates motion control commands to control the movement of the manipulator based on the mechanical coordinates of these two positions, for example, move 300mm to the left, and then move 500mm to the front, the generated motion control Commands are sent to the robot. The manipulator moves from the current position to the target position after receiving the motion control command and executing the command.

In another example, the motion control command generated by the computer is used to inform the manipulator of the target position. For example, after the computer obtains the mechanical coordinates of the target position, it sends a motion control command to the manipulator, and the motion control command includes the target position. After analyzing the motion control command, the manipulator obtains the mechanical coordinates of the target position. Further, the manipulator generates a movement command according to its current position and the received target position, and executes the movement command to move from the current position to the target position.

The embodiment of the present application does not limit the control method for moving the manipulator from the current position to the target position.

S1100, sending a test instruction to the manipulator, so that the manipulator performs a corresponding test operation on the target position on the target electronic device.

In one example, after the manipulator moves to the target position, it feeds back to the computer that it has moved to the target position. At this time, the computer sends test instructions to the manipulator, such as controlling the manipulator to perform operations such as touching or pressing. After receiving the test execution, the manipulator parses and executes the test instructions to realize the automated test of the test equipment.

In the positioning method of the manipulator system provided in this embodiment, at least two calibration points are set on the fixture platform of the manipulator, that is, the mechanical coordinates of the calibration points in the mechanical coordinate system are known. The image of the fixture platform is captured by the camera, and the camera coordinates of the calibration points on the fixture platform are obtained. Through the machine coordinates and camera coordinates corresponding to at least two calibration points on the fixture platform, the mapping relationship between the machine coordinate system and the camera coordinate system is obtained. Similarly, at least two calibration points may be determined on the target electronic device, that is, the device coordinates of the calibration points on the target electronic device are known. Then, the camera is used to capture the image of the target electronic device, and the camera coordinates of the calibration points on the target electronic device are obtained. Further, the mapping relationship between the device coordinate system and the camera coordinate system is obtained. Further, according to the mapping relationship between the above two coordinate systems, the mapping relationship between the equipment coordinate system and the machine coordinate system is obtained. Finally, based on the mapping relationship between the machine coordinate system and the device coordinate system, the device coordinates of any point on the target electronic device can be directly converted into machine coordinates. Finally, based on the mechanical coordinates of this point, the manipulator is controlled to move to the corresponding position and perform corresponding operations. It can be seen that using this scheme, the manipulator can be directly controlled based on the equipment coordinates of the input target electronic equipment. The automation script obtained by this scheme does not need to be written into the mechanical coordinates of the measured position. In other words, the development and debugging process of the automation script of this scheme is similar to that of the manipulator Device decoupling improves the development and debugging efficiency of automation scripts.

At the same time, this solution can directly obtain the equipment coordinates of the measured position without first using the camera to obtain the camera coordinates of the measured position, which reduces the time-consuming process of obtaining the mechanical coordinates of the measured position and improves the efficiency of the entire testing process. Test efficiency.

Further, after changing the camera position, the camera coordinate system also changes accordingly, the first mapping relationship between the machine coordinate system and the camera coordinate system, the second mapping relationship between the camera coordinate system and the device coordinate system, and the machine coordinate system The third mapping relationship with the device coordinate system will change accordingly. Similarly, after the test equipment is replaced, the physical size of the test equipment changes, the equipment coordinate system changes accordingly, the second mapping relationship between the equipment coordinate system and the camera coordinate system, and the third mapping relationship between the equipment coordinate system and the machine coordinate system Also follow the change. Using the solution of this application, at least two calibration points are respectively set on the fixture platform of the manipulator and the test equipment in advance, based on the calibration points on the fixture platform, the first coordinate system between the machine coordinate system and the camera coordinate system can be quickly and automatically calibrated. Mapping relations. The second mapping relationship between the device coordinate system and the camera coordinate system can be quickly and automatically calibrated based on the calibration points on the test device. After updating the first mapping relationship and the second mapping relationship, the third mapping relationship can be quickly updated. It can be seen that this scheme can quickly realize the automatic calibration among the three coordinate systems, and improves the efficiency of the coordinate system calibration process.

Further, in the scenario of performing automated testing on the tested control of the device under test, the tested location is the location of the tested control. In this scenario, the automation script of this solution does not need to write the mechanical coordinates of the measured position of the electronic device, but writes the control ID corresponding to the tested control. For the same type of tested electronic device, such as two different models The physical size of mobile phones (or other electronic devices) is different, but the control ID corresponding to the same control under test does not change. Therefore, there is no need to modify the automation script for this type of electronic device under test, that is, the automation script obtained by this scheme can be compatible with different The physical size of the device under test improves the compatibility of automation scripts with different physical sizes.

Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of the description, only the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be allocated according to needs It is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. For the specific working process of the above-described system, device, and unit, reference may be made to the corresponding process in the foregoing method embodiments, and details are not repeated here.

In the several embodiments provided in this embodiment, it should be understood that the disclosed system, device and method may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be Incorporation may either be integrated into another system, or some features may be omitted, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of this embodiment may be integrated into one processing unit, or each unit may physically exist separately, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium In, several instructions are included to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor execute all or part of the steps of the method described in each embodiment. The aforementioned storage medium includes: flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk, and other various media capable of storing program codes.

The above is only a specific implementation of the application, but the protection scope of the application is not limited thereto, and any changes or replacements within the technical scope disclosed in the application should be covered within the protection scope of the application . Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims (13)

1. A positioning method for a manipulator system, comprising: Obtain the first mapping relationship between the machine coordinate system and the camera coordinate system; Obtain a second mapping relationship between the camera coordinate system and the device coordinate system of the electronic device based on the device coordinates and camera coordinates corresponding to at least two device calibration points of the electronic device, the device calibration points being the display interface of the electronic device The calibration control on the computer, or the calibration point in the preset picture displayed by the electronic device; Obtaining a third mapping relationship between the equipment coordinate system and the machine coordinate system based on the first mapping relationship and the second mapping relationship; Acquiring the device coordinates of the target position of the electronic device in the device coordinate system, and converting based on the third mapping relationship to obtain the mechanical coordinates of the target position. 2. The method according to claim 1, wherein said obtaining the first mapping relationship between the mechanical coordinate system and the camera coordinate system comprises: Obtain the mechanical coordinates corresponding to at least two manipulator calibration points on the manipulator device; Obtain the camera coordinates of each manipulator calibration point in the camera coordinate system through the camera; Based on the machine coordinates and camera coordinates corresponding to the at least two manipulator calibration points, a first mapping relationship between the machine coordinate system and the camera coordinate system is obtained. 3. The method according to claim 2, wherein said acquiring mechanical coordinates corresponding to at least two manipulator calibration points on the manipulator device comprises: At least two calibration points are set on the fixture platform of the manipulator device, and the mechanical coordinates of each calibration point are obtained based on the equipment information of the manipulator device. 4. The method according to claim 1 or 2, wherein the camera coordinate system and the device coordinate system of the electronic device are obtained based on the device coordinates and camera coordinates corresponding to at least two device calibration points of the electronic device The second mapping relationship between, including: Obtain device coordinates corresponding to at least two device calibration points on the electronic device; Obtain the camera coordinates of each device calibration point in the camera coordinate system through the camera; Based on the device coordinates and camera coordinates corresponding to the at least two device calibration points, a second mapping relationship between the camera coordinate system and the device coordinate system is obtained. 5. The method according to claim 4, wherein the device calibration point is a calibration control on the display interface of the electronic device; obtain the device coordinates corresponding to at least two device calibration points on the electronic device ,include: Acquiring a control identifier corresponding to the calibration control; sending a control coordinate request to the electronic device, where the control coordinate request includes the control identifier; Receive the device coordinates corresponding to the calibration control sent by the electronic device. 6. The method according to claim 4, wherein the device calibration point is a calibration point in a preset picture displayed by the electronic device; obtain the corresponding information of at least two device calibration points on the electronic device Device coordinates, including: Based on the coordinates of the calibration point in the preset picture, the device coordinates of the calibration point in the device coordinate system are obtained. 7. The method according to claim 4, wherein the obtaining the camera coordinates of each device calibration point in the camera coordinate system through the camera comprises: receiving an image of the electronic device captured by a camera, the image including a calibration point of the device; The device calibration points contained in the image are identified based on image recognition technology, and the position coordinates of each device calibration point in the image are determined to obtain the camera coordinates of the device calibration points. 8. The method according to claim 1, wherein the target position is a non-control position of the display interface of the electronic device; the acquisition of the target position of the electronic device in the device coordinate system Device coordinates, including: receiving device coordinates corresponding to the target position sent by the electronic device, where the device coordinates are obtained after the electronic device receives a click operation on the target position. 9. The method according to claim 1, wherein converting the mechanical coordinates of the target position based on the third mapping relationship comprises: The device coordinates of the target position are input into the mapping relationship, and the machine coordinates corresponding to the target position are calculated. 10. The method of claim 1, further comprising: controlling the manipulator device to move to the target position of the electronic device based on the mechanical coordinates of the target position; The manipulator is controlled to perform corresponding operations at the target position. 11. The method according to claim 10, wherein controlling the manipulator device to move to the target position of the electronic device based on the mechanical coordinates of the target position comprises: generating a motion control instruction based on the mechanical coordinates of the target position and the mechanical coordinates of the current position of the manipulator device; sending the motion control command to the manipulator device, so that the manipulator device moves from the current position to the target position in response to the motion control command. 12. A computer device, characterized in that the electronic device comprises: one or more processors, memory and touch screen; the memory is used to store program code; the processor is used to run the program code, so that the The electronic device implements the manipulator system positioning method according to any one of claims 1 to 11. 13. A computer-readable storage medium, characterized in that instructions are stored thereon, and when the instructions are run on a computer device, the electronic device executes the manipulator according to any one of claims 1 to 11 System positioning method.

Images