CN111966068A - Augmented reality monitoring method and device for motor production line, electronic equipment and storage medium - Google Patents

Abstract

The present application provides an augmented reality monitoring method, device, electronic device and storage medium for a motor production line, and relates to the technical field of industrial production. The method includes: establishing a digital twin model of a motor production line; obtaining real-time data of the motor production line through a real-time data acquisition module; driving the operation of the digital twin model based on the real-time data to obtain virtual motor production line information; The production line information is sent to the virtual-real integration module and the monitoring module, so that the virtual-real integration module obtains augmented reality information based on the virtual motor production line information, and the monitoring module obtains augmented reality information based on the virtual motor production line information and the augmented reality The information is displayed in virtual reality or augmented reality on the motor production line. The method performs virtual reality display or augmented reality display on the motor production line based on real-time data and digital twin model, and realizes the visual monitoring of the whole process and full parameters of the digital workshop of the motor production line.

Description

Augmented reality monitoring method, device, electronic device and storage medium for motor production line

technical field

The present application relates to the technical field of industrial production, and in particular, to an augmented reality monitoring method, device, electronic device and storage medium for a motor production line.

Background technique

At present, the digital monitoring of the workshop generally uses 2D mode to monitor the operating parameters of key equipment, and some can be combined with simple three-dimensional models for visual display, but none of them can adapt to the entire process and full parameter monitoring requirements of the digital workshop for the motor production line.

SUMMARY OF THE INVENTION

In view of this, the purpose of the embodiments of the present application is to provide an augmented reality monitoring method, device, electronic device and storage medium for a motor production line, so as to improve the whole process, the whole process and the whole process that cannot be adapted to the digital workshop for the motor production line in the prior art. The problem with the monitoring requirements of the parameters.

The embodiment of the present application provides an augmented reality monitoring method for a motor production line, which is applied to a simulation module. The method includes: establishing a digital twin model of a motor production line; acquiring real-time data of the motor production line through a real-time data acquisition module, The real-time data includes production line information and product information; drive the digital twin model to operate based on the real-time data to obtain virtual motor production line information; send the virtual motor production line information to the virtual-real fusion module and the monitoring module to The virtual-real fusion module obtains augmented reality information based on the virtual motor production line information, and the monitoring module performs virtual reality display or enhancement on the motor production line based on the virtual motor production line information and the augmented reality information Reality shows.

In the above implementation manner, the three-dimensional virtual reconstruction of the motor production line is completed through the virtual reconstruction technology to obtain a digital twin model, and the running status of the motor production line is digitized and visualized based on real-time data such as production line information and product information. The visual monitoring of the whole process and all parameters of the digital workshop of the motor production line is convenient for workshop management and scheduling, and ensures the production quality of products.

Optionally, the establishing a digital twin model of the motor production line includes: acquiring a real image of the motor production line through a real camera; extracting target features from the real image to obtain a feature image; establishing a feature image based on the feature image the digital twin.

In the above implementation manner, the digital twin model is established based on the real image of the motor production line obtained by the real camera, which provides a model basis for the subsequent virtual reality and augmented reality display of the motor production line, thereby improving the accuracy of the model and virtual display. sex.

Optionally, the digital twin model includes sub-models of each device of the motor production line, and the devices of the motor production line include a six-axis robot, a paper insertion machine, a winding and embedding machine, a binding machine, and a conveyor rail. at least one.

In the above implementation manner, a digital twin model is established for each device on the motor production line, which improves the accuracy of the digital twin model of the motor production line.

Optionally, the acquiring the real-time data of the motor production line through the real-time data acquisition module includes: acquiring the programmable logic controller of each device of the motor production line through the Modbus interface of the real-time data acquisition module. PLC real-time data, the PLC real-time data is the production line information, and the PLC real-time data includes at least one of the angle of each joint of the robot, the control instruction of the robot gripper, the status information of the processing equipment, and the control instruction of the motion structure of the processing equipment; The product information corresponding to the motor production line in the production process execution management system of the manufacturing enterprise is collected through the open database of the real-time data collection module and connected to the ODBC interface, and the product information includes the work order number, material code and work order plan at least one of the.

In the above implementation manner, the real-time data and product information of the PLC are collected to determine the motion state of the motor production line and products on the production line, so that the real-time operation state of the workshop can be digitized in combination with the actual motion parameters of the equipment and the product information of the motor production line. This ensures the accuracy of the motor production line virtualization.

The embodiment of the present application also provides an augmented reality monitoring method for a motor production line, which is applied to a virtual-real fusion module. The method includes: collecting a real image of the motor production line through a real camera; obtaining virtual motor production line information through a simulation module; generating augmented reality information from the real image and the virtual motor production line information; sending the augmented reality information to a monitoring module, so that the monitoring module performs augmented reality display on the motor production line based on the augmented reality information .

In the above implementation process, the augmented reality information is generated through the real image of the motor production line and the virtual motor production line information, so that the virtual display and enhanced display of the motor production line can be simultaneously displayed, which improves the comprehensiveness of monitoring.

Optionally, the generating the augmented reality information based on the real image and the virtual motor production line information includes: acquiring virtual coordinate data of the motor production line in the virtual motor production line information; based on the virtual coordinates The data determines a virtual camera posture, the virtual camera posture includes position and posture data; the real camera posture of the real camera is determined based on the real image; based on the real camera posture and the virtual camera posture, the real image is Fusion with virtual objects in the virtual motor production line information to obtain the augmented reality information.

In the above implementation process, based on the matching of the virtual camera pose and the real camera pose in the virtual motor production line information, the virtual object and the real image are fused to complete the augmented reality display of the motor production line, which improves the image of the augmented reality display and the motor production. line matching.

The embodiment of the present application also provides an augmented reality monitoring device for a motor production line, which is applied to a simulation module. The device includes: a model establishment module for establishing a digital twin model of the motor production line; a real-time data acquisition module for The real-time data acquisition module obtains real-time data of the motor production line, and the real-time data includes production line information and product information; the virtual reality module is used to drive the operation of the digital twin model based on the real-time data to obtain a virtual motor production line information; a first sending module, configured to send the virtual motor production line information to the virtual-real fusion module and the monitoring module, so that the virtual-real fusion module obtains augmented reality information based on the virtual motor production line information, and the monitoring The module performs virtual reality display or augmented reality display on the motor production line based on the virtual motor production line information and the augmented reality information.

In the above implementation manner, the three-dimensional virtual reconstruction of the motor production line is completed through the virtual reconstruction technology to obtain a digital twin model, and the running status of the motor production line is digitized and visualized based on real-time data such as production line information and product information. The visual monitoring of the whole process and all parameters of the digital workshop of the motor production line is convenient for workshop management and scheduling, and ensures the production quality of products.

Optionally, the model building module is specifically used to: obtain a real image of the motor production line through a real camera; perform target feature extraction on the real image to obtain a feature image; build the digital twin based on the feature image Model.

In the above implementation manner, the digital twin model is established based on the real image of the motor production line obtained by the real camera, which provides a model basis for the subsequent virtual reality and augmented reality display of the motor production line, thereby improving the accuracy of the model and virtual display. sex.

Optionally, the digital twin model includes sub-models of each device of the motor production line, and the devices of the motor production line include a six-axis robot, a paper insertion machine, a winding and embedding machine, a binding machine, and a conveyor rail. at least one.

In the above implementation manner, a digital twin model is established for each device on the motor production line, which improves the accuracy of the digital twin model of the motor production line.

Optionally, the real-time data acquisition module is specifically configured to: collect the programmable logic controller (PLC) real-time data of the various devices of the motor production line through the Modbus interface of the real-time data acquisition module, and the PLC real-time data For the production line information, the PLC real-time data includes at least one of the robot joint angles, robot gripper control instructions, processing equipment status information, and processing equipment kinematic structure control instructions; through the opening of the real-time data acquisition module The database is connected to the ODBC interface to collect the product information corresponding to the motor production line in the production process execution management system of the manufacturing enterprise, and the product information includes at least one of a work order number, a material code and a work order plan.

In the above implementation manner, the real-time data and product information of the PLC are collected to determine the motion state of the motor production line and products on the production line, so that the real-time operation state of the workshop can be digitized in combination with the actual motion parameters of the equipment and the product information of the motor production line. This ensures the accuracy of the motor production line virtualization.

The embodiment of the present application also provides an augmented reality monitoring device for a motor production line, which is applied to a virtual-real fusion module. The device includes: a real image acquisition module for acquiring real images of the motor production line through a real camera; a virtual information acquisition module , which is used to obtain virtual motor production line information through the simulation module; the augmented reality module is used to generate augmented reality information based on the real image and the virtual motor production line information; the second sending module is used to send the augmented reality information Sending the data to the monitoring module, so that the monitoring module can perform augmented reality display on the motor production line based on the augmented reality information.

In the above implementation process, the augmented reality information is generated through the real image of the motor production line and the virtual motor production line information, so that the virtual display and enhanced display of the motor production line can be simultaneously displayed, which improves the comprehensiveness of monitoring.

Optionally, the augmented reality module is specifically configured to: acquire virtual coordinate data of the motor production line in the virtual motor production line information; determine a virtual camera posture based on the virtual coordinate data, where the virtual camera posture includes a position. and attitude data; determine the real camera attitude of the real camera based on the real image; based on the real camera attitude and the virtual camera attitude, compare the real image with the virtual object in the virtual motor production line information Fusion to obtain the augmented reality information.

In the above implementation process, based on the matching of the virtual camera pose and the real camera pose in the virtual motor production line information, the virtual object and the real image are fused to complete the augmented reality display of the motor production line, which improves the image of the augmented reality display and the motor production. line matching.

An embodiment of the present application further provides an electronic device, the electronic device includes a memory and a processor, wherein program instructions are stored in the memory, and when the processor reads and executes the program instructions, any one of the above implementations is executed steps in the method.

Embodiments of the present application further provide a readable storage medium, where computer program instructions are stored in the readable storage medium, and when the computer program instructions are read and run by a processor, any one of the foregoing implementations is executed steps in .

Description of drawings

In order to explain the technical solutions of the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings that need to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, therefore It should not be regarded as a limitation of the scope. For those of ordinary skill in the art, other related drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is a schematic flowchart of an augmented reality monitoring method for a motor production line applied to a simulation module provided by the present application.

FIG. 2 is a schematic flowchart of steps for constructing a digital twin model according to an embodiment of the present application.

FIG. 3 is a schematic flowchart of an augmented reality monitoring method for a motor production line applied to a virtual-real fusion module according to an embodiment of the present application.

FIG. 4 is a schematic flowchart of a step of generating augmented reality information according to an embodiment of the present application.

FIG. 5 is a schematic block diagram of an augmented reality monitoring system for a motor production line according to an embodiment of the present application.

FIG. 6 is a schematic block diagram of an augmented reality monitoring device for a motor production line applied to a simulation module according to an embodiment of the present application.

FIG. 7 is a schematic block diagram of an augmented reality monitoring device for a motor production line applied to a virtual room fusion module according to an embodiment of the present application.

Icons: 30-Augmented reality monitoring system for motor production line; 31-Real-time data acquisition module; 32-Real-time image acquisition module; 33-Simulation module; 34-Virtual-real fusion module; 35-Visual monitoring module; 40-Augmented reality for motor production line Monitoring device; 41-model establishment module; 42-real-time data acquisition module; 43-virtual reality module; 44-first sending module; 50-motor production line augmented reality monitoring device; 51-real image acquisition module; 52-virtual information acquiring module; 53-augmented reality module; 54-second sending module.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.

An embodiment of the present application provides an augmented reality monitoring method for a motor production line applied to a simulation module. Please refer to FIG. 1. FIG. 1 is a schematic flowchart of an augmented reality monitoring method for a motor production line applied to a simulation module provided by the present application. , the specific steps of the method can be as follows:

Step S12: Establish a digital twin model of the motor production line.

Digital twin is a multi-disciplinary, multi-physical, multi-scale, multi-probability simulation process that makes full use of physical models, sensor updates, and operating history to complete the mapping in virtual space, thereby reflecting the full life cycle of the corresponding physical equipment. process. A digital twin is a concept beyond reality that can be viewed as a digital mapping system of one or more important, interdependent equipment systems.

Specifically, the digital twin model refers to presenting physical objects in a virtual space in a digital manner, that is, creating a virtual model for physical objects in a digital manner to simulate their behavioral characteristics in the real environment. An integrated system of data, models and analysis tools. For the motor production line, it can integrate the manufacturing process in production and realize the digitalization of the whole process from process planning, production planning, and manufacturing execution.

Please refer to FIG. 2. FIG. 2 is a schematic flowchart of a digital twin model construction step provided by an embodiment of the present application. The digital twin model construction step may be specifically as follows:

Step S122: Obtain a real image of the motor production line through a real camera.

Optionally, the real image of the motor production line in this embodiment can be captured by oblique photography technology, which acquires rich top and side surfaces of objects by synchronously capturing images from one vertical, four oblique, and five different viewing angles. View high-resolution textures. It can not only truly reflect the situation of the object and obtain the texture information of the object with high precision, but also help to generate a real 3D model through positioning, fusion, modeling and other technologies.

Step S124 : extracting target features from the real image to obtain a feature image.

Target feature extraction refers to the operation of separating the target of interest from the background in a single image or a sequence of images, identifying and interpreting meaningful object entities from the image and extracting different image features, which can be applied to feature points and lines. extraction for image matching and 3D modeling.

Since the digital twin model in this embodiment needs to be constructed by a three-dimensional model, the target feature extraction method may be a three-dimensional point cloud target extraction.

Step S126: Establish a digital twin model based on the feature image.

Optionally, the digital twin model in this embodiment may include sub-models of each device of the motor production line, and the devices of the motor production line may include a six-axis robot, a paper inserting machine, a winding and embedding machine, a binding machine, and a conveying guide rail. At least one of the line equipments may also be other electrical machinery manufacturing related equipment.

The specific establishment process of the digital twin model can include: obtaining and according to the design requirements of the motor production line, based on the feature images of the six-axis robot, the paper inserting machine, the winding and embedding machine, the binding machine and the conveying guide rail, etc. in the simulation system. Carry out modeling, establish the 3D graphic model of each physical embedding equipment and the digital model of the workshop; complete the action planning of each embedding equipment, complete the production line logistics and motion planning, compile motion and motion control scripts, and conduct offline simulation operations until If the offline operation is normal, the construction of the digital twin model is completed.

Step S14: Obtain real-time data of the motor production line through the real-time data acquisition module, where the real-time data includes production line information and product information.

Specifically, step S14 may include the following sub-steps:

Step S142: Collect the PLC real-time data of each device of the motor production line through the Modbus interface of the real-time data collection module.

Modbus is a serial communication protocol, which is an industry standard for communication protocols in the industrial field, and is now a commonly used connection method between industrial electronic devices. Therefore, in this embodiment, using the Modbus interface to receive real-time data from the PLC of the motor production line can improve the The applicability of its data collection.

Optionally, the PLC real-time data may include the angle of each joint of the robot, the control instructions of the robot gripper, the status information of the processing equipment, the control instructions of the kinematic structure of the processing equipment, and the like.

Step S144: Collect product information corresponding to the motor production line in the production process execution management system of the manufacturing enterprise through the ODBC interface of the real-time data collection module.

Open Database Connectivity (ODBC) is generated to solve data sharing between heterogeneous databases. A database access interface standard ODBC based on Windows environment provides a unified interface for heterogeneous database access, allowing applications to use SQL (SQL ( Structured Query Language (Structured Query Language) is a data access standard, which can access different database management data, so that the application program can directly manipulate the data in the database, and avoid changing with the change of the database. In this embodiment, the production of the manufacturing enterprise is collected through the ODBC interface. The product information in the Manufacturing Execution System (MES) can improve the efficiency and integrity of data collection.

Optionally, the above-mentioned product information may include work order number, material code, work order plan, and the like.

Step S16: Drive the operation of the digital twin model based on the real-time data, and obtain the information of the virtual motor production line.

Specifically, before driving the operation of the digital twin model based on real-time data, it is also necessary to convert the coordinate system of the real image to the virtual coordinate system of the digital twin model, so as to determine the position of the motor production line in the virtual coordinate system.

The virtual motor production line information can be the virtual motor production line data after the virtual motor production line is synchronized with the real motor production line. The specific synchronization method can include: the PLC system for establishing the digital twin model and the PLC system for the physical motor production line and the host computer. Real-time synchronization of real-time data of physical motor production lines, monitoring data of the host computer and virtual motor production lines in the digital twin model through the dual synchronization technology of downlink commands and uplink information , to obtain the digital twin model of the implementation synchronization of the motor production line.

Step S18: Send the virtual motor production line information to the virtual-real integration module and the monitoring module, so that the virtual-real integration module obtains augmented reality information based on the virtual motor production line information, and the monitoring module determines the motor production based on the virtual motor production line information and the augmented reality information. virtual reality (Virtual Reality, VR) display or augmented reality (Augmented Reality, AR) display.

It should be understood that before the virtual motor production line information is sent to the virtual-real fusion module, the virtual coordinate system needs to be transformed so that it corresponds to the real coordinate system of the physical motor production line, and the digital twin model is rendered. Match the digital twin model to the physical motor production line.

In conjunction with the above-mentioned augmented reality monitoring method for a motor production line applied to a simulation module, an embodiment of the present application further provides an augmented reality monitoring method for a motor production line applied to a virtual-real fusion module. Please refer to FIG. 3 , which is a schematic flowchart of an augmented reality monitoring method for a motor production line applied to a virtual-real fusion module provided by an embodiment of the present application. The specific steps of the method may be as follows:

Step S22: Collect a real image of the motor production line through a real camera.

Step S24: Obtain the virtual motor production line information through the simulation module.

Optionally, the acquisition method of the real image and virtual motor production line information in step S22 and step S24 is the same as that in the above-mentioned method for augmented reality monitoring of motor production line applied to the simulation module, and details are not repeated here.

Step S26: Generate augmented reality information based on the real image and virtual motor production line information.

Please refer to FIG. 4. FIG. 4 is a schematic flowchart of an augmented reality information generation step provided by an embodiment of the present application. The augmented reality information generation step may be specifically as follows:

Step S262: Acquire virtual coordinate data of the motor production line in the virtual motor production line information.

The virtual coordinate data can be obtained based on the above-mentioned virtual coordinate system.

Step S264: Determine a virtual camera pose based on the virtual coordinate data, where the virtual camera pose includes position and pose data.

Step S266: Determine the real camera pose of the real camera based on the real image.

Step S268: Based on the real camera posture and the virtual camera posture, the real image is fused with the virtual object in the virtual motor production line information to obtain augmented reality information.

Optionally, the virtual-real fusion technology in this embodiment may be performed based on textures or pixels, or may be other virtual-real fusion methods commonly used in the art.

Step S28 : Send the augmented reality information to the monitoring module, so that the monitoring module can perform augmented reality display on the motor production line based on the augmented reality information.

Based on the above-mentioned augmented reality monitoring method for a motor production line applied to the simulation module and the virtual-real integration module, this embodiment also provides an augmented reality monitoring system for a motor production line. Please refer to FIG. 5 , which is an embodiment of the present application. A block diagram of an augmented reality monitoring system for a motor production line.

The augmented reality monitoring system 30 for a motor production line includes a real-time data acquisition module 31 , a real-time image acquisition module 32 , a simulation module 33 , a virtual-real fusion module 34 and a visual monitoring module 35 .

The real-time data acquisition module 31 collects the PLC real-time data of each device of the motor production line in the PLC system through the Modbus interface, collects the product information in the MES system through the ODBC interface, and processes the above-mentioned different types of PLC real-time data and product information. It is output to the simulation module 33 through a unified output interface.

Optionally, a multi-thread connection is adopted between the real-time data collection module 31 and the equipment PLC, and the real-time data collection frequency is 50-60 Hz.

The real-time image acquisition module 32 is used to acquire the real image of the electrode production line, transmit the real image to the simulation module 33 to generate a digital twin model, and transmit the real image to the virtual-real fusion module 34 at the same time.

The simulation module 33 is used to generate a digital twin model based on real images and real-time data, and send the digital twin model to the virtual-real fusion module 34 or to the visual monitoring module 35 for virtual reality display.

The virtual-real fusion module 34 is configured to generate augmented reality information according to the real image and the digital twin model, and send the augmented reality information to the visual monitoring module 35 for augmented reality display.

The above-mentioned visual monitoring module 35 may include a server, a display screen, a VR helmet, and AR glasses.

When the visual monitoring module 35 receives the virtual reality information of the digital twin model through the server, it can display the virtual reality to the staff through the VR helmet or the display screen, and the staff can conduct roaming monitoring of the production line workshop. At the same time, it can also display the current order, the completion of the current process, the running status of the equipment, and the process processing information of each equipment motion mechanism to the staff.

When the visual monitoring module 35 receives the augmented reality information through the server, it can display the augmented reality to the staff through the AR glasses, and the staff can view the real-time movement position of the invisible parts in the current field of view or the installation parts in the hidden space through the AR glasses, Realize human-machine collaborative installation operations.

Optionally, in AR mode, workers can call the electronic workbook of the corresponding process through voice or gesture, and the current work process can be projected to the current installation location in AR mode to guide workers to complete the installation of the current process.

Therefore, the staff can visually monitor the processing process and logistics status of the motor production line through the AR or VR mode, which is convenient for the management and scheduling of the motor production line. At the same time, they can also guide the workers to complete the current operation process through the AR glasses to ensure product quality.

In order to cooperate with the above-mentioned augmented reality monitoring method for a motor production line applied to a simulation module, this embodiment further provides an augmented reality monitoring device 40 for a motor production line applied to a simulation module.

Please refer to FIG. 6 , which is a schematic block diagram of an augmented reality monitoring device for a motor production line applied to a simulation module according to an embodiment of the present application.

The augmented reality monitoring device 40 for the motor production line includes:

A model establishment module 41, used to establish a digital twin model of the motor production line;

The real-time data acquisition module 42 is used to acquire real-time data of the motor production line through the real-time data acquisition module, and the real-time data includes production line information and product information;

The virtual reality module 43 is used to drive the operation of the digital twin model based on the real-time data, and obtain the information of the virtual motor production line;

The first sending module 44 is used to send the virtual motor production line information to the virtual-real fusion module and the monitoring module, so that the virtual-real fusion module obtains augmented reality information based on the virtual motor production line information, and the monitoring module is based on the virtual motor production line information and augmented reality information. Reality information for virtual reality display or augmented reality display on the motor production line.

Optionally, the model building module 41 is specifically configured to: obtain a real image of the motor production line through a real camera; perform target feature extraction on the real image to obtain a feature image; and establish a digital twin model based on the feature image.

Optionally, the digital twin model includes sub-models of each equipment of the motor production line, and the equipment of the motor production line includes at least one of a six-axis robot, a paper inserting machine, a winding-inserting integrated machine, a binding machine, and a conveying guide rail.

Optionally, the real-time data acquisition module 42 is specifically used for: collecting the programmable logic controller PLC real-time data of each device of the motor production line through the Modbus interface of the real-time data acquisition module, the PLC real-time data is production line information, and the PLC real-time data includes: At least one of the angle of each joint of the robot, the control command of the robot gripper, the state information of the processing equipment, and the control command of the kinematic structure of the processing equipment; through the open database of the real-time data acquisition module, the ODBC interface is connected to collect the information related to the motor in the production process execution management system of the manufacturing enterprise. Product information corresponding to the production line, the product information includes at least one of the work order number, material code and work order plan.

In order to cooperate with the above-mentioned augmented reality monitoring method for a motor production line applied to the virtual-real fusion module, this embodiment further provides an augmented reality monitoring device 50 for a motor production line applied to the virtual-real fusion module.

Please refer to FIG. 7 , which is a schematic block diagram of an augmented reality monitoring device for a motor production line applied to a virtual room fusion module according to an embodiment of the present application.

The augmented reality monitoring device 50 for a motor production line includes:

The real image acquisition module 51 is used to collect real images of the motor production line through a real camera;

The virtual information acquisition module 52 is used for acquiring virtual motor production line information through the simulation module;

an augmented reality module 53, configured to generate augmented reality information based on real images and virtual motor production line information;

The second sending module 54 is configured to send the augmented reality information to the monitoring module, so that the monitoring module can perform an augmented reality display on the motor production line based on the augmented reality information.

Optionally, the augmented reality module 53 is specifically configured to: acquire virtual coordinate data of the motor production line in the virtual motor production line information; determine the virtual camera posture based on the virtual coordinate data, and the virtual camera posture includes position and posture data; determine the real image based on the real image. The real camera pose of the camera; based on the real camera pose and the virtual camera pose, the real image is fused with the virtual objects in the virtual motor production line information to obtain augmented reality information.

An embodiment of the present application further provides an electronic device, the electronic device includes a memory and a processor, where program instructions are stored in the memory, and when the processor reads and executes the program instructions, executes the program instructions provided in this embodiment. The steps in any one of the methods described in the augmented reality monitoring method for a motor production line.

It should be understood that the electronic device may be an electronic device with a logic computing function, such as a personal computer (Personal Computer, PC), a tablet computer, a smart phone, a personal digital assistant (Personal Digital Assistant, PDA).

Embodiments of the present application also provide a readable storage medium, where computer program instructions are stored in the readable storage medium, and when the computer program instructions are read and run by a processor, the augmented reality of the motor production line is executed. Steps in the monitoring method.

To sum up, the embodiments of the present application provide an augmented reality monitoring method, device, electronic device and storage medium for a motor production line, wherein the method applied to the simulation module includes: establishing a digital twin model of the motor production line; The acquisition module acquires real-time data of the motor production line, the real-time data includes production line information and product information; drives the digital twin model to operate based on the real-time data, and obtains virtual motor production line information; line information is sent to the virtual-real fusion module and the monitoring module, so that the virtual-real fusion module obtains augmented reality information based on the virtual motor production line information, and the monitoring module obtains augmented reality information based on the virtual motor production line information and the augmented reality information A virtual reality display or an augmented reality display is performed on the motor production line.

In the above implementation manner, the three-dimensional virtual reconstruction of the motor production line is completed through the virtual reconstruction technology to obtain a digital twin model, and the running status of the motor production line is digitized and visualized based on real-time data such as production line information and product information. The visual monitoring of the whole process and all parameters of the digital workshop of the motor production line is convenient for workshop management and scheduling, and ensures the production quality of products.

In the several embodiments provided in this application, it should be understood that the disclosed device may also be implemented in other manners. The apparatus embodiments described above are merely illustrative, for example, the block diagrams in the accompanying drawings show the architecture, functions and operations of possible implementations of the apparatus according to various embodiments of the present application. In this regard, each block in the block diagrams may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function(s) . It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams, and combinations in the block diagrams, can be implemented by special purpose hardware-based systems that perform the specified functions or actions, or by combinations of special purpose hardware and computer instructions.

In addition, each functional module in each embodiment of the present application may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

If the functions are implemented in the form of software function modules and sold or used as independent products, they may be stored in a computer-readable storage medium. Therefore, this embodiment also provides a readable storage medium with computer program instructions stored in it, and when the computer program instructions are read and executed by a processor, execute any one of the block data storage methods in the method. A step of. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, removable hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, RanDom Access Memory), magnetic disk or optical disk and other media that can store program codes .

The above descriptions are merely examples of the present application, and are not intended to limit the protection scope of the present application. For those skilled in the art, various modifications and changes may be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the protection scope of this application. It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application.

It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element defined by the phrase "comprises" does not preclude the presence of additional identical elements in a process, method, article, or device that includes the element.

Claims (10)

1. An augmented reality monitoring method for a motor production line is applied to a simulation module, and comprises the following steps:

establishing a digital twin model of a motor production line;

acquiring real-time data of the motor production line through a real-time data acquisition module, wherein the real-time data comprises production line information and product information;

driving the digital twin model to operate based on the real-time data to obtain virtual motor production line information;

will virtual motor produces line information transmission to virtual reality and fuses module and monitoring module, so that virtual reality fuses the module based on virtual motor produces line information and obtains augmented reality information, and monitoring module based on virtual motor produces line information with augmented reality information is right the motor is produced and is carried out virtual reality and show or augmented reality and show.

2. The method of claim 1, wherein the establishing a digital twin model of a motor production line comprises:

acquiring a real image of the motor production line through a real camera;

extracting target features of the real image to obtain a feature image;

and establishing the digital twin model based on the characteristic image.

3. The method of claim 1, wherein the digital twin model comprises sub-models of individual devices of the motor production line including at least one of a six-axis robot, a paper inserter, a winding and embedding machine, a binding machine, and a conveyor rail.

4. The method of claim 3, wherein the obtaining real-time data of the motor production line through a real-time data acquisition module comprises:

collecting PLC (programmable logic controller) real-time data of each device of the motor production line through a Modbus interface of the real-time data collection module, wherein the PLC real-time data are production line information, and the PLC real-time data comprise at least one of joint angles of a robot, robot gripper control instructions, machining device state information and control instructions of a machining device motion structure;

and connecting an ODBC interface through an open database of the real-time data acquisition module to acquire the product information corresponding to the motor production line in a production process execution management system of a manufacturing enterprise, wherein the production information comprises at least one of a work order number, a material code and a work order plan.

5. An augmented reality monitoring method for a motor production line is applied to a virtual-real fusion module, and comprises the following steps:

acquiring a real image of a motor production line through a real camera;

acquiring virtual motor production line information through a simulation module;

generating augmented reality information based on the real image and the virtual motor production line information;

and sending the augmented reality information to a monitoring module so that the monitoring module can display augmented reality based on the augmented reality information on the motor production line.

6. The method of claim 5, wherein generating augmented reality information based on the real image and the virtual machine line information comprises:

acquiring virtual coordinate data of the motor production line in the virtual motor production line information;

determining a virtual camera pose based on the virtual coordinate data, the virtual camera pose comprising position and pose data;

determining a real camera pose of the real camera based on the real image;

and fusing the real image and the virtual object in the virtual motor production line information based on the real camera gesture and the virtual camera gesture to obtain the augmented reality information.

7. The utility model provides an augmented reality monitoring device is produced to motor line which characterized in that is applied to simulation module, the device includes:

the model establishing module is used for establishing a digital twin model of the motor production line;

the real-time data acquisition module is used for acquiring real-time data of the motor production line through the real-time data acquisition module, wherein the real-time data comprises production line information and product information;

the virtual reality module is used for driving the digital twin model to operate based on the real-time data to obtain the virtual motor production line information;

the first sending module is used for sending the virtual motor production line information to the virtual-real fusion module and the monitoring module, so that the virtual-real fusion module obtains the augmented reality information based on the virtual motor production line information, and the monitoring module obtains the augmented reality information based on the virtual motor production line information and the augmented reality information is right the motor production line carries out virtual reality display or augmented reality display.

8. The utility model provides an augmented reality monitoring device is produced to line to motor which characterized in that is applied to virtual reality and fuses the module, the device includes:

the real image acquisition module is used for acquiring a real image of the motor production line through a real camera;

the virtual information acquisition module is used for acquiring the virtual motor production line information through the simulation module;

the augmented reality module is used for generating augmented reality information based on the real image and the virtual motor production line information;

and the second sending module is used for sending the augmented reality information to the monitoring module so that the monitoring module can perform augmented reality display on the motor production line based on the augmented reality information.

9. An electronic device comprising a memory having stored therein program instructions and a processor that, when executed, performs the steps of the method of any of claims 1-6.

10. A readable storage medium having stored thereon computer program instructions for executing the steps of the method according to any one of claims 1 to 6 when executed by a processor.

Images