CN118262071A - Engineering machinery mixed reality digital twin management and control method and system - Google Patents

Abstract

The invention discloses a method and a system for controlling a digital twin of a mixed reality of engineering machinery, which belong to the field of digital twin, and realize the bidirectional digital twin control of the engineering machinery by carrying out real-time dynamic sensing on multi-source heterogeneous data in the working environment of the engineering machinery, generating virtual-real data by fusing the mixed reality working environment and setting interactive tasks; the virtual-real data fusion mixed reality operation environment generation is to construct a color point cloud map by utilizing a server to acquire color point clouds, combine with a pre-constructed three-dimensional model based on GPS positioning data, acquired engineering machinery motion tracks and engineering machinery local IMU data, realize the fusion registration of virtual-real data of the engineering machinery, reconstruct an operation scene with higher virtual-real consistency in three-dimensional mode, analyze the material state and the equipment motion state based on semantic information, and perform danger early warning in combination with a collision detection technology in the virtual scene. The invention optimizes the engineering machinery operation plan, improves the resource utilization rate and reduces the operation cost.

Description

A mixed reality digital twin control method and system for engineering machinery

Technical Field

The present invention relates to the field of digital twin technology, and in particular to a mixed reality digital twin control method and system for engineering machinery.

Background technique

Engineering machinery (including gantry cranes, ship loaders, stackers, excavators, loaders, etc.) is an important equipment for loading and unloading bulk goods (such as coal, ore, grain, etc.). Efficient and intelligent loading and unloading equipment is the key to building smart bulk ports and realizing intelligent construction. However, existing engineering machinery mainly uses monitoring videos to transmit real-time images of the site to perceive the loading and unloading environment, and remotely controls the position of the whole machine and the grab/bucket to complete the loading and unloading actions. The whole process requires a high level of driver skills and a high degree of concentration. During the remote control operation, due to the lack of three-dimensional information of the working environment by fisheye or ordinary cameras, the scene on site has problems such as shrinkage, deformation and serious lack of distance sense, which makes it difficult for the driver to accurately judge the three-dimensional spatial position of the operating target and obstacles, and the fine operation involved cannot be completed accurately. Although some engineering machinery systems integrate lidar for environmental modeling, the point cloud data processing is difficult to perform in real time, resulting in the driver still relying on video during the operation.

With the continuous development of mixed reality technology (MR), researchers have studied this technology and applied it to multiple fields. For example, the Chinese invention patent with publication number CN110322553A provides a method and system for implementing layout of mixed reality scenes with lidar point clouds, but this method is purely based on matching with laser point clouds, and is only applicable to local area positioning and tracking. It has limitations for positioning of large-scale scenes and has high requirements for the environment. The invention patent with publication number WO2021138940A1 provides a remote virtual-real high-precision matching and positioning method for augmented reality and mixed reality, which solves the technical problem of inaccurate matching and positioning in existing methods, but the matching method is based on marker matching, depends on external markers, is only applicable to specific scenes or application environments, has a limited scope of application, and is more sensitive to environmental changes. The Chinese invention patent with publication number CN117151417A provides a port operation management method, system and storage medium based on digital twins. By integrating digital twin technology, the port's operation data can be visualized so that the operation data can be presented to users more intuitively. However, its effectiveness is highly dependent on accurate and timely data input, such as cargo loading information and transport vehicle resource data. Any data errors or delays may affect the performance of the system and the accuracy of decision-making.

In view of this, it is necessary to develop a mixed reality digital twin control method and system for engineering machinery to overcome the above problems.

Summary of the invention

The technical problem to be solved by the present invention is to provide a mixed reality digital twin control method and system for engineering machinery, which combines the functions of three-dimensional material modeling, dynamic perception, mixed reality virtual-reality superposition, and online learning and optimization. It does not rely on data input and can achieve a higher level of operation control and improved resource utilization.

In order to solve the above technical problems, the technical solution adopted by the present invention is:

A mixed reality digital twin control method for construction machinery, which realizes bidirectional digital twin control of construction machinery through real-time dynamic perception of multi-source heterogeneous data in the construction machinery operation environment, generation of mixed reality operation environment with virtual and real data fusion, and interactive task setting;

The virtual-real data fusion mixed reality working environment generation is to use the server to construct a color point cloud map with the collected color point cloud, and combine it with the pre-constructed three-dimensional model of engineering machinery and the three-dimensional model of ancillary facilities based on GPS positioning data, the acquired motion trajectory of engineering machinery and the local IMU data of engineering machinery to achieve the virtual-real data fusion and registration of engineering machinery, as well as the three-dimensional reconstruction of the working scene with high virtual-real consistency, and further analyze the material status and equipment motion status based on semantic information, and at the same time combine the collision detection technology in the virtual scene to carry out equipment-equipment, equipment-personnel, and equipment-surrounding objects danger warning.

A further improvement of the technical solution of the present invention is that the real-time dynamic perception of multi-source heterogeneous data in the working environment specifically includes: by setting up a visual-lidar fusion perception system on the construction machinery, the three-dimensional color point cloud of the working scene and the motion trajectory of the construction machinery are obtained in real time, and then the three-dimensional color point cloud is analyzed to further obtain the semantic point cloud and three-dimensional bounding box information of obstacles, personnel and surrounding scenes, and the motion data of various parts of the construction machinery are jointly obtained through the controller and IMU, and the global position information of the construction machinery is obtained through GPS, and stored in the database in real time.

A further improvement of the technical solution of the present invention lies in that: the analysis of the three-dimensional color point cloud specifically refers to preprocessing, semantically segmenting and triangulating the collected color laser point cloud data by utilizing the point cloud semantic segmentation network, further obtaining semantic point clouds and three-dimensional bounding box information of obstacles, personnel and surrounding scenes, and realizing three-dimensional modeling and shape analysis of objects, so as to more accurately identify and detect the position, form and characteristics of materials; wherein the point cloud data preprocessing includes point cloud downsampling, point cloud denoising and point cloud resampling.

A further improvement of the technical solution of the present invention is that the color point cloud map is constructed by using different processing methods for dynamic and static point cloud data; wherein, for static areas, when the 3D point cloud is dense enough, the update of the area will be stopped; for dynamic areas, incremental detection is performed, and the changed point cloud data is updated in real time, thereby constructing a real-time color point cloud map.

The further improvement of the technical solution of the present invention is that based on the GPS positioning data, the acquired engineering machinery motion trajectory and the local IMU data of the engineering machinery, the specific model is as follows:

Among them, the construction machinery center in the virtual scene is set to coincide with the GPS installation position in the actual scene. Indicates the coordinates of the center of the construction machinery in the virtual scene coordinate system; P _gps (x, y, z) = (N + H) * cosB * cosL, (N + H) * cosB * sinL, [N * (1-e ² ) + H * sinB) indicates the representation of GPS coordinates converted into plane coordinates, B is longitude, N is latitude, and H is geodetic height; Indicates the coordinates of the construction machinery at the current moment, which are calculated based on the acquired motion trajectory of the construction machinery; RT indicates the rotation and translation matrix of the coordinates of the starting position of the construction machinery relative to the GPS installation position; Indicates the rotation angle of the center of the construction machinery around x, y, and z in the virtual scene; Indicates the rotation angle of the construction machinery around x, y, and z in the actual scene at the current moment; Indicates the rotation angles of the key parts of the engineering machinery in the virtual scene around x, y, and z. It represents the rotation angles around x, y, and z obtained by the IMU installed on the key parts of the engineering machinery in the actual scene corresponding to the virtual scene.

A further improvement of the technical solution of the present invention is that the virtual and real data fusion alignment of the engineering machinery is achieved by aligning the rough positioning coordinates of the large machine obtained by GPS, the IMU information of key components, and the engineering machinery's own motion trajectory information and the local color 3D point cloud of the engineering machinery output by the visual-lidar SLAM, and is achieved by fusion matching of the 3D point cloud and similar feature points of the engineering machinery.

A further improvement of the technical solution of the present invention is that the interactive task setting specifically includes: setting the work task objectives for the generated mixed reality work scene according to the task requirements through gesture and voice interaction, and dynamically displaying and prompting the virtual-reality fusion scene.

A further improvement of the technical solution of the present invention lies in that the bidirectional digital twin management and control specifically includes: on the one hand, realizing the scene perception function through real-time online monitoring of bulk materials, the operating status of the construction machinery itself, the surrounding environment and the changing status of materials; on the other hand, based on the generated mixed reality map, online learning, optimization and adjustment of the pre-trained construction machinery operation trajectory are carried out in the mixed reality environment, as well as experimental verification and safety testing of the construction machinery loading and unloading operations, and the optimal trajectory is generated for specific models and specific tasks, thereby realizing online digital twin control of construction machinery operations.

A mixed reality digital twin control system for construction machinery, including a dynamic real-time perception module for port scenes, a mixed reality scene fusion matching module, and a human-machine collaborative interaction and control module;

The dynamic real-time perception module of the port scene is used to obtain the color three-dimensional point cloud data of the motion trajectory and working environment of the engineering machinery by fusing and matching the monocular camera sensor and the laser radar sensor; obtain the semantic information of materials, people, equipment, tracks, ground, ships, and vehicles, as well as the minimum bounding box of the semantic point cloud by calling the semantic segmentation network on the point cloud data; obtain the motion information by installing IMU sensors at the key parts of the lifting, rotation, and amplitude change of the engineering machinery, and obtain the location information of the large machine by installing GPS on the top of the engineering machinery cab;

The mixed reality scene fusion matching module is used to fuse the pre-established construction machinery virtual model, the auxiliary facility model and the collected three-dimensional color point cloud according to GPS data, IMU and motion trajectory to establish more accurate equipment location information; then the location information and the object minimum bounding box information obtained by positioning and semantic matching are converted into a coordinate system so as to align them with the coordinate system of the construction machinery virtual model and the ground auxiliary facilities established in Unity; according to the converted location information and object information, the virtual construction machinery model is displayed in the Unity3D software and superimposed on the real scene;

The human-machine collaborative interaction and control module is used to use voice and gesture recognition technology to control the task setting and motion trajectory recording of construction machinery in a mixed reality digital twin environment, and to achieve trajectory optimization and prediction, and to send control commands to the construction machinery in real time in combination with the Unity end to realize the motion control of the actual construction machinery.

A further improvement of the technical solution of the present invention is that the engineering machinery at least includes a portal crane, a ship loader, a stacker-reclaimer, an excavator and a loader in the port equipment.

Due to the adoption of the above technical solution, the technical progress achieved by the present invention is:

1. The present invention is based on color laser point cloud and combines the functions of three-dimensional material modeling, dynamic perception, mixed reality virtual-real superposition, and online learning and optimization. It can improve the operating efficiency and safety of construction machinery, and can analyze, predict motion and simulate data in a mixed reality digital twin environment to optimize the operation plan of construction machinery, improve resource utilization and reduce operating costs.

2. The present invention introduces mixed reality technology and digital twin technology into the unmanned operation of construction machinery. Through the real-time fusion of the real-time collected three-dimensional color laser point cloud, the three-dimensional model of the construction machinery and the model of the ancillary facilities, it can realize the real-time three-dimensional material modeling, dynamic environmental perception and monitoring of the on-site working environment; and through interactive task setting, the pre-trained construction machinery operation trajectory can be learned, optimized and adjusted online in the mixed reality environment, as well as the experimental verification and safety testing of the construction machinery loading and unloading operations, and the optimal trajectory can be generated for specific models and specific tasks, thereby realizing the operation control of the actual construction machinery.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

FIG1 is an overall block diagram of a mixed reality digital twin control system for construction machinery provided in an embodiment of the present invention;

Figure 2 is a flow chart of the engineering machinery mixed reality digital twin control method provided in an embodiment of the present invention.

Detailed ways

It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or apparatuses.

The present invention is further described in detail below with reference to the accompanying drawings and embodiments:

The engineering machinery in the present invention includes portal cranes, ship loaders, stackers, excavators, loaders, etc. in port equipment. The portal crane is taken as an example for description below.

The present invention adopts livox solid-state laser radar, monocular camera and IMU (Inertial measurement unit, referred to as IMU, is a device for measuring the three-axis attitude angle and acceleration of an object) to collect color point clouds of engineering machinery operation scenes according to the method of autonomously moving to construct three-dimensional true color point cloud maps in "a robot system and method for autonomously moving to construct three-dimensional true color point cloud maps" proposed in the Chinese invention patent with publication number CN117367441A. The livox solid-state laser radar, monocular camera and IMU are set up on the boom near the driver's cab of the portal crane. The portal crane moves at a uniform speed along the track direction to collect the three-dimensional color point cloud of the operation scene and the motion trajectory of the portal crane.

As shown in Figure 1, a mixed reality digital twin management and control system for construction machinery includes a dynamic real-time perception module of the port scene, a mixed reality scene fusion and matching module, and a human-machine collaborative interaction and control module.

The dynamic real-time perception module of the port scene obtains the motion trajectory of the gantry crane by fusing and matching the monocular camera sensor and the lidar sensor. And color 3D point cloud data of the working environment:

By calling the PointNet++ semantic segmentation network on the point cloud data, we can obtain some semantic information of materials, people, equipment, tracks, ground, ships, vehicles, etc.

And the minimum bounding box of the semantic point cloud

By installing IMU sensors at the key parts of the portal crane for lifting, rotation and luffing, motion information can be obtained. Install GPS on the top of the gantry crane cab to obtain the location information P _gps (x, y, z) of the crane.

The mixed reality scene fusion matching module pre-builds the virtual model of the portal crane, the auxiliary facilities model and the collected 3D color point cloud based on GPS data, IMU and motion trajectory The fusion is performed to establish more accurate equipment location information. The location information and object minimum bounding box information obtained by positioning and semantic matching are then converted into coordinate systems to align them with the coordinate systems of the portal crane virtual model and ground auxiliary facilities established in Unity. Based on the converted location information and object information, the virtual portal crane model is displayed in the Unity3D software and superimposed on the real scene.

Based on the joint configuration of GPS, engineering machinery motion trajectory and local IMU data, the specific model is as follows:

In the formula, the center of the construction machinery in the virtual scene is assumed to coincide with the GPS installation position in the actual scene. represents the coordinates of the center of the construction machinery in the virtual scene coordinate system, P _gps (x, y, z) = (N + H) * cosB * cosL, (N + H) * cosB * sinL, [N * (1-e ² ) + H * sinB) represents the GPS coordinates converted into plane coordinates (B is longitude, N is latitude, H is geodetic height), Represents the coordinates of the construction machinery at the current moment, which are calculated from the acquired construction machinery motion trajectory. RT represents the rotation and translation matrix of the coordinates of the starting position of the construction machinery relative to the GPS installation position. Here, Indicates the rotation angle of the center of the construction machinery around x, y, and z in the virtual scene; Indicates the rotation angle of the construction machinery around x, y, and z in the actual scene at the current moment; Indicates the rotation angles of the key parts of the engineering machinery in the virtual scene around x, y, and z. It represents the rotation angles around x, y, and z obtained by the IMU installed on the key parts of the engineering machinery in the actual scene corresponding to the virtual scene.

Among them, the Unity3D software displays the scene using TCP/IP protocol, transmits the data obtained by the perception layer to the MySQL database, and enables the server layer to read and process the MySQL database data. The color point cloud data of the bulk carrier is processed, including pre-processing such as point cloud downsampling and point cloud noise reduction based on the PCL library, and then the pre-processed point cloud is meshed; the control signal of the port machinery is read to process the signal and perform operation planning; the surrounding environment information and personnel location information are processed for danger warning, fault diagnosis, collision detection and personnel positioning, etc., to build an XR environment.

Build an XR environment, including 3D point cloud visualization of materials, color point cloud map display, 3D equipment model construction, operation trajectory tracking, personnel model construction and fusion display. Use the Unity3D engine to build a service layer, including service models and user interfaces. The service model includes historical playback, material measurement, equipment monitoring, throughput estimation, operation task planning and measurement range delineation. There are many user interfaces, including web browser interface, client GUI interface, and mobile application client interface.

The human-machine collaborative interaction and control module uses voice and gesture recognition technology to control the task setting and motion trajectory recording of the portal crane in the mixed reality digital twin environment, and realizes trajectory optimization and prediction, and combines with the Unity end to send control commands to the portal crane in real time to realize the motion control of the actual machine. The specific steps are as follows:

(1) The mixed reality digital twin client obtains human gesture input through the camera, and uses the gesture recognition algorithm to analyze and recognize the gesture.

(2) Recording through a microphone, using Baidu speech recognition API to convert the recording into text. The recognition results can include control commands or keywords.

(3) Based on the results of gesture recognition and voice recognition, the corresponding control commands are parsed, such as "move to the first hatch", "stop operation", "rise", "fall", "rotate", etc., and the corresponding control commands are sent to the mixed reality digital twin end.

The specific control process of the control module is:

(1) On the remote operation end, by recording the operation task, operation target and color point cloud of the operation environment, the operation trajectory of the portal crane under the specific task is obtained, and the operation trajectory library under the specific task is established;

(2) By optimizing and fine-tuning the operation trajectory, the optimal operation trajectory for a specific task is predicted in a mixed reality environment. A kinematic model of a gantry crane is established in a mixed reality environment, and experimental verification and safety testing are performed to ultimately select the optimal motion trajectory.

(3) Based on the optimal motion trajectory, specific task control is achieved through communication between the Unity mixed reality digital twin system and the gantry crane PLC control system.

As shown in Figure 2, it is a flow chart of a mixed reality digital twin control method for engineering machinery provided by the present invention. By processing the perception data, a mixed reality virtual scene is established, and automatic operation task generation is realized based on human-computer interaction. The rationality and safety of the operation task are guaranteed by the three-dimensional motion simulation prediction of the engineering machinery in the mixed reality environment. If the safety standards are met, the PLC control signal is used to publish the optimal motion trajectory to realize the actual control operation. If there are safety hazards or it does not meet the requirements, the task is re-planned, or remote manual operation is adopted.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. The method is characterized in that bidirectional digital twin management and control of the engineering machinery is realized by carrying out real-time dynamic sensing on multi-source heterogeneous data in the engineering machinery working environment, generating virtual-real data by fusing the mixed reality working environment and setting interactive tasks;

the virtual-real data fusion mixed reality operation environment is generated by utilizing a server to construct a color point cloud map of an acquired color point cloud, combining the color point cloud with a pre-constructed engineering machine three-dimensional model and an auxiliary facility three-dimensional model, realizing fusion registration of virtual-real data of the engineering machine based on GPS positioning data, acquired engineering machine motion tracks and local IMU data of the engineering machine, further analyzing a material state and a device motion state based on semantic information, and simultaneously carrying out danger early warning on devices, personnel and surrounding objects by combining collision detection technology in the virtual scene.

2. The method for controlling and managing mixed reality digital twinning of engineering machinery according to claim 1, wherein the method for dynamically sensing multi-source heterogeneous data in real time in the working environment specifically comprises: the visual-laser radar fusion sensing system is erected on the engineering machinery, the three-dimensional color point cloud of the operation scene and the motion trail of the engineering machinery are obtained in real time, then the three-dimensional color point cloud is analyzed, the semantic point cloud and the three-dimensional bounding box information of obstacles, personnel and surrounding scenes are further obtained, the motion data of all parts of the engineering machinery are obtained through the combination of the controller and the IMU, the global position information of the engineering machinery is obtained through the GPS, and the global position information is stored in the database in real time.

3. The method for managing and controlling the mixed reality digital twin of the engineering machinery according to claim 2, wherein the analysis of the three-dimensional color point cloud is specifically that the preprocessing, the semantic segmentation and the triangulation of the collected color laser point cloud data are performed by utilizing a point cloud semantic segmentation network, so that the semantic point cloud and the three-dimensional bounding box information of obstacles, personnel and surrounding scenes are further obtained, the three-dimensional modeling and the shape analysis of objects are realized, and the positions, the shapes and the characteristics of materials are more accurately identified and detected; the point cloud data preprocessing comprises point cloud downsampling, point cloud denoising and point cloud resampling.

4. The method for managing and controlling the mixed reality digital twinning of the engineering machinery according to claim 1, wherein the color point cloud map is constructed by adopting different processing methods for dynamic and static point cloud data; wherein, for a static region, when the 3D point cloud is sufficiently dense, updating of the region will be stopped; and performing increment detection on the dynamic region, and updating the changed point cloud data in real time, so as to construct a real-time color point cloud map.

5. The method for controlling the mixed reality digital twin of the engineering machinery according to claim 1, wherein the method is characterized in that the method is based on GPS positioning data, acquired motion tracks of the engineering machinery and local IMU data of the engineering machinery, and a specific model is as follows:

wherein, the center of the engineering machine in the virtual scene is set to coincide with the GPS installation position in the actual scene, Representing the coordinates of the engineering machinery center under a virtual scene coordinate system; p _gps(x,y,z)＝(N+H)*cosB*cosL,(N+H)*cosB*sinL,[N*(1-e²) +h sinB) represents a representation after the GPS coordinates are converted into plane coordinates, B is longitude, N is latitude, H is geodetic altitude; representing the coordinates of the engineering machinery at the current moment, wherein the coordinates are calculated by the acquired motion trail of the engineering machinery; RT represents a rotation translation matrix of coordinates of a movement starting position of the engineering machinery relative to a GPS mounting position; (h, p, r) represents a rotation angle around x, around y and around z of the center of the engineering machinery in the virtual scene; representing the rotation angle of the engineering machinery around x, around y and around z in an actual scene at the current moment; representing the rotation angle of the key parts of the engineering machinery around x, around y and around z in the virtual scene, And the rotation angles around x, around y and around z, which are obtained by the IMU installed at the key part of the engineering machine corresponding to the virtual scene in the actual scene, are represented.

6. The method for managing and controlling the hybrid reality digital twin of the engineering machinery according to claim 1, wherein the fusion registration of virtual and real data of the engineering machinery is realized by aligning the rough positioning coordinates of a large machine obtained by GPS, the IMU information of a key part, the motion track information of the engineering machinery and the local color 3D point cloud of the engineering machinery output by a vision-laser radar SLAM and fusion matching of the 3D point cloud and similar characteristic points of the engineering machinery.

7. The method for controlling and managing mixed reality digital twinning of engineering machinery according to claim 1, wherein the interactive task setting specifically comprises: setting a task target of the generated mixed reality operation scene according to task requirements through gesture and voice interaction, and carrying out dynamic information fusion display and prompt on the virtual-real fusion scene.

8. The method for controlling the mixed reality digital twin of the engineering machinery according to claim 1, which is characterized by specifically comprising the following steps: on one hand, the scene sensing function is realized by real-time on-line monitoring of bulk materials, the running state of engineering machinery and the change state of surrounding environment and materials; on the other hand, according to the generated mixed reality map, on-line learning, optimization and adjustment are carried out on the operation track of the pre-trained engineering machinery in the mixed reality environment, the experimental verification and the safety test of the loading and unloading operation of the engineering machinery are carried out, and the optimal track is generated aiming at a specific model and a specific task, so that the on-line digital twin control of the operation of the engineering machinery is realized.

9. The system for managing and controlling the mixed reality digital twin of the engineering machinery is characterized in that the system for managing and controlling the mixed reality digital twin of the engineering machinery used by the method for managing and controlling the mixed reality digital twin of the engineering machinery according to any one of claims 1-8 comprises a dynamic real-time sensing module of a harbor scene, a mixed reality scene fusion matching module and a man-machine cooperative interaction and control module;

The dynamic real-time sensing module of the port scene is used for obtaining color three-dimensional point cloud data of the engineering machinery motion trail and the operation environment by carrying out fusion, matching and display on the monocular camera sensor and the laser radar sensor; invoking semantic segmentation network through point cloud data to obtain semantic information of parts of materials, people, equipment, tracks, ground, ships and vehicles and a minimum bounding box of semantic point cloud; acquiring motion information by installing an IMU sensor at a lifting, rotating and luffing key part of the engineering machinery, and acquiring position information of a mainframe by installing a GPS at the top of a cab of the engineering machinery;

The mixed reality scene fusion matching module is used for fusing a pre-established engineering machinery virtual model, an accessory facility model and an acquired three-dimensional color point cloud according to GPS data, an IMU and a motion trail so as to establish more accurate equipment position information; then, converting the coordinate system of the position information and the minimum bounding box information of the object obtained by positioning and semantic matching to align the coordinate system of the engineering machinery virtual model and ground auxiliary facilities built in the unity; displaying the virtual engineering machine model in the Unity3D software according to the converted position information and object information, and superposing and displaying the virtual engineering machine model in a real scene;

The man-machine cooperative interaction and control module is used for controlling task setting and motion track recording of the engineering machinery in the mixed reality digital twin environment by using a voice and gesture recognition technology, realizing track optimization and prediction, and realizing motion control of the actual engineering machinery by combining a Unity end to send a control command to the engineering machinery in real time.

10. The system of claim 9, wherein the work machine comprises at least a gantry crane, a ship loader, a stacker-reclaimer, an excavator, and a loader in a harbour site.