CN120631189B

CN120631189B - Virtual reality interaction and content generation method and system based on large model

Info

Publication number: CN120631189B
Application number: CN202511128561.4A
Authority: CN
Inventors: 杨影; 柴作鸿; 洪福星
Original assignee: Beijing Luster LightTech Co Ltd
Current assignee: Beijing Luster LightTech Co Ltd
Priority date: 2025-08-13
Filing date: 2025-08-13
Publication date: 2025-10-28
Anticipated expiration: 2045-08-13
Also published as: CN120631189A

Abstract

The present application provides a method and system for virtual reality interaction and content generation based on a large model. Among them, the present application collects raw point cloud data and aligns it with semantic labels to construct a topological map containing object positions, categories, and spatial constraints; dynamically calibrates the point cloud and maps it to a unified coordinate system, extracts occlusion boundaries to generate a disparity mapping table; uses gesture recognition to identify the blank area defined by the user, and generates a physically compliant new object parameter candidate set in combination with the topological map; uses the large model to filter parameters, and retains physically valid parameters after collision response, gravity, and lighting verification; generates virtual object geometry and texture in the occluded area based on the incremental neural radiation field, optimizes depth and rendering accuracy in combination with the mapping table, outputs it to the interactive interface, and updates the topological parameters. The present application realizes the dynamic generation of physically consistent and rendering-optimized new objects based on user interaction in virtual reality scenes, enhancing realism and interactive experience.

Description

Virtual reality interaction and content generation method and system based on large model

Technical Field

The application relates to the technical field of dynamic scene interactive content generation, in particular to a virtual reality interaction and content generation method and system based on a large model.

Background

In virtual reality applications, users are highly required to generate virtual objects conforming to the environment semantics in dynamic scenes through gestures. The requirement requires that the system has millisecond response capability, ensures that a new object strictly follows the physical rule and semantic logic of a scene, and simultaneously realizes high-reality rendering, so that the geometric details, texture illumination and the original scene of the generated object are seamlessly fused.

The current mainstream scheme adopts a technology of combining a generated nerve radiation field with scene prior extraction, namely analyzing scene point clouds through a pre-trained 3D segmentation network, extracting object types and spatial layout, generating a candidate object model by the generated nerve radiation field based on semantic tags after a blank area is defined by a gesture of a user, carrying out static collision verification by utilizing a lightweight physical engine, and finally rendering the object to a target position.

However, this solution has significant drawbacks. Firstly, physical verification only aims at static collision, ignores the violations of model penetration, suspension and the like caused by interaction of dynamic entities and generated objects, secondly, relies on coarse-granularity semantic tags for spatial constraint, is difficult to model complex topological relations, is easy to generate suspended or position-conflicting objects, thirdly, does not align the dynamic parallax characteristics of scenes, and causes depth distortion, edge artifacts and visual immersion sense damage of the generated objects.

Disclosure of Invention

The application provides a virtual reality interaction and content generation method and system based on a large model, which are used for solving the problems of object suspension, mold penetration and rendering distortion caused by neglecting dynamic environment physical interaction and weak space topology constraint in the prior art.

In a first aspect, the present application provides a method for generating virtual reality interaction and content based on a large model, including:

Collecting original point cloud data, synchronously obtaining semantic segmentation tags, and aligning the original point cloud data with the semantic segmentation tags according to space coordinates to construct a topological graph containing object positions, categories and space constraint relations;

Performing dynamic parallax calibration on the original point cloud data, mapping the calibrated original point cloud data to a unified space coordinate system, extracting a boundary of a shielding area and generating a dynamic parallax mapping table;

Acquiring blank region coordinates defined by a user in a virtual interface through a gesture recognition module, and generating a new object candidate parameter set conforming to a physical rule by combining a space constraint relation of the topological graph;

Analyzing the topological graph and the dynamic parallax mapping table by using a large model, screening the candidate parameter set of the new object based on the blank region coordinates, simulating collision response, gravity action and illumination interference effect of the new object and a dynamic entity in a scene by using a physical consistency verification module, and reserving the verified parameter combination as a physical verification parameter;

Generating geometric structure and texture information of the virtual object in the boundary of the shielding area according to the physical verification parameters, optimizing rendering precision of depth information of the geometric structure and the texture information by combining the dynamic parallax mapping table, outputting the optimized virtual object to an interactive interface, and updating the topological graph and the physical verification parameters according to interactive behavior data of a user and the optimized virtual object.

Optionally, generating geometry and texture information of the virtual object in the boundary of the occlusion region according to the physical verification parameter, and optimizing rendering precision of depth information of the geometry and the texture information in combination with the dynamic parallax mapping table includes:

Generating a virtual object in the boundary of the shielding area according to the size range, the position coordinates and the direction angle in the physical verification parameters by using an incremental nerve radiation field technology;

Extracting three-dimensional shape data and surface color data of the virtual object, taking the three-dimensional shape data as a geometric structure, and taking the surface color data as texture information;

extracting a depth correction value in a boundary of a corresponding shielding region in the dynamic parallax mapping table, optimizing and correcting the depth value of each coordinate point in the geometric structure according to the depth correction value, and optimizing and adjusting the surface detail rendering precision of the texture information according to the corrected depth value so as to output the virtual object with the geometric structure and the texture information optimized synchronously.

Optionally, the analyzing the topological graph and the dynamic parallax mapping table by using a large model, screening the candidate parameter set of the new object based on the blank region coordinates, simulating the collision response, the gravity effect and the illumination interference effect of the new object and the dynamic entity in the scene by using a physical consistency verification module, and reserving the verified parameter combination as a physical verification parameter, including:

analyzing object position coordinates, object categories and space constraint relations in the topological graph by using a large model, and analyzing depth correction values in the dynamic parallax mapping table;

Screening new objects with position coordinates falling in the blank area coordinate range from the new object candidate parameter set, and carrying out physical verification on the screened new objects through a physical consistency verification module;

verifying whether the collision response of the new object and the scene dynamic entity is through or overlapped, verifying whether the new object is overturned or suspended in the balance state under the action of gravity, and verifying whether the projection direction of the new object interfered by illumination conflicts with the scene light source;

And reserving all new object parameter combinations which do not have the mode penetration or overlapping, do not overturn or hang and have no projection direction conflict as physical verification parameters.

Optionally, the obtaining, by the gesture recognition module, coordinates of a blank area defined by a user in a virtual interface, and generating, in combination with a spatial constraint relationship of the topological graph, a new object candidate parameter set according with a physical rule includes:

capturing a closed area defined by a user in a virtual interface through a gesture recognition module, and extracting a three-dimensional coordinate range of the closed area as blank area coordinates;

reading the position coordinates of all objects in the topological graph and the space constraint relation between object pairs, detecting whether the blank area coordinates meet preset placement conditions, and judging an effective placement area when the blank area coordinates and the position coordinates of the supporting objects meet the vertical projection relation;

And calculating the size range, the position coordinates and the direction angle combination of the new object according to the object category and the space constraint relation of the adjacent object in the topological graph based on the physical rule in the effective placement area so as to generate a new object candidate parameter set.

Optionally, the outputting the optimized virtual object to the interaction interface, and updating the topological graph and the physical verification parameter according to the interaction behavior data of the user and the optimized virtual object, includes:

Rendering and outputting the optimized virtual object to an interactive interface;

capturing interaction behavior data comprising displacement track coordinates and rotation angles in the interaction process of the user and the optimized virtual object;

Updating the position coordinates of the corresponding objects in the topological graph according to the displacement track coordinates, and reestablishing the space constraint relation among the objects in the topological graph based on the rotation angle;

and simultaneously, according to the displacement track coordinates and the rotation angle, adjusting the position coordinates and the direction angles in the physical verification parameters, so that the adjusted physical verification parameters directly correspond to the interaction behavior data.

Optionally, the performing dynamic parallax calibration on the raw point cloud data, mapping the calibrated raw point cloud data to a unified space coordinate system, extracting a boundary of an occlusion region, and generating a dynamic parallax mapping table, including:

Calculating the coordinate offset of the same point at adjacent moments based on the original point cloud data, and reversely correcting the three-dimensional coordinate of each point according to the offset;

converting all points in the corrected original point cloud data into a preset global coordinate system;

Scanning a sudden change region of a depth value in the global coordinate system, and marking the outline of the sudden change region as a shielding region boundary when the depth difference value of adjacent points exceeds a set threshold value;

and distributing a depth correction value to each coordinate point in the boundary of the shielding area, and generating a dynamic parallax mapping table storing the association relation among the coordinate points, the depth correction value and the time stamp.

Optionally, the collecting the original point cloud data, synchronously obtaining a semantic segmentation tag, aligning the original point cloud data with the semantic segmentation tag according to space coordinates to construct a topological graph including object positions, categories and space constraint relations, including:

Acquiring three-dimensional coordinates of each point in original point cloud data, synchronously acquiring semantic segmentation labels of each point, and aligning the three-dimensional coordinates of each point with the semantic segmentation labels according to space coordinates;

The points which are adjacent to the aligned three-dimensional coordinates and are the same as the semantic segmentation labels are aggregated into independent objects, and the position coordinates of the independent objects and the associated object types are recorded;

Detecting object pairs meeting preset azimuth conditions according to the position coordinates of each independent object, and establishing a space constraint relation between the object pairs;

and generating a structural topological graph based on the position coordinates of each independent object, the object types and the space constraint relation among the object pairs.

In a second aspect, the present application provides a virtual reality interaction and content generation system based on a large model, including:

The acquisition module is used for acquiring original point cloud data, synchronously acquiring semantic segmentation tags and aligning the original point cloud data with the semantic segmentation tags according to space coordinates so as to construct a topological graph containing object positions, categories and space constraint relations;

The calibration module is used for carrying out dynamic parallax calibration on the original point cloud data, mapping the calibrated original point cloud data to a unified space coordinate system, extracting the boundary of the shielding area and generating a dynamic parallax mapping table;

The generation module is used for acquiring blank area coordinates defined by a user in the virtual interface through the gesture recognition module, and generating a new object candidate parameter set conforming to a physical rule by combining the space constraint relation of the topological graph;

The verification module is used for analyzing the topological graph and the dynamic parallax mapping table by utilizing a large model, screening the candidate parameter set of the new object based on the blank region coordinates, simulating the collision response, the gravity effect and the illumination interference effect of the new object and the dynamic entity in the scene by using the physical consistency verification module, and reserving the parameter combination passing verification as a physical verification parameter;

and the output module is used for generating geometric structures and texture information of the virtual objects in the boundary of the shielding area according to the physical verification parameters, optimizing rendering precision of depth information of the geometric structures and the texture information by combining the dynamic parallax mapping table, outputting the optimized virtual objects to an interaction interface, and updating the topological graph and the physical verification parameters according to interaction behavior data of users and the optimized virtual objects.

In a third aspect, the application provides a computing device, which comprises a processing component and a storage component, wherein the storage component stores one or more computer instructions, and the one or more computer instructions are used for being called and executed by the processing component to realize the virtual reality interaction and content generation method based on the large model in the first aspect.

In a fourth aspect, the present application provides a computer storage medium storing a computer program, where the computer program is executed by a computer to implement a virtual reality interaction and content generation method based on a large model as described in the first aspect.

In the embodiment of the application, initial point cloud data are acquired, semantic segmentation labels are synchronously acquired, the initial point cloud data and the semantic segmentation labels are aligned according to space coordinates to construct a topological graph containing object positions, categories and space constraint relations, dynamic parallax calibration is carried out on the initial point cloud data, the calibrated initial point cloud data are mapped to a unified space coordinate system, an occlusion region boundary is extracted and a dynamic parallax mapping table is generated, blank region coordinates defined by a user in a virtual interface are acquired through a gesture recognition module, the space constraint relations of the topological graph are combined to generate a new object candidate parameter set conforming to physical rules, the topological graph and the dynamic parallax mapping table are analyzed through a large model, the new object candidate parameter set is screened based on the blank region coordinates, the collision response, the gravity effect and the illumination interference effect of a new object and a dynamic entity in a scene are simulated through a physical consistency verification module, the verified parameter combination is reserved as physical verification parameters, the geometric structure and texture information of the virtual object are generated in the boundary of the virtual object according to the physical verification parameters, the depth information of the geometric texture structure and the texture information are combined with the dynamic parallax mapping table, and the virtual parallax information is optimized according to the optimized interaction precision of the virtual object and the virtual object interaction precision is simultaneously output to the virtual interaction precision and the virtual interaction precision after the virtual object is optimized.

The technical scheme of the application has the following beneficial effects:

The method comprises the steps of constructing a topological graph by synchronously aligning an original point cloud with semantic tags, accurately modeling object positions, categories and space constraint relations, generating a mapping table based on dynamic parallax calibration to capture occlusion boundary and parallax change, generating a new object candidate parameter set of physical compliance by combining blank region coordinates defined by user gestures and utilizing topological graph space constraint, analyzing topological graph and mapping table screening parameters through a large model, dynamically simulating collision response, gravity action and illumination interference effect through a physical consistency verification module, ensuring physical reality of dynamic entity interaction of the new object and a scene, generating a geometric structure and texture in the occlusion boundary based on an incremental nerve radiation field, optimizing depth and rendering precision by means of the mapping table, realizing seamless fusion of the virtual object and the original scene, continuously updating the topological graph and the physical parameters according to user interaction behaviors, forming a closed-loop optimization mechanism, and remarkably improving physical consistency, semantic rationality and visual sense of object generation in the dynamic virtual reality scene.

Further, in the boundary of the shielding area, the geometric structure and texture information of the virtual object are generated by applying an incremental nerve radiation field technology based on physical verification parameters, depth correction values of corresponding areas in a dynamic parallax mapping table are synchronously extracted, the depth values of all coordinate points of the geometric structure are optimized and corrected accordingly, the surface detail rendering precision of the texture information is cooperatively adjusted, and finally the virtual object with synchronously optimized geometry and texture is output. The method accurately optimizes the geometric depth information of the virtual object through the depth correction value of the dynamic parallax mapping table, eliminates perspective deviation with the original object of the scene, adaptively adjusts texture rendering details based on the depth correction value, solves the problems of artifact and illumination distortion at the shielding boundary, achieves seamless fusion of the generated object and the dynamic scene in geometric fitting degree and visual consistency, and remarkably improves physical reality and visual immersion effects of the virtual object.

These and other aspects of the application will be more readily apparent from the following description of the embodiments.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 shows a flow chart of a virtual reality interaction and content generation method based on a large model provided by the application;

FIG. 2 illustrates a scene graph of a large model-based virtual reality interaction and content generation method provided by the application;

FIG. 3 is a schematic diagram of a virtual reality interaction and content generation system based on a large model according to the present application;

FIG. 4 illustrates a schematic diagram of a computing device provided by the present application.

Detailed Description

In order to enable those skilled in the art to better understand the present application, the following description will make clear and complete descriptions of the technical solutions according to the embodiments of the present application with reference to the accompanying drawings.

In some of the flows described in the specification and claims of the present application and in the foregoing figures, a plurality of operations occurring in a particular order are included, but it should be understood that the operations may be performed out of order or performed in parallel, with the order of operations such as 101, 102, etc., being merely used to distinguish between the various operations, the order of the operations themselves not representing any order of execution. In addition, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that, the descriptions of "first" and "second" herein are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, and are not limited to the "first" and the "second" being different types.

Research shows that in the automatic generation technology of the virtual reality scene, the existing method has three main problems that firstly, the physical collision detection can only process static objects, interaction between moving objects and generated objects cannot be calculated correctly, so that the objects penetrate through each other or are suspended, and other unreal phenomena are caused, secondly, the space layout only depends on simple object classification labels, complex relations such as a table needs to support objects above are difficult to express, objects with unreasonable positions (such as suspended cups) or mutually conflicting objects are easy to generate, and finally, depth difference caused by visual angle change is not considered when a picture is rendered, so that the generated objects have the problems of edge saw teeth, unnatural light shadow, and the like, and the sense of reality is seriously influenced.

Aiming at the problems, the scheme provides an improved virtual reality generation technology, wherein a logic connection diagram is established by analyzing the spatial relationship between objects, the object placement is ensured to be in accordance with common sense (such as a cup is required to be placed on a desktop), a physical simulation function is added, the collision effect and the light change of a moving object and a scene are calculated in real time, the interaction reality is improved, meanwhile, the rendering mode is optimized, and the edge of the object and the shadow effect are automatically adjusted according to the observation angle, so that the object is naturally fused with the scene. The technical scheme has the core advantages that the reasonable placement of the objects is ensured through logic constraint, the penetration and suspension problems are solved through physical simulation, and the picture flaws are eliminated through optimized rendering, so that the high unification of physical rules, logical relations and visual effects is realized in a dynamic scene.

The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to fall within the scope of the application.

Fig. 1 is a flowchart of a method for generating virtual reality interaction and content based on a large model according to an embodiment of the present application, as shown in fig. 1, the method includes:

101. Collecting original point cloud data, synchronously obtaining semantic segmentation tags, and aligning the original point cloud data with the semantic segmentation tags according to space coordinates to construct a topological graph containing object positions, categories and space constraint relations;

optionally, step 101 may specifically include the following steps:

1011. acquiring three-dimensional coordinates of each point in original point cloud data, synchronously acquiring semantic segmentation labels of each point, and aligning the three-dimensional coordinates of each point with the semantic segmentation labels according to space coordinates;

1012. the points which are adjacent to the aligned three-dimensional coordinates and are the same as the semantic segmentation labels are aggregated into independent objects, and the position coordinates of the independent objects and the associated object types are recorded;

1013. Detecting object pairs meeting preset azimuth conditions according to the position coordinates of each independent object, and establishing a space constraint relation between the object pairs;

1014. and generating a structural topological graph based on the position coordinates of each independent object, the object types and the space constraint relation among the object pairs.

In the above scheme, the original point cloud data refers to a spatial position information set acquired by a three-dimensional scanning device, and includes each axial coordinate value of each sampling point on the surface of the object, which can be used for reconstructing the solid geometry of the object. The semantic segmentation label refers to a category identifier allocated to each point in the point cloud by a computer vision algorithm, and can be used for distinguishing the functional attribution of different objects. The space coordinate alignment refers to a technical process of matching the position of the point cloud coordinate and the semantic label in the three-dimensional space, and can be used for ensuring the consistency of the geometric position and the semantic attribute of the object. The independent objects refer to entity units formed by aggregation of point groups which are spatially adjacent and have the same semantic labels, and can be used for representing complete physical entities in a scene. The spatial constraint relationship refers to a logical association describing the orientation rule between two independent objects, and can be used to define the spatial dependence between the objects. The topological graph refers to a structured network constructed by taking independent objects as nodes and space constraint relations as edges, and can be used for machine understanding of the layout logic of a scene.

In the embodiment of the application, firstly, step 1011 scans a target scene through a laser radar, acquires three-dimensional space coordinates of each point in an original point cloud, and synchronously calls a pre-trained semantic segmentation model to classify and predict each point to generate a corresponding semantic label. And then, carrying out space matching on the semantic tag and the point cloud coordinate by adopting a coordinate transformation technology, and mapping tag data to the point cloud coordinate system by calculating an affine transformation matrix. The process ensures that each space point has geometric position and semantic property at the same time, and lays a data foundation for subsequent object aggregation. For example, in an urban road scan, coordinates (20.1,5.3,0) are labeled "road surface" and (20.1,5.3,1.5) are labeled "car".

Then, based on the tagged point cloud data output in step 1011, the object aggregation is performed by using the euclidean distance clustering algorithm, and all the neighboring points of the same semantic tags within the radius of 0.2 meter are searched by taking each point as the center. When the neighboring points meet the distance threshold and the labels are identical, they are merged into independent object entities, e.g., 120 "street light" points dispersed are aggregated into one complete street light. The geometric center coordinates of the cluster of points are then calculated, and the X, Y, Z coordinates of all points are averaged separately, e.g., 120 points in the X range 10.0-10.3, center coordinates (10.15,2.1,3.0). The final output contains a structured list of object position coordinates and categories, e.g. object ID-001: { category: street lamp, coordinates (10.15,2.1,3.0) }.

Then, through step 1013, using the independent object list, all object pairs are traversed and the spatial orientation parameters are calculated, the center coordinates of the two objects are extracted, and their relative distances in the X, Y, Z axis direction are calculated. And matching the calculation result with a preset rule base, if the Z-axis difference value is larger than a vertical distance threshold value, establishing a relation of A above B, and if the horizontal distance is smaller than an attachment threshold value, establishing a relation of A attached B. For example, the Z difference between the shelf and the tray is 1.7 meters > preset 1.5 meters, triggering the generation of a spatial constraint relationship of "shelf above tray".

Finally, step 1012 integrates the object list containing the position coordinates and object categories of each individual object with the spatial constraint relation of step 1013 through step 1014, constructs a structured topological graph, converts each individual object into graph nodes, the node attributes contain center coordinates and categories, such as node 1: { coordinates (2,3,0.5), category: desk }, converts the spatial constraint relation into directed edges connecting the nodes, and the edge attributes record the relation types, such as node 1 and node 2{ coordinates (2,3,1.5), category: bookshelf-over-desk "edges are established. Finally, a machine-resolvable network diagram is generated, such as a shelf node, a forklift node and a relation edge of 'shelf over forklift' in a warehouse scene, which jointly describe equipment layout logic.

In practical application, in a virtual reality equipment operation training scene, firstly, scanning a virtual operation table environment through a laser radar, collecting three-dimensional coordinates of 5000 points on the surface of the operation table, such as points P1 (1.0,2.0,0.8) and P2 (1.0,2.1,0.9), synchronously calling a semantic segmentation model to predict categories P1 marking an operation panel and P2 marking a knob of each point, binding a label to coordinate points such as coordinates (1.0,2.0,0.8) and precisely associating the coordinate points with the operation panel through an affine transformation matrix, then executing object aggregation based on a labeled point cloud, searching adjacent identical label points with a radius of 0.15 meter by adopting a DBSCAN algorithm, aggregating the label points of the X range of 1.0-1.2, the Y range of 2.0-2.3 and the Z range of 0.8-1.0320 into an independent object A, calculating center coordinates,,,Then detecting the space relation, calculating the Z-axis difference value of the center (1.1,2.15,0.9) of the object A and the center (1.1,2.15,0.3) of the object B, which is equal to (1.1,2.15,0.3), of |0.9-0.3|=0.6 m, establishing a relation of attaching an operation panel to the wrench because the Z-axis difference value is smaller than a vertical relation threshold value of 1.0m and the horizontal distance is smaller than an attaching threshold value of 0.1 m, finally generating a topological graph, taking the object A { type: operation panel, coordinates (1.1,2.15,0.9) }, the object B { type: virtual wrench, coordinates (1.1,2.15,0.3) } as nodes, adding an attaching relation edge, and outputting a structured scene model to a large model to drive a virtual training content generating system.

According to the whole scheme of 101, object entities are accurately reconstructed through space coordinate alignment, the position and type attributes of independent objects are generated based on semantic tag aggregation, and the space orientation constraint relation among the objects is automatically detected. The finally constructed topological graph describes the distribution logic and interaction rules of all objects in the scene in a machine-resolvable form, provides a space cognition basis for dynamic interaction, behavior prediction and content generation in a virtual reality environment, and remarkably improves the accuracy and efficiency of environment understanding.

102. Performing dynamic parallax calibration on the original point cloud data, mapping the calibrated original point cloud data to a unified space coordinate system, extracting a boundary of a shielding area and generating a dynamic parallax mapping table;

Optionally, step 102 may specifically include the following steps:

1021. Calculating the coordinate offset of the same point at adjacent moments based on the original point cloud data, and reversely correcting the three-dimensional coordinate of each point according to the offset;

1022. converting all points in the corrected original point cloud data into a preset global coordinate system;

1023. Scanning a sudden change region of a depth value in the global coordinate system, and marking the outline of the sudden change region as a shielding region boundary when the depth difference value of adjacent points exceeds a set threshold value;

1024. And distributing a depth correction value to each coordinate point in the boundary of the shielding area, and generating a dynamic parallax mapping table storing the association relation among the coordinate points, the depth correction value and the time stamp.

In the above scheme, the dynamic parallax calibration refers to a technical process of reversely correcting the position of the point cloud by calculating the coordinate offset of the moving object at adjacent moments, and the dynamic parallax calibration comprises a distortion elimination mechanism caused by visual angle change, and can be used for improving the accuracy of three-dimensional reconstruction of the dynamic scene. The unified space coordinate system refers to a preset three-dimensional global reference standard, comprises fixed original point positions and coordinate axial definitions, and can be used for fusing the space standard unification of multi-frame point cloud data. The occlusion region boundary refers to a depth value discontinuous region outline formed by covering objects with each other, and comprises edge features with depth jump exceeding a physical rationality threshold, and the edge features can be used for identifying the geometric range of a vision invisible region in a scene. The dynamic parallax mapping table refers to a related database for storing depth correction values and time stamps of coordinate points in an occlusion region, and comprises spatial position, depth compensation quantity and time dimension information, and can be used for updating the object spatial relationship in a virtual environment.

In the embodiment of the present application, firstly, step 1021 identifies the corresponding point location at the adjacent time by a point cloud matching algorithm, for example, extracts the bolt coordinates (0.5,0.6,1.0) in the t1 frame and the nearest neighbor (0.7,0.6,1.0) in the t2 frame, and calculates the three-dimensional offset. And then correcting the coordinates by applying a reverse correction formula, wherein the correction formula is as follows: For example, according to the amount of offset The rice is used for the production of rice,,The t2 frame coordinates (0.7,0.6,1.0) are corrected to (0.5,0.6,1.0). The process eliminates point position drift caused by the movement of the mechanical arm and outputs dynamic point cloud data with consistent positions.

Then, based on 1021 the modified point cloud data, spatial mapping is performed using a predetermined conversion matrix, the matrix including rotation components and translation components. Taking correction point (0.5,0.6,1.0) as an example, rotating component is rotated about Z axis by 30 degrees, translation component is +0.1 m, -0.2 m, 0, and rotation calculation is performed,And performing translation superposition, namely, global coordinates of = (0.133+0.1, 0.770-0.2, 1.0+0) = (0.233,0.570,1.0), and finally converting the full scene point cloud into a unified world coordinate system to realize multi-frame data space alignment.

Then, the depth jump is scanned point by point in the global coordinate system by step 1023, and the depth difference is calculated, taking the mechanical arm shielding area as an example, taking the point A (0.233,0.570,1.0) and the adjacent point B (0.235,0.572,0.2) thereof, and calculating the depth difference |1.0-0.2|=0.8 meters. When the value exceeds the set threshold, the two-point line is marked as a boundary line segment. And connecting all boundary line segments meeting the conditions to form a closed contour, and marking as a shielding area boundary.

Finally, through step 1024, for the coordinate points in the occlusion boundary, searching for non-occlusion points with radius of 0.3 m, and calculating the depth average value to generate the dynamic parallax mapping table. For example, the non-occlusion points within 0.3 m radius of point C (0.240,0.575,0.3) at t2 have D (0.230,0.568,1.0) and E (0.228,0.565,0.95), and the depth average value is calculated to beRice is obtained ""Dynamic disparity map table entry. The table continuously updates the space-time compensation parameters of the occlusion region, driving the virtual reality system to render the occluded object.

In practical application, in a virtual circuit board maintenance scene, dynamic point cloud is firstly captured through a laser radar, capacitance coordinates (1.0,2.0,0.5) at time t1 are detected, the device vibration is shifted to (1.2,2.0,0.5) at time t2, and X-direction offset is calculatedThe stable coordinates (1.2-0.2,2.0,0.5) = (1.0,2.0,0.5) are corrected in opposite directions, then coordinate system conversion is performed, a preset conversion matrix is applied, 45 degrees of rotation is performed around the Z axis, translation (+ 0.3, -0.1, 0) is performed, and global coordinates are calculated:,, The method comprises the steps of outputting global coordinates (-0.293,1.828,0.5), then extracting a shielding boundary, scanning a resistance point (-0.295,1.830,0.1) which is shielded by tweezers, marking the outlines of the tweezers as shielding areas, wherein the depth difference is 0.5-0.1|=0.4 meter, marking the outlines of the tweezers as shielding areas, finally generating a mapping table, searching non-shielding point depth values {0.50,0.48,0.52} within the radius of 0.2 meter for the shielding point (-0.297,1.832,0.15), calculating the average value to be 0.50, generating an item 'coordinate (-0.297,1.832,0.15) | correction value: 0.50|timestamp: t 2', and driving a large model to generate a virtual reality operation prompt based on the mapping table 'the existence of a capacitor to be replaced behind the tweezers'.

According to the whole scheme of 102, the point cloud position drift caused by a moving object is eliminated through dynamic calibration, the space reference of multi-frame data is unified, the depth abrupt change area outline generated by object shielding is accurately identified, and a time-space associated depth correction mapping table is generated. The mapping table updates three-dimensional space parameters of the shielding area, provides high-precision space consistency data for the virtual reality scene, remarkably improves the authenticity and immersion of dynamic environment reconstruction, and lays a reliable space cognition foundation for interactive content generation driven by a large model.

103. Acquiring blank region coordinates defined by a user in a virtual interface through a gesture recognition module, and generating a new object candidate parameter set conforming to a physical rule by combining a space constraint relation of the topological graph;

optionally, step 103 may specifically include the following steps:

1031. capturing a closed area defined by a user in a virtual interface through a gesture recognition module, and extracting a three-dimensional coordinate range of the closed area as blank area coordinates;

1032. Reading the position coordinates of all objects in the topological graph and the space constraint relation between object pairs, detecting whether the blank area coordinates meet preset placement conditions, and judging an effective placement area when the blank area coordinates and the position coordinates of the supporting objects meet the vertical projection relation;

1033. and calculating the size range, the position coordinates and the direction angle combination of the new object according to the object category and the space constraint relation of the adjacent object in the topological graph based on the physical rule in the effective placement area so as to generate a new object candidate parameter set.

In the above scheme, the gesture recognition module is a technical component for capturing the motion track of the hand of the user through the motion sensor and converting the motion track into the virtual space operation instruction, and comprises fingertip position tracking and motion track analysis functions, which can be used for recognizing the interaction intention of the user in the virtual environment. The blank region coordinates refer to a three-dimensional space range of the non-placed object, which is manually defined by a user in the virtual interface, and comprise boundary vertex coordinates and space volume parameters of the closed region, so that a target position reference can be provided for new object generation. The candidate parameter set refers to a new object attribute configuration set automatically calculated based on scene physical rules, and comprises selectable items such as a size range, a placement coordinate, a rotation angle and the like, and can be used for generating a virtual object instance conforming to a space constraint relation.

In the embodiment of the application, the Leap Motion sensor captures the Motion track of the fingertip of the user in the virtual space, and the continuous position point sequence, such as track points, is recorded. Connecting track points by adopting a polygon closing algorithm to generate a quadrilateral region, then calculating the minimum circumscribing cube of the region, traversing X, Y, Z coordinate values of all track points, taking an X minimum value of 1.0 and a maximum value of 2.0, a Y minimum value of 2.0 and a maximum value of 3.0, and a Z constant value of 0.5, and finally outputting a blank region coordinate range、、。

Next, based on the blank area coordinates output in step 1031 and the attributes of the object in the topology, the actual height of the object is first calculated. Then, vertical projection detection is performed, and the absolute value of the height difference between the bottom surface of the blank area and the top surface of the object is calculated. And comparing the absolute value of the height difference with a preset threshold value, judging that the height difference is invalid if the absolute value of the height difference is larger than the threshold value, otherwise, judging that the height difference is valid, marking the height difference as a valid placement area, and outputting the coordinate range of the area.

Finally, the effective area passing through step 1032 is verified by step 1033, size calculation is performed in combination with the space constraint of the adjacent objects of the topological graph, and the final integration parameters of position calculation and angle calculation are generated into a candidate set. For example, the effective area is X1.0-2.0, Y2.0-3.0 and Z0.85, the adjacent bookshelf width of the topological graph is 0.4m, the distance between objects is equal to or larger than 0.1m, the width range of a new book is set to be [0.4-0.05,0.4+0.05] = [0.35,0.45 m ] according to the bookshelf width of 0.4m and the distance rule, the geometric center of a blank area is taken, the X center is 1.5, the Y center is 2.5, Z=0.85, the orientation angle of an object is set to be 0 DEG according to the normal vector (0, 1, 0) of a scene wall surface, and the final integration parameter is generated to form a candidate set。

In practical application, as shown in FIG. 2, the gesture of the user demarcates the upper right corner area of the desk, the track pointThe system extracts the blank areas X1.0-1.2, Y0.8-1.0 and Z0.85. And in the verification stage, the height of the top surface of the desk is 0.9 meter, and the difference of the height of 0.85-0.9 I=0.05 meter is smaller than the threshold value of 0.1 meter, so that the desk is judged to be effective. The parameter generation stage is based on calculation of the diameter of the adjacent desk lamp by 0.15 m, the book width range is 0.15-0.03,0.15+0.03] = [0.12,0.18] m, the angle of the center (1.1,0.9,0.85) is matched with the edge direction of the desk by 30 degrees, and the output parameter set drives the virtual reality system to render a book model.

According to the whole scheme of 103, the target operation area in the virtual environment is accurately identified by capturing gesture actions of the user, and whether the area accords with the physical placement rule is verified based on the scene object position relation diagram. In the effective area passing verification, the reasonable size range, the central position coordinate and the scene adaptive rotation angle of the new object are automatically calculated and generated by combining the sizes of the adjacent objects and the space constraint relation. And finally, outputting a candidate parameter set conforming to the environment logic, realizing the closed loop from the gesture instruction of the user to the intelligent generation of the virtual object, and obviously improving the naturalness of the virtual reality interaction and the content generation accuracy.

104. Analyzing the topological graph and the dynamic parallax mapping table by using a large model, screening the candidate parameter set of the new object based on the blank region coordinates, simulating collision response, gravity action and illumination interference effect of the new object and a dynamic entity in a scene by using a physical consistency verification module, and reserving the verified parameter combination as a physical verification parameter;

optionally, the step 104 may specifically include the following steps:

1041. analyzing object position coordinates, object categories and space constraint relations in the topological graph by using a large model, and analyzing depth correction values in the dynamic parallax mapping table;

1042. Screening new objects with position coordinates falling in the blank area coordinate range from the new object candidate parameter set, and carrying out physical verification on the screened new objects through a physical consistency verification module;

1043. Verifying whether the collision response of the new object and the scene dynamic entity is through or overlapped, verifying whether the new object is overturned or suspended in the balance state under the action of gravity, and verifying whether the projection direction of the new object interfered by illumination conflicts with the scene light source;

1044. and reserving all new object parameter combinations which do not have the mode penetration or overlapping, do not overturn or hang and have no projection direction conflict as physical verification parameters.

In the above scheme, the physical consistency verification module refers to a calculation engine for simulating physical behaviors of the virtual object in the dynamic scene, and comprises collision response detection, gravity balance analysis and illumination projection matching functions, and can be used for ensuring mechanical and optical compatibility of the newly generated object and the environment. The mode penetration or overlapping refers to the phenomenon of unreasonable space crossing between virtual objects, and the phenomenon includes that geometric boundaries penetrate through each other or overlap areas exceed tolerance thresholds, and can be used for identifying position conflicts of object placement. The toppling or suspending refers to a suspension state of the object caused by the fact that the gravity center deviates from the supporting surface, and the suspension state comprises insufficient coverage rate of the supporting area or the gravity center projection exceeds the range of the substrate, so that the object stability defect can be detected. The projection direction conflict refers to that the object shadow vector and the scene light source direction have logic contradiction, and can be used for verifying illumination consistency.

In the embodiment of the present application, first, step 1041 loads the structured data of the topological graph through the large model, and reads the object space attribute recorded therein, such as the bookshelf center coordinate (1.0,0.9,0.8), and the dimension length width height [0.4,2.0,0.3] m. The dynamic disparity map table entry is parsed at the same time, e.g. "coordinate (0.3,1.0,0.3) | correction value 0.95|timestamp t2", the original depth value of the occlusion region is replaced by the correction value, e.g. the depth of coordinate (0.3,1.0,0.3) is updated from 0.3 meters to 0.95 meters. The complete three-dimensional scene is rebuilt through a space fusion algorithm, and particularly the real surface of the shielded object is restored, so that an accurate space reference is established for physical verification, for example, the actual height of the shielded corner of the desk is 0.95 meter.

Then, based on the three-dimensional scene reconstructed in step 1041, a new object candidate parameter set is traversed, and position coordinates are detected item by item according to a blank region coordinate range defined by the user. For example, three groups of book parameters, namely A (1.1,0.9,0.85), B (1.3,1.0,0.87) and C (0.9,0.8,0.86), are included, and the coordinate ranges of the delimited blank areas are X, 1.0-1.2, Y, 0.8-1.0 and Z, 0.85-0.90, and then the position coordinate detection is carried out. It is detected that x=1.1 e [1.0,1.2] in parameter a (1.1,0.9,0.85), y=0.9 e [0.8,1.0], z=0.85 e [0.85,0.90] results in conclusion retention, that x=1.3 >1.2 in parameter B (1.3,1.0,0.87) results in conclusion elimination, and that x=0.9 <1.0 in parameter C (0.9,0.8,0.86) results in conclusion elimination. And outputting the screened effective candidate set to a physical verification module.

Then, through step 1043, three-dimensional physical simulation is performed on the candidate parameters selected, the bounding box distance between the new object and the dynamic entity is calculated, and collision response verification is performed to determine whether the model penetration or the overlap occurs. For example, the book size [0.15,0.2,0.02], the current position (1.1,0.9,0.88) of the mechanical arm, the size [0.1,0.1,0.3] are calculated to obtain the height of the top surface of the book=the position z+the height=0.85+0.02=0.87 m, the height of the bottom surface of the mechanical arm=0.88 m, the vertical distance |0.88-0.87|=0.01 m is smaller than the safety threshold value 0.05 m, and the collision is judged. Detecting the projection coverage rate of the bottom surface of the object on the supporting surface to verify whether the equilibrium state under the action of gravity is overturned or suspended. For example, the coordinates of four corner points are (1.025,0.8,0.85), (1.175,0.8,0.85), (1.025,1.0,0.85) and (1.175,1.0,0.85), the desk plane z=0.90, the four points z=0.85 are smaller than the supporting plane z=0.90, the projection area occupied ratio is 0%, and suspension is determined. And calculating an included angle between the shadow direction vector of the object and the direction of the scene light source, and verifying whether the projection direction interfered by illumination conflicts with the scene light source. For example, the book shadow direction vector is (0, -0.9, 0), the scene light source is (0, 1, 0), and the included angle is calculatedObtainingAnd (5) judging the illumination conflict when the angle is larger than 90 degrees.

Finally, a decision is generated according to the verification result of step 1043 through step 1044. Finally, only parameters passing through full verification are reserved, and if not, an empty set is output and regeneration is triggered. For example, the parameters A such as collision detection failure, gravity balance failure and illumination verification failure are eliminated, if the parameters are adjusted to the position (1.1,0.9,0.91), the verification is passed if the collision distance is |0.91+0.02-0.88|=0.05 m larger than or equal to the threshold value, the verification is passed if the bottom surface Z=0.91 is close to the supporting surface Z=0.90 < tolerance 0.02, and the shadow vector (0, -1, 0) forms an included angle of 180 degrees to the light source (0, 1, 0) and still fails.

In practical application, in a virtual equipment maintenance scene, a user demarcates an equipment table operation area through gestures, a large model firstly analyzes bolt positions (0.6,1.1,0.92) in a topological graph and correction values of a dynamic parallax mapping table to restore real space positions of shielded parts, and then screens candidate parameter set positions (0.6,1.1,0.90) of tools. And then performing triple physical verification, wherein the distance between the height of the tool bottom surface and the lowest position of the moving mechanical arm is only 0.03 meter, the distance is smaller than a safety threshold value of 0.05 meter, the collision risk is judged, the tool bottom surface Z=0.90 meter is lower than the equipment table supporting surface Z=0.92 meter, the bottom surface is completely suspended, the supporting coverage rate is 0%, the included angle between a tool shadow vector (-0.5, 0) and the scene light source direction (0, 1, 0) is 120 degrees, the tolerance exceeds 90 degrees, direction conflict is generated, due to the failure of full verification, the system prompts that the tool height is adjusted, a user lifts the position to (0.6,1.1,0.95), re-verifies, the collision distance is 0.95-0.93|0.02 meter and still is still smaller than 0.05 meter, the collision distance is 0.05 meter and still is up to (0.6,1.1,0.98), the upper limit Z=0.93 of the movement of the mechanical arm reaches the standard, the bottom surface Z=0.98 is close to the supporting surface Z=0.92, the height difference of 0.06 meter triggers a suspended warning, and finally a cushion height scheme is adopted, so that the actual height of the tool reaches 0.98 meter, the distance of 0.98-0.98 m and the shadow mask is 0.98, the distance of 0.98 is 0.93|0.93 to the completely-0.93 m, the shadow mask is completely corresponds to the real-plane, and the shadow mask (0.1, the shadow mask is completely-0.93 m, the shadow mask is completely-matched with the light source, the visual projection system is completely, and the visual system is matched with the operation model is generated.

According to the whole scheme of 104, the compatibility of the new object and the dynamic scene is verified through multidimensional physical simulation, the contradiction among space penetration, gravity instability and illumination logic is eliminated, the virtual content generation is ensured to accord with the physical rules of the real world, and the interaction credibility of the immersive environment is improved.

105. Generating geometric structure and texture information of the virtual object in the boundary of the shielding area according to the physical verification parameters, optimizing rendering precision of depth information of the geometric structure and the texture information by combining the dynamic parallax mapping table, outputting the optimized virtual object to an interactive interface, and updating the topological graph and the physical verification parameters according to interactive behavior data of a user and the optimized virtual object.

Alternatively, step 105 may specifically include the steps of:

1051. generating a virtual object in the boundary of the shielding area according to the size range, the position coordinates and the direction angle in the physical verification parameters by using an incremental nerve radiation field technology;

1052. extracting three-dimensional shape data and surface color data of the virtual object, taking the three-dimensional shape data as a geometric structure, and taking the surface color data as texture information;

1053. extracting a depth correction value in a boundary of a corresponding shielding region in the dynamic parallax mapping table, optimizing and correcting the depth value of each coordinate point in the geometric structure according to the depth correction value, and optimizing and adjusting the surface detail rendering precision of the texture information according to the corrected depth value so as to output the virtual object with the geometric structure and the texture information optimized synchronously.

1054. Rendering and outputting the optimized virtual object to an interactive interface;

1055. Capturing interaction behavior data comprising displacement track coordinates and rotation angles in the interaction process of the user and the optimized virtual object;

1056. updating the position coordinates of the corresponding objects in the topological graph according to the displacement track coordinates, and reestablishing the space constraint relation among the objects in the topological graph based on the rotation angle;

1057. and simultaneously, according to the displacement track coordinates and the rotation angle, adjusting the position coordinates and the direction angles in the physical verification parameters, so that the adjusted physical verification parameters directly correspond to the interaction behavior data.

In the above scheme, the incremental neural radiation field technology refers to a method for gradually optimizing a three-dimensional object generation process through a machine learning model, and comprises geometric iterative construction and surface texture detail enhancement capability, and can be used for creating a high-precision virtual object in an occlusion region. The geometric structure refers to space frame data describing the three-dimensional form of an object, and comprises vertex coordinates, edge connection relations and surface grid information, and can be used for defining the physical outline of a virtual object. The texture information is a data set for recording visual characteristics of the surface of the object, and comprises optical properties such as color values, glossiness, roughness and the like, and can be used for rendering a virtual appearance with a material sense of reality. The interactive behavior data refer to action parameters including a space coordinate sequence of a displacement track and a rotation angle change value when a user operates a virtual object, and can be used for driving a scene to be updated dynamically.

In the embodiment of the application, firstly, based on physical verification parameters through step 1051 and step 1052, an incremental neural radiation field model is called to generate a three-dimensional object, the model firstly generates a basic cube frame, and details are gradually added through 8 grid subdivision iterations. For example, book size [0.4,0.3,0.05] m, position (1.5,2.5,0.85), angle 0,8 grid subdivision iterations step by step add e.g. spine grooves, cover textures, and finally output a book grid model containing 2000 vertices. And then extracting geometrical structure data recording all vertex coordinates and connection relations and texture information data storing optical attribute parameters of the surface. For example, as with vertex P1 (1.52,2.53,0.86), cover color RGB (120,80,30), spine roughness 0.3).

Next, spatial correction is performed and texture information is adjusted synchronously in combination with the dynamic parallax map in step 1053. For example, inquiring about the depth correction value of the book position area (1.5, 2.5) by 0.95 m, traversing the geometric structure vertexes for depth optimization, for example, correcting the Z coordinate of the vertex P1 from 0.86 to 0.96, integrally lifting by 0.1 m, and enhancing the surface detail according to the depth change amount delta Z= +0.1 m, wherein the calculation formula of the intensity of the shade of the book is as follows: ", cover gloss was reduced by 20% to simulate a depth fade effect. And finally outputting the optimized geometric structure and the self-adaptive texture data set.

The optimized geometry data is then input to a graphics rendering engine, such as drawing a book model at virtual reality interface coordinates (1.5,2.5,0.95), displaying a shadow fade effect of 0.6 intensity at the spine, via steps 1054 and 1055. When the user operates the handle, the motion sensor captures interaction data at a frequency of 100Hz, for example, the displacement trajectory is recorded as a time stamp sequence, and the rotation angle is calculated from the change of 0 ° to 15 ° around the Z-axis by the gyroscope data, forming a complete interaction behavior data set.

Finally, the topology is updated based on the interaction behavior data and the spatial constraint relationship is reconstructed, for example, the book position coordinates in the topology are updated from (1.5,2.5,0.95) to (1.6,2.6,0.95), and then a new distance to the bookshelf is calculatedThe distance between the books is reduced to trigger the change of the spatial relationship, so that the books are close to the bookshelf and are independently placed instead of the original books. At the same time, the physical verification parameters are updated, for example, the position item is synchronized to be (1.6,2.6,0.95), the angle item is updated to be 15 degrees, and the parameter set is ensured to be matched with the current state of the object.

In practical application, in mechanical maintenance training, the system firstly generates a hexagonal cylinder model based on the position (0.6,1.1,0.94) of the physical parameter bolt size [0.1,0.1,0.05] to extract 500 vertex coordinates and metal texture RGB (180,180,180), then searches the mapping table to obtain depth correction value of 0.95 m, lifts all vertex Z coordinates to 0.95 and increases rust concave-convex texture of 0.2mm, renders the rust-carrying bolt to the operation interface, and generates displacement track by user screwing operationThe final synchronous physical parameter is the angle 60 degrees of the position (0.6,1.1,0.90), and the closed loop from the virtual object generation to the state update is completed.

According to the overall scheme 105, compatibility of newly generated objects and dynamic scenes is ensured through a triple physical verification mechanism, unreasonable penetration among the objects is eliminated by the collision detection module, the defect of overturning or hanging is prevented by the gravity balance module, and the projection direction is ensured to be consistent with the scene light source logic by the illumination verification module. Only the parameter combinations passing all verification are reserved, the problems of interference, support instability and light and shadow distortion of moving objects in a virtual environment are solved, and physical compliance guarantee is provided for high-reliability virtual reality interaction.

The following is a complete example for steps 101-105, in which the system obtains raw point cloud data of the engine assembly by laser radar scanning and applies a semantic segmentation model to mark the category of each point synchronously in an aero-engine maintenance training scenario. Binding the point cloud and the labels through space coordinate alignment, aggregating adjacent same label points into a topological graph containing positions, categories and relations based on Euclidean distance clustering algorithm, wherein the vertical distance between an independent object and a bracket (1.5,3.5,0.3) is 0.6 m < a threshold value is 0.7 m, establishing a space constraint relation that a turbine is fixed above the bracket, and finally generating the topological graph containing the positions, the categories and the relations.

When the movement of the trainee causes the change of the visual angle, the system detects the coordinate shift of the bolts of the adjacent frames, and reversely corrects DeltaX=0.2 meters to obtain stable coordinates (0.5,0.6,1.0). After conversion to world coordinate system, (0.23,0.42,1.0). Scanning a depth abrupt change region in a global coordinate system, detecting a gasket point (0.25,0.43,0.3) shielded by a tool and a bolt depth difference |1.0-0.3|=0.7 m > threshold value 0.5m, and marking as a shielding boundary. And allocating a depth correction value of 0.95 m for the point, and generating a dynamic parallax map to drive a large model to restore the shielded component.

The learner gesture demarcates the engine right wing area. The large model verifies that the difference between the height of the bottom surface Z=0.85 of the area and the height of the supporting surface Z=0.9 of the wing is 0.05 meter < 0.1 meter of the threshold value, and the judgment is valid. Based on the rules of 0.15 m diameter of adjacent hydraulic pipes and 0.05m part spacing, sensor candidate parameters including size [0.12,0.18] m, position center (1.1,0.9,0.85) and angle matching wing 30 degrees are generated.

And (3) carrying out triple verification on sensor parameters (1.1,0.9,0.85) of the large model screening position in a delimited area, wherein the distance |0.87-0.85|=0.02 m < safety threshold value 0.05 m between the large model screening position and the movable mechanical arm (1.1,0.9,0.87), judging collision, the bottom surface Z=0.85 < support surface Z=0.9 of the sensor, judging that the projection coverage rate is 0% suspended, and the included angle between shadow vectors (0, -1, 0) and a ceiling lamp (0, 1, 0) is 180 degrees and is more than 90 degrees. After the position is adjusted to (1.1,0.9,0.92), the position is verified by collision and gravity, but the illumination still fails. Finally, the parameter (1.1,0.9,0.92) +shadow direction optimization is adopted to pass the full verification.

Based on the verified parameters, generating a sensor model by the incremental nerve radiation field, inquiring a depth correction value of a mapping table by 0.95 meter, lifting Z coordinates of all vertexes from 0.92 to 0.95, and synchronously enhancing the glossiness of the surface metal texture. Generating displacement track by screwing sensor by students after renderingThe rotation angle is 45 deg.. And (3) updating a topological graph by using the system, wherein the height of a sensor is 0.98 m, the connection completion relation between the sensor and the turbine is rebuilt, and the physical parameters are synchronized to be 45 degrees at a position (1.1,0.9,0.98) angle, so that the intelligent closed loop which is interactively updated from the environment perception is completed.

Fig. 3 is a schematic structural diagram of a virtual reality interaction and content generation system based on a large model according to an embodiment of the present application, where, as shown in fig. 3, the system includes:

the acquisition module 31 is configured to acquire original point cloud data, synchronously acquire semantic segmentation tags, and align the original point cloud data with the semantic segmentation tags according to spatial coordinates to construct a topological graph including object positions, categories and spatial constraint relationships;

the calibration module 32 is configured to perform dynamic parallax calibration on the raw point cloud data, map the calibrated raw point cloud data to a unified space coordinate system, extract a boundary of the occlusion region, and generate a dynamic parallax mapping table;

the generating module 33 is configured to obtain coordinates of a blank area defined by a user in a virtual interface through the gesture recognition module, and generate a new object candidate parameter set according with a physical rule in combination with a spatial constraint relationship of the topological graph;

The verification module 34 is configured to analyze the topological graph and the dynamic parallax mapping table by using a large model, screen the candidate parameter set of the new object based on the blank region coordinates, simulate the collision response, the gravity effect and the illumination interference effect of the new object and the dynamic entity in the scene by using a physical consistency verification module, and reserve the verified parameter combination as a physical verification parameter;

And an output module 35, configured to generate geometry and texture information of a virtual object within the boundary of the occlusion region according to the physical verification parameter, optimize rendering accuracy of depth information of the geometry and the texture information in combination with the dynamic parallax map, and output the optimized virtual object to an interaction interface, and update the topology map and the physical verification parameter according to interaction behavior data of a user and the optimized virtual object.

The virtual reality interaction and content generation system based on the large model shown in fig. 3 may execute the virtual reality interaction and content generation method based on the large model shown in the embodiment shown in fig. 1, and its implementation principle and technical effects are not repeated. The specific manner in which the various modules and units of the large model-based virtual reality interaction and content generation system perform operations in the foregoing embodiments has been described in detail in connection with the embodiments of the method, and will not be described in detail herein.

In one possible design, a large model-based virtual reality interaction and content generation system of the embodiment shown in FIG. 3 may be implemented as a computing device, as shown in FIG. 4, which may include a storage component 41 and a processing component 42;

The storage component 41 stores one or more computer instructions for execution by the processing component 42.

The processing component 42 is configured to implement a large model-based virtual reality interaction and content generation method according to the embodiment described above with respect to fig. 1.

Wherein the processing component 42 may include one or more processors to execute computer instructions to perform all or part of the steps of the methods described above. Of course, the processing component may also be implemented as one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic elements for executing the methods described above.

The storage component 41 is configured to store various types of data to support operations at the terminal. The memory component may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.

Of course, the computing device may necessarily include other components as well, such as input/output interfaces, display components, communication components, and the like.

The input/output interface provides an interface between the processing component and a peripheral interface module, which may be an output device, an input device, etc.

The communication component is configured to facilitate wired or wireless communication between the computing device and other devices, and the like.

The computing device may be a physical device or an elastic computing host provided by the cloud computing platform, and at this time, the computing device may be a cloud server, and the processing component, the storage component, and the like may be a base server resource rented or purchased from the cloud computing platform.

The embodiment of the application also provides a computer storage medium which stores a computer program, and the computer program can realize the virtual reality interaction and content generation method based on the large model shown in the embodiment of the figure 1 when being executed by a computer.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.

The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.

It should be noted that the above-mentioned embodiments are merely for illustrating the technical solution of the present application, and not for limiting the same, and although the present application has been described in detail with reference to the above-mentioned embodiments, it should be understood by those skilled in the art that the technical solution described in the above-mentioned embodiments may be modified or some technical features may be equivalently replaced, and these modifications or substitutions do not make the essence of the corresponding technical solution deviate from the spirit and scope of the technical solution of the embodiments of the present application.

Claims

1. A virtual reality interaction and content generation method based on a large model, comprising:

generating geometric structure and texture information of a virtual object in the boundary of the shielding area according to the physical verification parameters, optimizing rendering precision of depth information of the geometric structure and the texture information by combining the dynamic parallax mapping table, outputting the optimized virtual object to an interactive interface, and updating the topological graph and the physical verification parameters according to interactive behavior data of a user and the optimized virtual object;

Generating geometry and texture information of a virtual object in the boundary of the shielding area according to the physical verification parameter, and optimizing rendering precision of depth information of the geometry and the texture information by combining the dynamic parallax mapping table, wherein the method comprises the following steps:

2. The method for generating virtual reality interaction and content based on a large model according to claim 1, wherein the analyzing the topological graph and the dynamic parallax mapping table by using the large model, screening the candidate parameter set of the new object based on the blank region coordinates, simulating collision response, gravity effect and illumination interference effect of the new object and dynamic entities in the scene by using a physical consistency verification module, and reserving the verified parameter combination as a physical verification parameter, comprising:

3. The method for generating virtual reality interaction and content based on a large model according to claim 1, wherein the step of obtaining, by the gesture recognition module, blank region coordinates defined by a user in a virtual interface, and generating a new object candidate parameter set conforming to a physical rule in combination with a spatial constraint relationship of the topological graph includes:

4. The method for generating virtual reality interaction and content based on a large model according to claim 1, wherein the outputting the optimized virtual object to the interaction interface and updating the topological graph and the physical verification parameters according to interaction behavior data of the user and the optimized virtual object comprise:

5. The method for generating virtual reality interaction and content based on a large model according to claim 1, wherein the performing dynamic parallax calibration on the raw point cloud data, mapping the calibrated raw point cloud data to a unified space coordinate system, extracting a boundary of an occlusion region, and generating a dynamic parallax mapping table includes:

6. The method for generating virtual reality interaction and content based on a large model according to claim 1, wherein the collecting original point cloud data, synchronously obtaining semantic segmentation labels, aligning the original point cloud data and the semantic segmentation labels according to space coordinates to construct a topological graph including object positions, categories and space constraint relations, comprises:

7. A virtual reality interaction and content generation system based on a large model, comprising:

The output module is used for generating geometric structures and texture information of virtual objects in the boundary of the shielding area according to the physical verification parameters, optimizing rendering precision of depth information of the geometric structures and the texture information by combining the dynamic parallax mapping table, outputting the optimized virtual objects to an interactive interface, and updating the topological graph and the physical verification parameters according to interactive behavior data of users and the optimized virtual objects;

8. A computing device comprising a processing component and a storage component, the storage component storing one or more computer instructions for execution by the processing component to implement a large model-based virtual reality interaction and content generation method of any of claims 1-6.

9. A computer storage medium, characterized in that a computer program is stored, which, when being executed by a computer, implements a virtual reality interaction and content generation method based on a large model as claimed in any of claims 1 to 6.