CN100407798C - 3D geometric modeling system and method - Google Patents

Abstract

The invention discloses a three-dimensional geometric modeling system, which includes: several video input devices for collecting video streams of designers' design actions; multiple single video stream visual analysis units for detecting motion areas and non-motion areas in video streams Area, estimate the direction and speed of object movement and predict the next position, calculate the edge contour of the moving object and estimate the contour features; the multi-video stream visual analysis unit is used for binocular stereo matching, 3D reconstruction, and object trajectory fitting and calculate the section of moving objects; the real-time interactive semantic recognition unit is used to process the output of the multi-video visual analysis unit to obtain human-computer interaction semantics; the 3D geometric modeling unit is used to obtain 3D geometric design modeling; the 3D model rendering unit, It is used to draw the geometric model to the video output device; the video output device is used to display the geometric shape of the object and the three-dimensional geometric shape designed by the geometric shape designer.

Description

3D geometric modeling system and method

technical field

The invention relates to a three-dimensional geometric modeling system and method, in particular to a computer-aided conceptual design-based real-time interactive three-dimensional geometric modeling system and method.

Background technique

At present, in the field of industrial product design, the computer-aided design (CAD) technology used in the later stage of detailed design has been quite mature. For example, in the automobile manufacturing industry, the entire automobile model design pipeline is almost entirely completed through computer aids. However, the conceptual design of the product is still achieved through hand-drawn sketches. Designers draw sketches according to their own creative intentions. The customer selects some styles that meet the needs from the dozens of sketches drawn, and gives back to the designers in combination with some personalized requirements. Designers then request further refinement of the sketch. After many iterations of submission and feedback, the conceptual design of a new model was finally determined. Obviously, flat hand-drawn sketches have no visual intuition, are not convenient for collaborative design in different places, and cannot provide direct data for accurate modeling of detailed design in the next step. From the perspective of domestic and foreign automobile industry development, the current automobile market tends to be saturated, the competition is increasingly fierce, and the automobile development cycle is shortened to one year. These new challenges push the automotive industry to take advantage of faster and more efficient design methods to speed up product development. Today, as the degree of informatization is getting higher and higher, designers hope to be able to freely express design intentions through automated auxiliary conceptual design tools, and realize automatic connection with other processes, so as to realize the complete informatization of the automotive design process.

Therefore, the computer-aided conceptual design geometric modeling method and device are studied, and the human geometric design concept is input into the computer system through natural human-computer interaction to establish an intuitive three-dimensional geometric shape. Such devices and methods have important application value to the automobile manufacturing industry.

Such a natural real-time interactive 3D geometric design system involves three technical fields, namely 3D geometric modeling technology, natural real-time human-computer interaction technology and visual computing technology supporting natural real-time interactive geometric design.

In terms of 3D geometric modeling technology, the design and expression of the surface shape can be done by using the commonly used method of determining the control vertices, such as the NURBS method, or by using the curved surface (or 3D shape, or curve) to generate through motion. The surface motion generation method has a wide range of applications, for example, the design and expression of slender strip-shaped object surfaces; the shape of an aircraft can also be decomposed into a union of strip-shaped surfaces (solids). The motion generation method is intuitive and simple, which simplifies many surface modeling tasks, so it is very popular among designers. This surface form, called swept surface, is more efficient and quality than other modeling methods in many occasions, rather than being limited to the modeling method of static objects such as push and pull of control vertices.

The research on 3D geometric modeling of surfaces generated by real-time interactive motion involves the theory of human-computer interaction and its realization methods. The human-computer interaction model is one of the key factors restricting the application to establish the control of the moving object by the human to achieve the purpose of modeling. Better human-computer interaction technology makes computers easier to use and can increase productivity. The HMGR project (Hand Motion Gesture Recognition System) of the Human-Computer Interaction Technology Laboratory at the University of Washington studies gesture recognition, and they use hidden Markov models for gesture recognition. Through this system, interactive user interface designers can implement a multi-channel input system, which converts hand movements in three-dimensional space into gesture symbols, so that voice input, static sign language and other forms of input can be combined. But the device uses data glove sensors and is limited to simple applications.

The heuristic 3D rendering interface developed by Brown University in the United States aims to improve the usability of the gesture interface and the command-based modeling system. In this system, the user conveys cues to the system as to what operation needs to be performed through the relevant geometric components in the high-brightness scene. The system infers possible user actions based on such clues, and displays them in the form of thumbnails. The user completes the editing operation by clicking the thumbnail. The manipulation cue mechanism enables the user to specify geometric relationships between graphical components in the scene. Multi-thumbnail manipulation hints can be used to solve the problem in the case of low discriminability of the manipulation model.

Japanese Patent Application No. 00118340 provides a device, a recognition method and a program recording medium for performing hand gesture recognition on complex hand images.

Japan's Nara Institute of Technology established a hybrid 3D object modeling system NIME (Immersive Modeling Environment). Inheriting the advantages of the traditional 2D GUI interface and 3D immersive modeling environment, the system combines the 2D/3D modeling environment into a whole by using an inclined rear-projection display device. On the display surface, the interaction is modeled using a two-dimensional GUI interface. At the same time, the use of field-sequential stereoscopic imaging and a pen-shaped input device with six degrees of freedom enables seamless conversion of 2D/3D modeling environments.

The 3D geometric modeling method of real-time interactive motion generated surface also involves the theory and technology of visual computing. The research goal of visual computing is to enable the computer to have the ability to recognize three-dimensional environmental information through two-dimensional images, that is, the shape reconstruction of three-dimensional objects, the spatial motion of objects and their geometric positions in space. Motion envelope, an important theoretical tool for establishing a motion envelope geometric model. Real-time dense stereo disparity matching has been a reality for nearly a decade. But until recently, systems capable of truly real-time processing required dedicated hardware such as digital signal processors (DSPs) or programmable gate arrays (FPGAs). For example, the stereo matching system of J. Woodfill and Von Herzen uses 16 Xilinx 4025 FPGAs to process 320*240 pixel images at a speed of 42 frames per second. P.Corke and Dunn use a similar algorithm, implemented with FPGA hardware, and can process 256*256 pixel images at a speed of 30 frames per second.

The Chinese patent with the patent application number 03153504 uses phase and stereo vision technology to project a grating onto the surface of an object, thereby realizing the three-dimensional surface profile measurement of the object.

In terms of hand-drawn sketch recognition tools for conceptual design, the hand-drawn sketch tool of the Computer Graphics Research Group of Brown University in the United States combines some characteristics of paper-pen sketches and computer CAD systems to provide rough 3D polyhedron modeling based on gesture interaction. This tool adopts the early traditional 2D interface concept, and provides users with the ability to sketch various three-dimensional basic elements in the form of hand-drawn sketches according to simple placement rules.

The University of Tokyo in Japan has designed an interactive free-form surface design tool based on hand-drawn sketches. The goal is to establish a simple and fast free-form model design system, such as the model design of small round animals or similar objects. The user interactively draws 2D strokes on the screen to construct 3D polygonal surfaces from 2D silhouettes.

The ARCADE research project of the Fraunfofer Institute for Computer Graphics in Germany uses a virtual design desktop to carry out free surface modeling technology research. The user stands in front of the virtual design table and uses the data glove gesture to realize the modeling of the free surface. The ARCADE system achieves efficient and precise modeling capabilities using 3D input devices. Interaction techniques include free-space object creation and creation based on other objects, implicit Boolean operations, 3D picking and rapid movement based on layout context modification, discrete manipulation, two-handed input, and more.

The patent application No. 00103458 discloses a device for writing and editing documents with a pen and editing gestures during word processing.

From the above implementation system, it can be seen that all aspects of technology involved in natural interactive 3D geometric modeling tools have been extensively studied driven by the application requirements of product concept design. However, there are still many problems in the above system that need to be solved. to be resolved. To sum up, the main defects include the following aspects:

Human-computer interaction lacks enough naturalness. Whether it is a virtual design table, a data glove, a 3D mouse, a tactile sensor, or an online hand-drawn sketch recognition device, the user needs to wear and directly touch the 3D input tool. This way of using complex tools and direct human-computer physical association will bring inconvenience to users. Unnatural design tools will affect the instant capture of designers' creative thinking and inspiration. One of the most direct evidences of this is that, despite the trial and error involved in the manual sketching process, it is still the primary design tool in the actual design process.

Insufficient directness of geometry interaction. Freehand sketches are required to translate concepts into 2D models and then into 3D models through recognition tools. The online hand-drawn sketching system binds these two processes together through equipment, which in turn increases the constraints on the designer. The data glove-based system realizes the operation of the model in the virtual scene through hand movements. The designer establishes and modifies the model in an indirect way. When the designer tries to change the shape of the model, he needs to operate the control points. Feedback on conceptual models conceived by designers. Design intentions are often constrained and interrupted by design tools such as data gloves, resulting in discontinuity in the design process and affecting design results.

The scope of application and the way of application are not flexible enough. Sketch systems, pen input, data gloves, force feedback devices, and virtual design desktops can only achieve a single way of external input. For example, during the design process, the designer may hope to obtain a certain silhouette of a specific object in a simple way, and establish a geometric model from the outline. Existing systems use special equipment to directly introduce users into the conceptual design process, which is still relatively difficult.

The implementation technology itself is not perfect. Hand-drawn sketches need to convert the 2D model into a 3D model, and there is a problem of recognition error rate. Due to the arbitrariness and great freedom of hand-drawn sketches, sample collection is very difficult, especially the semantics of sketches are more ambiguous and uncertain. It is neither possible to enumerate and identify targets with template definitions, nor to use predefined dictionaries. library to support its semantic interpretation; online hand-drawn sketches improve the accuracy of recognition, but in actual use, the interaction process of geometric design is often interrupted by system interaction, and the use efficiency is not high; there are also limitations in the use of sensors in data gloves. Issues such as spatial scope and positioning resolution.

System versatility is insufficient. The systems described above use specialized equipment and are therefore expensive. It not only increases product design cost, but also increases training cost and use cost. This undoubtedly raises the threshold for users to enter and limits the scope of application. The popularity of computers has been attributed to the reduction in cost and the increase in versatility.

Contents of the invention

The present invention is proposed in order to overcome the problems existing in the existing systems. The purpose of the present invention is to use a general, convenient and economical physical device, through a natural and real-time interaction method, according to the surface shape of the hands or/and hand-held objects in the three-dimensional space and the spatial trajectory, and use the geometric modeling method based on motion to complete the three-dimensional The creation, modification and editing of geometric shapes realize the geometric modeling of the conceptual design of the three-dimensional product shape.

In order to achieve this object, the invention provides a kind of three-dimensional geometric modeling method, comprising:

1. The video input step is used to collect video streams from multiple video input devices (101) distributed around the designer.

Two, single video stream visual analysis step, make the multiple video streams that above-mentioned collection video stream step gathers respectively through a single video stream visual analysis process, to detect the moving area and the non-moving area in the video stream, estimate the direction of moving object motion And speed and predict the next movement position and calculate the edge contour of the moving object and estimate the contour features.

3. The multi-video stream visual analysis step is used to receive the processing results of the above-mentioned several single video stream visual analysis, perform binocular stereo matching, and perform three-dimensional reconstruction and object motion trajectory fitting and calculation of moving objects based on the obtained contours and features Section, and provide the obtained object model, object motion trajectory and object cross-sectional outline to the real-time interactive semantic recognition processing step.

4. The real-time interactive semantic recognition step is used to process the output of the multi-video visual analysis step to obtain human-computer interaction semantics, and interpret the obtained human-computer interaction semantics by using the semantic definitions pre-stored in a semantic model storage unit.

5. The step of three-dimensional geometric modeling, which is used to process the output of the object model, object movement trajectory, object cross-section outline and real-time interactive semantic recognition step output by the multi-video visual analysis step, so as to obtain a three-dimensional geometric design model, and process the result stored in the 3D geometric model storage unit.

6. A 3D model rendering step, for rendering the 3D geometric model stored in real time in the 3D geometric model storage unit to the video output device.

7. A video output step, for displaying the three-dimensional geometric shape designed by the designer on the video output device.

In addition, Benming also provides a 3D geometric modeling system, including:

1. A plurality of video input devices are distributed around the designer to collect the video stream of the designer's design actions.

Two, corresponding to the single video stream visual analysis unit of each video input device, the multiple video streams collected by the above video input device are respectively processed by a single video stream visual analysis unit to detect motion in the video stream Areas and non-moving areas, estimate the direction and speed of the moving object and predict the next moving position, calculate the edge contour of the moving object and estimate the contour features.

3. The multi-video stream visual analysis unit is used to receive the processing results of the above-mentioned multiple single video stream visual analysis units, perform binocular stereo matching, and perform three-dimensional reconstruction and object motion trajectory fitting based on the obtained contours and features, and calculate The cross-section of the moving object, and provide the obtained object model, object movement trajectory and object cross-sectional outline to the real-time interactive semantic recognition unit.

4. The real-time interactive semantic recognition unit is used to process the output of the multi-video visual analysis unit to obtain human-computer interaction semantics, and interpret the obtained human-computer interaction semantics by using the semantic definitions pre-stored in a semantic model storage unit.

5. The three-dimensional geometric modeling unit is used to comprehensively process the output results of the multi-video visual analysis unit and the output of the real-time interactive semantic recognition unit, so as to obtain a three-dimensional geometric design model, and store the processing results in a three-dimensional geometric model storage unit middle.

6. A 3D model rendering unit, configured to use the 3D geometric model stored in real time in the 3D geometric model storage unit as input, and render the geometric model on the video output device.

7. The video output device is used to display the three-dimensional geometric shape designed by the designer.

According to another aspect of the present invention, a method for visual analysis and processing of a single video stream is provided, including: an image analysis method for processing video signals collected from a video input device to obtain feature videos with different resolution scales and different feature elements Streaming, which provides input for motion detection and stereo matching; real-time motion detection methods, which detect moving regions and non-moving background regions in video streams; motion estimation and prediction methods, which estimate the direction and speed of motion and predict the next motion position. The contour calculation method is used to calculate the edge contour of the moving object and estimate the contour features.

According to another aspect of the present invention, a method for visual analysis and processing of multiple video streams is provided, including: a stereo matching algorithm, which makes the data output from the motion detection of two videos go through stereo matching and disparity calculation to obtain the region segmentation and Depth information of moving objects; 3D model establishment method, which establishes a 3D stereo model through the depth data output by stereo matching; method of recovering stereo from the contour, according to the contour description data obtained by contour calculation, the 3D model of the moving body and its main projection features have been established. And the algorithm of recovering the shape from the contour, to obtain the three-dimensional shape of the moving object; the cross-section calculation method, according to the contour data and the motion trajectory, to establish the cross-sectional contour relative to the motion trajectory; the trajectory fitting method, to smooth the data obtained from motion estimation and prediction processing to obtain a smooth trajectory;

According to the fifth aspect of the present invention, a real-time interactive semantic recognition method is provided, including: a collision detection method, which determines the semantic type as operational semantics according to the trajectory and outline of the moving object, and then performs with the established three-dimensional geometric model Collision detection, to determine the location and method of collision; operation semantic analysis method, according to the result of collision detection, to determine the operation semantics such as the semantic operation object and operation type; interactive semantic analysis method, according to the input of the motion trajectory determination and cross-sectional contour processing, to obtain the analysis result of the interactive semantics; the voice semantic analysis method, to obtain the semantic analysis of the interactive voice from the voice analysis;

According to the sixth aspect of the present invention, a three-dimensional geometric modeling method is provided, including: a repetitive processing method for eliminating repetitive and overlapping motions of moving objects in video images; a motion processing method for eliminating Trembling and trembling of moving objects during the movement; including calculation methods, used to calculate the envelope surface generated by the movement of objects based on the motion trajectory and cross-sectional contours that have been eliminated and repeated; modeling editing methods, used for all The established 3D geometric model is modified.

According to the seventh aspect of the present invention, there is provided a video output device comprising: a display device for displaying the geometric model, spatial position and posture of the moving object, and displaying the established three-dimensional geometric modeling;

According to the eighth aspect of the present invention, a drawing method is provided, which draws a three-dimensional geometric design model on a display device, and draws a moving object's geometric model, relative position and posture on the display device.

Using this system or applying this method to carry out product shape concept design will produce beneficial effects in the following aspects.

1 By using cheap, non-dedicated physical equipment, reduce product design costs and broaden the scope of application.

2 The natural real-time interaction method facilitates the introduction of ordinary users into the open concept design cycle process, which helps to eliminate the passivity and one-sidedness in the geometric design of product appearance, and directly incorporates various factors into the product development, design and manufacturing process.

3 A design environment based on vision and general equipment, which facilitates the introduction of environmental features. Environmental characteristics are an important factor affecting product design, including environmental physical characteristics at the micro level and social and cultural characteristics at the macro level. The device of the invention allows to go out of the design room and carry out conceptual design in the environment of product use, which is a solution to realize the introduction of good environmental features.

4 Support 3D visualization design. The main problem of conceptual design is to model the design product. Modeling approaches for conceptual design range from formal definitions to high-level visual representations. At present, the most used model expressions include language model, geometric model, graphic model, object model, knowledge model and image model. The model closest to human thinking and reasoning is the image model. This invention is an important way to apply the visual thinking model to design practice.

5 Direct 3D digital products. The invention provides a direct three-dimensional digital product as a system output, which can obtain fast design evaluation feedback.

Description of drawings

1 is a block diagram of the structure of a three-dimensional geometric modeling system 100 according to a first embodiment of the present invention;

2 is a schematic diagram of a specific layout of cameras according to a first specific embodiment of the present invention;

Fig. 3 is the block diagram of single/multiple video stream visual analysis unit structure according to the first specific embodiment of the present invention;

Fig. 4 is the operation flowchart of the multiple video stream visual analysis unit according to the first specific embodiment of the present invention;

Fig. 5 shows the coordinate system of the stereo restoration operation of the present invention;

Fig. 6 is a flow diagram of semantic recognition according to the first specific embodiment of the present invention;

Fig. 7 is a flow chart of three-dimensional geometric modeling according to the first specific embodiment of the present invention;

8 is a schematic diagram of a computer system according to a first embodiment of the present invention;

FIG. 9 is a structural block diagram of a three-dimensional geometric modeling system 200 according to a second specific embodiment of the present invention;

Fig. 10 is a flowchart of semantic recognition according to the second specific embodiment of the present invention;

FIG. 11 is a structural block diagram of a three-dimensional geometric modeling system 300 according to a third embodiment of the present invention;

FIG. 12 shows a specific layout of cameras according to a third specific embodiment of the present invention;

Fig. 13 is an operation flow chart of the multi-video stream visual analysis unit according to the third specific embodiment of the present invention;

Fig. 14 is a schematic diagram of a computer system according to a third embodiment of the present invention.

Fig. 15 shows an example of an interactive gesture mode according to the first specific embodiment of the present invention

Detailed ways

First specific embodiment

FIG. 1 is a block diagram showing the structure of a three-dimensional geometric modeling system 100 according to a first embodiment of the present invention. As shown in FIG. 1 , the video input device 101 may be a digital camera, which is used to capture geometric design and modeling action images of a three-dimensional geometric modeling designer. In this embodiment, video input device 101 is made up of four digital cameras C01, C02, C03 and C04, and its specific layout mode is shown in Figure 2 in an embodiment of the present invention, and they are respectively placed in the concept designer Right, right front, left front and left four different positions, the height and posture from the ground should be suitable for the designer’s gesture expression, that is, the designer’s design actions should be fully recorded by the camera without affecting the design Division operations and other activities. A single video stream visual analysis unit 104 is provided corresponding to each digital camera. Each digital camera is directly connected to the corresponding single video stream visual analysis unit 104 of the digital camera according to a known connection mode through a common interface, and each single video stream visual analysis unit 104 collects from its corresponding video input device 101 respectively. Process the continuous video stream, detect the moving area and non-moving area in the video stream, estimate the direction and speed of the object’s motion and predict the next moving position, calculate the edge contour of the object and estimate the contour features, and provide the processing results to the multi-video stream visual analysis unit 105. The multi-video stream visual analysis unit 105 receives the processing outputs of the above four single video stream visual analysis units 104, including the detection result of the moving area, the moving direction and speed of the moving object, the contour and contour features of the object. Afterwards, the multi-video stream visual analysis unit 105 processes the input data, performs binocular stereo matching, performs 3D reconstruction and object trajectory fitting based on contours and features, and calculates cross-sections of moving objects. The output results generated by the processing of the above-mentioned single video stream visual analysis unit 105 are object models, object motion trajectories, and object cross-sectional outlines. The processing result of the multi-video stream visual analysis unit 105 is provided to the real-time interactive semantic recognition unit 106 as an input. The real-time interactive semantic recognition unit 106 is used to process the output of the multi-video visual analysis unit 105 to obtain human-computer interaction semantics, and interpret the obtained human-computer interaction semantics by using the semantic definitions pre-stored in the semantic model storage unit 110 . The 3D geometric modeling unit 107 comprehensively processes the output results of the multi-video visual analysis unit 105 and the output of the real-time interactive semantic recognition unit 106 to obtain a 3D geometric design model. The processing results of the three-dimensional geometric modeling unit 107 are stored in the three-dimensional geometric model storage unit 111 . The 3D model drawing unit 108 draws the geometric model on the video output device 109 based on the 3D geometric model stored in real time in the 3D geometric model storage unit 111 . The video output device 109 is used to display the geometric shape of the object and the three-dimensional geometric shape designed by the geometric shape designer.

The operation and system structure of the three-dimensional geometric modeling system of this embodiment will be described in detail below with reference to the accompanying drawings. The system consists of several basic working states such as system initialization, camera calibration, object model establishment, and motion geometry design.

<system initialization>

First describe the initial working state of the system. When the 3D modeling designer starts the 3D geometric modeling system 100, the system initialization process starts first. System initialization includes system initial parameter loading and establishment, background image statistical model establishment.

The system initialization process is based on the predetermined settings and the current system configuration. First, the system initialization working environment parameters are loaded. In this example, the system initial working environment parameters are shown in Table 1.

Table 1 Example initialization parameter list:

System working parameters

User ID user list Initial Model ID model list Camera parameter table Camera layout parameter table Image resolution scale factor Background reference image table working file directory data file directory

user table

User ID Initial Geometry ID Most recent work model ID

model sheet

Model ID model type model data structure pointer model file pointer

Camera parameter table

Number of cameras camera watch pointer Camera layout parameter pointer

camera table

ID of the first camera First camera parameter table Second camera ID Second camera parameter table ...

i-th camera parameter table

Camera layout parameter table

First camera layout parameters header pointer Second camera layout parameters header pointer ...

i-th camera layout parameter table

Baseline length to the 1st camera Baseline direction to camera 1 Baseline length to the 2nd camera Baseline direction to 2nd camera ...

After the initial working parameters are loaded, the video input device 101 (C01, C02, C03 and C04) continuously collects multiple images of the operating background, and provides these images to the single video stream visual analysis unit 104 corresponding to each video input device, It builds a statistical model of the initial background by processing these images. For example, in this embodiment, as shown in FIG. 3 , the single video stream visual analysis unit 104 further includes an image analysis unit 1041 for processing the video signal collected by the video input device 101 to obtain a statistical model of the background, namely Feature video streams with different resolution scales and different feature elements.

The image analysis unit 1041 in a specific embodiment of the present invention adopts the following method of establishing a background statistical model. For the background B _i corresponding to each camera C _i in the system, an initial background model M _i is established. For each pixel point p in _Bi , define μ _p as the expectation of the color value of the point, and σ _p ² is the variance of the color value distribution, which has the following formula:

${\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; no</span>}}\underset{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; t</span>}}$

${\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; no</span>}}\underset{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

Among them, h _p ^t is the color value of point p on the tth frame image. In this way, (μ _p , σ _p ² ) of each point p constitutes the background model of _Bi :

${\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; \{</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; B</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; \}</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

In addition, the system generates initial object models with simple, regular geometric surfaces, such as cuboids, spheres, etc., according to predetermined settings. By modifying the system settings, it is possible to choose what kind of predefined objects are generated or not to generate any initial objects.

<camera calibration>

When the system is used for the first time or the layout, position and attitude of the camera are changed, or the camera is replaced, camera calibration is required, that is, a camera calibration process of establishing camera parameters is started. In this working state, each camera in the system will acquire images and calculate the internal parameters and external parameters of each camera according to the camera calibration method known to those skilled in the art. If the system is not used for the first time and the layout, position and attitude of the camera have not been changed, and the camera has not been replaced, then there is no need to perform camera calibration to establish camera parameters.

<Object model creation>

After the above-mentioned camera calibration work process is completed, the designer can use the system described in this embodiment to perform three-dimensional geometric modeling with a hand or a hand-held object at an appropriate position in front of the camera group. The geometric model established according to the shape of the hand or the geometric shape of the held object is called the three-dimensional model of the object, or simply called the object model. The object model is a dynamic model that changes in real time with changes in the spatial position, posture, and shape of the hand and the held object. The invention uses the object model as a design tool to carry out three-dimensional geometric modeling design. At the same time, the simple hand-held object model itself can also be used as the initial model for 3D geometric modeling design.

<Sports Geometry Design>

After the system has gone through three operating processes of initialization, camera calibration, and object model establishment, it enters into the working state of motion geometry modeling design. Designers use the system described in this embodiment to use their hands or handheld objects at appropriate positions in front of the camera group, and input their own three-dimensional geometric design concepts into the three-dimensional geometric modeling system 100 through the shape and motion of the hand or handheld objects to obtain 3D geometry modeling results. The established 3D geometric model is stored in the 3D geometric model storage unit 111 in real time according to the prescribed file format and output to the video output device 109 such as a CRT or LCD display in real time. For example, it may be stored through a storage medium, such as a disk storage, or may be transmitted and stored through a computer network device or a removable storage device. In a specific embodiment, the format of a three-dimensional geometric model can be stored in the format shown in Table 2.

Table 2 Data structure of 3D geometric model

model sheet

Model code Model ID Model Type Encoding

The model type identifier Model attribute table Model parameter table version number the number of objects Object list pointer

object list

object type Object ID The parent object pointer child object pointer The number of point data number of sides Number of noodles point data structure pointer Point data length A pointer to a thread data structure Line data length Face data structure pointer face data length

point data structure

Point number (8 bytes) x (4 bytes) y (4 bytes) z (4 bytes)

edge data structure

side number (8 bytes)

Points (8 bytes) Point number data link head pointer

surface data structure

Face number (8 bytes) side number (8 bytes) The edge number data link head pointer

As shown in Table 2, the data structure of the 3D geometric model includes: model code, model ID, model type code, model type ID, model attribute table, model parameter table, version number, object quantity object list pointer and other data items. Among them, the model ID is used to uniquely identify the model. The model type code is used to indicate the type of the model. In this embodiment, the 3D geometric model generated as a design result and the object model generated as a design tool are stored using the same data structure. Therefore, the model type is used to distinguish between 3D geometric models and object models. At the same time, for the convenience of design, the system classifies the predefined object models and assigns a unique model type identification. The model attribute table defines the attributes of the geometric model, such as scale attributes, location attributes, and so on. The version number is used to indicate the version of the geometric model data structure. A model can consist of several objects, and the object list describes the properties of the object, including object type, object ID, parent object pointer, child object pointer, geometric data storage structure, and so on.

Designers can express the geometric design ideas of their conceptual designs to computers in various forms.

The first way is to directly carry out three-dimensional geometric design through the designer's hand. Specifically, the designer expresses the three-dimensional geometric design concept through the motion envelope of the hand and gestures.

The second way is to design three-dimensional geometric shapes by holding objects. Specifically, it can be divided into: ①The designer simply expresses the 3D geometric design concept to be established through the shape of the hand-held object. For example, if the designer wants to design a ball, he can place a ball in front of the camera, and the system will automatically create The three-dimensional geometric shape of the ball is used as the object model. Then, the designer copies the object model as a 3D geometric model output by issuing an order; ②The designer expresses the motion envelope 3D geometric model through the joint shape and motion of the hand-held object, for example, the designer holds a ball in front of the camera, The system will automatically establish the three-dimensional geometric shape of the ball as the object model. The designer holds the ball to move in an arc-shaped space, and the system builds a three-dimensional geometric design model formed by the ball's motion envelope according to the object model of the three-dimensional geometric shape of the ball and the trajectory of the ball, that is, a space Arc-shaped pipe and output.

The third way is to edit and modify the established 3D geometric model. Designers use hands or/and handheld objects to perform 3D geometric editing on existing 3D geometric models according to predetermined interactive semantic models, such as stretching, twisting, etc., to generate new 3D geometric models.

Of course, as those skilled in the art can easily understand, the designer can use the above three methods in combination to express the design concept.

Regardless of the above methods or combinations of methods, the 3D geometric modeling system described in this embodiment performs the following processing on the video data obtained through multiple cameras:

●Single video stream visual analysis

In this link, each single video stream visual analysis unit 104 respectively processes the continuous video stream collected from its corresponding video input device 101, detects the moving area and non-moving area in the video stream, and estimates the direction of object motion and speed and predict the next movement position, and calculate the edge contour of the object and estimate the contour features, and provide the processing results to the multi-video stream visual analysis unit 105. Specifically, it includes the following operations:

I. Image Analysis

As shown in FIG. 3 , each single video stream visual analysis unit 104 also includes an image analysis unit 1041 , and for each video input device 101 input, the image analysis unit 1041 will perform the following processing steps. Firstly, each image frame is acquired from each video input device 101, and then a layered structure is established according to the image resolution, and a layered image sequence is output.

In one embodiment of the present invention, the image analysis unit 1041 implements the establishment method of image layered data as follows: adopt a three-level pyramid structure to establish a set of image sequences { _ML , M _L-1 , M _L for the original image M _{L -2} }, where M _i-1 is the image obtained by reducing the resolution of M _i by half. And _ML is called the bottom layer of the pyramid, or the high-resolution layer; ML _-1 is called the middle layer of the pyramid, or the middle resolution layer; and _ML-2 is called the top layer of the pyramid, or the low-resolution layer. The image pyramid data structure is shown in Table 3.

Table 3 Image data cache queue description

  Maximum table length Head of line pointer end of line pointer Frame data pointer 1 Frame data pointer 2 Frame data pointer 3 Frame data pointer 4

Frame data pointer 5 Frame data pointer 6 Frame data pointer 7

frame data structure

Frame buffer sequence flag Frame type tag   Raw image data pointer Middle resolution image data pointer   Low-resolution image data pointer successor pointer   Forward pointer

During the processing, the time series image data is sequentially stored in the image processing buffer queue, and the queue length is 7 image frames. Queues are implemented using static round-robin tables.

II. Real-time motion detection

As shown in FIG. 3 , each single video stream visual analysis unit 104 further includes a motion detection unit 1042 for performing real-time motion detection on the processing results of each video analysis unit 1041 . The goal of motion detection is to detect motion areas and motion directions on images to obtain accurate motion area segmentation and motion area edges. The real-time motion detection unit detects the motion area through the image difference algorithm on the middle image of the multi-resolution image layer and obtains the motion direction through the optical flow method. In this embodiment, the motion detection unit 1042 performs the following operations on the layered data processing result from the image analysis unit 1041:

a) Background removal

For the image I _i ^t collected by each camera C _i at time t, the foreground area is extracted according to the following method:

Assuming the color value of point p in image I _i ^t is h _p , the image is binarized by the following formula:

${\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; 1</span>}& <span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; 3</span>}{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& <span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; 3</span>}{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}\end{array}\right.<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

All points p in the image I _i ^t where d _p is zero constitute the foreground area F _i .

b) Image difference calculation:

The difference image I _d (i, j) is a binary image d(i, j):

$\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; 0</span>}& <span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; \&epsiv;</span>}\\ \mathrm{<span\; class="notranslate">\; 1</span>}& \mathrm{<span\; class="notranslate">\; other</span>}\end{array}\right\}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\text{<span class="notranslate"> 5</span>}<span\; class="notranslate">\; )</span>$

Among them, f _k (i, j) is two adjacent frames of images in the time series image, and ε is a small positive number. In the difference image, pixel positions with a value of 1 indicate where motion occurred. From this, the motion area is obtained using this formula.

c) Calculation of plane motion parameters

The operation of calculating the motion speed c(u, v) on the projection plane according to the optical flow method will be described below. The basic calculation steps are as follows:

① For all pixels (i, j) on an image, estimate the initial value of optical flow c(i, j) = 0;

②Let k represent the number of iterations, and for all pixels (i, j), use formulas (6) and (7) to calculate the value:

${\mathrm{<span\; class="notranslate">\; u</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; u</span>}}}^{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>\frac{\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 6</span>}<span\; class="notranslate">\; )</span>$

${\mathrm{<span\; class="notranslate">\; v</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; v</span>}}}^{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>\frac{\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 7</span>}<span\; class="notranslate">\; )</span>$

in,

P(i,j)=f _x (i,j)u+f _y (i,j)v (8)

$\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 9</span>}<span\; class="notranslate">\; )</span>$

u and v represent the mean value in the u neighborhood and the v neighborhood, and the mean value can be calculated by using the image local smoothing operator. Value λ according to the size of the noise in the image. When the noise is large, take a smaller value; when the noise is small, take a larger value.

③ when

$\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; E.</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; \&epsiv;</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 10</span>}<span\; class="notranslate">\; )</span>$

When , the iterative process terminates; among them,

${\mathrm{<span\; class="notranslate">\; E.</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 11</span>}<span\; class="notranslate">\; )</span>$

III. Contour Calculation

As shown in FIG. 3 , the single video stream visual analysis unit 104 also includes a contour calculation unit 1043 , which obtains the hand area on the moving object area through a detection algorithm based on skin color. Remove the hand area to obtain the area holding the object. Based on this, the edge contour of the area is calculated. In this embodiment, the contour calculation unit 1043 performs hand edge acquisition and fine contour detection operations on the processing results from the motion detection unit 1042:

a) hand edge acquisition operation

In this embodiment, a strategy of fusion of motion information and skin color information is used for segmentation and contour detection of the hand region.

First describe the region acquisition based on motion information:

In this process, the camera is in a static state, and the color image sequence it shoots is a color image sequence composed of R, G, and B components. In the contour calculation unit 1043, define s=(x, y) to represent the image plane space coordinate system, t to represent the time coordinate system, and i to represent any component in the RGB space. Then I _t ⁱ represents the brightness image of component i at time t. Using t-Δt, t, t+Δt time continuous 3 frames of images to calculate the i-component moving image d t ⁱ at time _t is

${\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; min</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&Delta;t</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; ,</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&Delta;t</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 12</span>}<span\; class="notranslate">\; )</span>$

i=r, g, b (13)

Combining (r, g, b) components, the moving image d _t of the color sequence at time t can be obtained as

${\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; max</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{\mathrm{<span\; class="notranslate">\; g</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{\mathrm{<span\; class="notranslate">\; b</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 14</span>}<span\; class="notranslate">\; )</span>$

Finally, the moving image is smoothed and binarized to obtain the moving image

Next, the hand area recognition based on skin color detection is described:

We all know that the brightness of the same color is different under different distributions of light, but the perception of color is basically constant. The contour calculation unit 1043 utilizes the distribution of human skin color in the Luv space and the feature of brightness to detect skin color in the Luv color space. The operation steps of the contour calculation unit 1043 are as follows:

① Color space conversion. Convert RGB color space to Luv color space;

② The skin color detection result (or initial skin color feature) of the previous frame is used as the initial value, and the moving average algorithm is used to segment the moving area. The density distribution function of each color element in the current frame is used as a probability density function. The difference between the mean and the central value of the probability function defining the region is the average transition vector. The average transition vector is always along the direction of maximum probability density, so that the actual direction of maximum density can be found by searching. The specific calculation method is:

Let the color feature vector x _i of pixel p _i in the image be defined as:

x _i = (L, u, v) (15)

Wherein, L, u, v are the relative brightness of the image and u ^* , v ^* chromaticity coordinates. Let x ₀ represent the color feature vector of point p ₀ , and x _i represent the feature vector of point p _i in the window. In this embodiment, the window size is 7. Through the iterative calculation of the following two steps, the point where the density gradient is zero is obtained.

Calculate the mean moving vector m _h,G(x) (x)

${\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; (</span>{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\frac{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; (</span>{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\frac{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 16</span>}<span\; class="notranslate">\; )</span>$

where h is the color resolution and g(x) is the multivariate normal function

$\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; )</span>}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 17</span>}<span\; class="notranslate">\; )</span>$

Translate the kernel function G _{(x) by m h, G(x)} (x). Among them, x is the feature vector of the current window center, m _{h, G(x)} (x) is the difference between the weighted average weighted by G in the window and the window center. The above iterative process must converge and converge to the point where the density gradient is zero according to a smooth trajectory.

③ After determining the local maximum point, determine the feature class associated with the maximum point according to the local structure of the feature space, and obtain the actual position of the skin color in the space. Combine the detection results based on skin color and the detection results based on motion to obtain the edge contour of the hand.

b) Fine contour detection

As mentioned above, motion detection results on multi-resolution mid-level imagery have yielded lower-resolution region segmentation. The steps of the contour calculation unit 1043 of this embodiment to perform contour calculation on the original image to obtain a more accurate object contour according to the region segmentation result and the contour detection step will be described below. In this embodiment, the edge detection method of S.M.Smith and J.M.Brady is used to detect edges, and this method uses a 5*5 circular window template. The specific steps are:

①Establish an edge detection area according to the result of area segmentation, and the area is within a certain pixel width near the edge;

②Put the center of the window on each pixel position in the edge detection area, and calculate the number n(r ₀ ) of the points whose brightness is similar between the window center point r ₀ and other pixel points r in the window to determine whether the pixel is Image edge points. Calculate n(r ₀ ) using the following formula (18)

$\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; r</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 18</span>}<span\; class="notranslate">\; )</span>$

Among them, c(r, r ₀ ) represents the similarity between the brightness I(r) of the point r in the window and the brightness I(r ₀ ) of the window center point r ₀

$\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; e</span>}}^{<span\; class="notranslate">\; -</span>{<span\; class="notranslate">\; (</span>\frac{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>}{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 6</span>}}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 19</span>}<span\; class="notranslate">\; )</span>$

Among them, t represents the brightness threshold. Obviously, when the brightness difference between two points is smaller than t, c(r, r ₀ )=1. The center and direction of the edge can be calculated by n(r ₀ ), and the edge is refined by non-maximum value suppression.

Thus, the contour calculation unit 1043 obtains the contours of the edges of the hand and the object.

IV. Motion Estimation and Prediction

As shown in FIG. 3 , the single video stream visual analysis unit 104 further includes a motion estimation and prediction unit 1044 for tracking the motion trajectory of the contour calculated by the contour calculation unit 1043 . One embodiment of the present invention uses the center of the target contour as the focal point to track motion and obtain time-discrete motion trajectories. The result of motion tracking is a series of planar coordinates of the center of the contour with time labels, which is represented by a quadruple (x, y, t, i), where i is used to represent the camera. In order to improve the time efficiency of the system, this embodiment uses Kalman filter for motion prediction, and the prediction result is used as the input of the motion detection device to provide a pre-estimation of motion detection in the next frame.

Thus, the components and structures in the single video stream visual analysis unit 104 have been described, as well as their specific operations. The output of the single video stream visual analysis unit includes: a resolution-layered image sequence, a sequence of contour features of the image sequence, and a sequence of moving spatial points. The data structure of the output result of the single video stream visual analysis unit 104 is shown in Table 4.

Table 4 Single video stream analysis output data structure description

  Camera ID Frame sequence identifier Time stamp Frame type identification Image data attribute table Point feature data attribute table Regional feature data attribute table Edge feature data attribute table   Raw image data pointer   Medium resolution image data pointer   Low-resolution image data pointer The feature point data table pointer The feature structure data table pointer   Sports area data table pointer The background area data table pointer   Data Sheet for Motion Zone Edges

Single video analysis output sequence includes camera identification, frame sequence identification, time identification, frame type identification, image data attribute table, point feature data attribute table, area feature data attribute table, edge feature data attribute table, original image data pointer, middle resolution High-rate image data pointer, low-resolution image data pointer, feature point data table pointer, feature structure data table pointer, motion area data table pointer, background area data table pointer, motion area edge data table, etc. Wherein, the camera identifier is used to distinguish cameras of different systems. The frame sequence ID is the frame sequence number of the camera. The time stamp is used to record the acquisition time of the image frame. The attribute table of various characteristic data describes information such as the length of characteristic data for a class. All kinds of data pointers give the address of image data and feature data storage structure.

●Visual analysis of multiple video streams

The multi-video stream visual analysis unit 105 of the present embodiment will be described below with reference to FIGS. 3 and 4 . This unit accepts the output of all four single video stream visual analysis units 104, and performs the following processing:

I. Stereo matching

Four video input devices can be combined in pairs to form three input pairs. However, the multi-video stream visual analysis unit 105 of this embodiment includes a stereo matching unit 1051, which uses two pairs of dual-view inputs in three pairs of input pairs to construct a dual-view stereo match, thereby calculating the three-dimensional space coordinates of the object, that is, by stereo Match the plane coordinates (x, y) on the photographic plane to calculate the depth coordinate (z coordinate) relative to the camera, and then determine the three-dimensional space coordinates of the object according to the camera parameters. Depth reconstruction can be achieved by stereo matching of dual-view images. First, a target point is determined in the non-background area on the image obtained above in unit 1051, and a template window with a size of m×n is defined centering on the target point. In order to find the matching point of the target point on another image, a gray matrix with a size of (m+c)×(n+d) is defined in the possible matching area on another image as the search window, and the block matching algorithm is used to Perform image matching. The block matching algorithm is to move the template window in the search window and calculate the similarity matrix whose size is (c+1)×(d+1) according to the matching measure. The image block in the search window corresponding to the maximum (minimum) value in the similarity matrix is the best match of the template window.

The stereo matching unit 1051 of this embodiment calculates the similarity measure using the calculation formula of the sum of the absolute values of differences as shown in formula (20):

$\mathrm{<span\; class="notranslate">\; \&rho;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; I</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 20</span>}<span\; class="notranslate">\; )</span>$

Among them, I(u, v) and I'(u, v) are two-view images; m and n are the width and height of the matching window, and c+m and d+n are the width and height of the search area. In specific implementation, a faster implementation method than the naive implementation of the difference absolute value sum method can be obtained by successively selecting different thresholds.

The stereo matching unit 1051 of this embodiment uses a moving window in the possible target boundary area, that is, moves the position of the window on the input image to obtain more coverage areas, and selects the best matching position therefrom. In other areas the standard window is used.

II. SFS & FBR (Stereo Restoration from Silhouette and Feature-Based 3D Reconstruction)

As shown in FIG. 3 and FIG. 4 , the multi-video stream visual analysis unit 105 also includes a SFS&FBR unit 1052 and an object model storage unit 1056 . Among them, the SFS&FBR unit 1052 is a three-dimensional reconstruction unit based on contour restoration and feature-based, and the object model storage unit 1056 is used to store the reconstructed object model. The SFS&FBR unit 1052 is used to restore the object according to the contour calculation result of the single video stream visual analysis unit 104 and the established object model stored in the object model storage unit 1056, using the method of recovering shape from the contour and the feature-based target recognition algorithm surface shape. The unit performs contour recovery shape operation based on spatial variation, and the algorithm is as follows:

① For the spatial point P, calculate the point P′ on the image according to formulas (24) and (25), (x _h , y _h )

${\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; =</span>\frac{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&CenterDot;</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; h</span>}}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&CenterDot;</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; a</span>}}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; twenty\; four</span>}<span\; class="notranslate">\; )</span>$

${\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; =</span>\frac{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Center\; Dot;</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; v</span>}}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Center\; Dot;</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; a</span>}}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 25</span>}<span\; class="notranslate">\; )</span>$

in,

$\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; \&Center\; Dot;</span>{\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; h</span>}}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; +</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; a</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 26</span>}<span\; class="notranslate">\; )</span>$

$\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; v</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; \&CenterDot;</span>{\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; v</span>}}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; +</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; a</span>}}<span\; class="notranslate">\; \&CenterDot;</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 27</span>}<span\; class="notranslate">\; )</span>$

是摄像机光轴方向的单位向量，是像平面坐标系上水平方向的单位向量，

是像平面坐标系上垂直方向的单位向量。详细的坐标系统见图5，即W-XYZ世界坐标系和c-xy影像平面坐标系。P' is the projection of point P in the world coordinate system (X, Y, Z) on the image plane coordinate system. C is the vector from the origin of the world coordinate system to the projection center,

is the unit vector in the direction of the camera optical axis, is the unit vector in the horizontal direction of the image plane coordinate system,

is a unit vector in the vertical direction on the image plane coordinate system. The detailed coordinate system is shown in Figure 5, that is, the W-XYZ world coordinate system and the c-xy image plane coordinate system.

② If P′ is located in the background area of the image, then the spatial point P is the point to be removed; otherwise, P is retained;

③ Use a simple spatial octree algorithm to simplify the calculation process;

④Through multi-view contour clipping, several reconstruction results from different angles are obtained;

III. Object model building

As shown in FIG. 3 and FIG. 4 , the multi-video stream visual analysis unit 105 also includes a model building unit 1053 , which is used to calculate the three-dimensional coordinates of the object in the three-dimensional space by using the beam method after stereo matching is obtained. The specific operation of this unit is as follows:

Assume that a group of points X _j in the three-dimensional space is captured by a group of cameras whose matrix is P ⁱ . Use x ⁱ _j to mark the coordinates of the i-th spatial point on the image plane of the j-th camera, then the set of image coordinates x ⁱ _j is known, and the camera matrix P ⁱ and the spatial point X _j are obtained such that

P ⁱ X _j = x ⁱ _j (21)

If no further constraints are made on X _j or P ⁱ , the above reconstruction is a projective reconstruction, that is, the difference between X _j and the real reconstruction is an arbitrary three-dimensional projective transformation.

和真正投影到图像点

的空间点

，即Due to factors such as noise and matching errors, the equation x ⁱ _j =P ⁱ X _j will not be completely satisfied. It is usually assumed that this type of error satisfies a Gaussian distribution, and then the maximum likelihood solution is obtained. Here, it is necessary to estimate the projection matrix

and the real projection to the image point

point of space

,Right now

${{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; x</span>}}}^{\mathrm{<span\; class="notranslate">\; i</span>}}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; =</span>{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; P</span>}}}^{\mathrm{<span\; class="notranslate">\; i</span>}}{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; x</span>}}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; twenty\; two</span>}<span\; class="notranslate">\; )</span>$

And minimize the image distance between the reprojected point and the image point in each frame image, namely:

$\mathrm{<span\; class="notranslate">\; min</span>}\underset{{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; P</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; W</span>}{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; x</span>}}}_{\mathrm{<span\; class="notranslate">\; j</span>}}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; P</span>}}}^{\mathrm{<span\; class="notranslate">\; i</span>}}{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; x</span>}}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; ,</span>{{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; twenty\; three</span>}<span\; class="notranslate">\; )</span>$

where d(x, y) is the geometric image distance between homogeneous points x and y. Maximum likelihood values for X _j and ^Pi are estimated by adjusting the ray beams between each camera center and point in 3D space.

The initial values used in the above method are the projection matrix parameters obtained in the camera calibration process, the estimated value of the previous frame and the estimated value of the initial 3D reconstruction. Moreover, the Euclidean 3D reconstruction is obtained by constraining the camera parameters.

The 3D model representation obtained by the above method is a point cloud model. The point cloud model is then transformed into a geometric model represented by a subdivided surface. Since this conversion process is well known to those skilled in the art, it will not be repeated here.

The model building unit 1053 builds a three-dimensional model of an object, including a three-dimensional model of a hand, and a three-dimensional model of a hand-held object with a simple geometric shape. The simple geometric shape hand-held object can be an elastic steel bar with a certain shape. The simple geometric shape of the handheld object can also be a sphere, a cuboid, etc., and the cross-sectional shapes of these objects are used as the cross-section lines of the motion-generating surface.

According to the operation control command, the system can reconstruct the three-dimensional surface shape of the stationary object in the environment, and the reconstruction result is an object model that can be edited interactively, and the result is stored in the object model storage unit 1056 . Table 5 describes the data file format of the 3D geometric model.

Table 5 3D geometric model data file format

point data structure

Point number (8 bytes) x (4 bytes) y (4 bytes) z (4 bytes)

edge data structure

Edge number (8 bytes) Points (8 bytes) Point number (4 bytes) .....

surface data structure

Face number (8 bytes) Number of sides (8 bytes) Edge number (8 bytes) .....

IV. Trajectory Fitting

The motion speed and object outline obtained by the single video stream visual analysis unit 104 are the projection of the object's spatial motion on the camera's photographing plane, and are based on the motion trajectory coordinate sequence of the camera's image plane, so these plane coordinates are discrete in time and space. As shown in Figures 3 and 4, the multi-video stream visual analysis unit 105 also includes a trajectory fitting unit 1055 and a motion trajectory storage unit 1058, and the trajectory fitting unit 1055 uses a plurality of camera photographing planes and cameras that are spatially distributed Relative to the orientation parameters, the spatial continuous motion trajectory description is estimated by spatial coordinate intersection and curve fitting. The spatial point coordinate sequence is output and stored in the motion track storage unit 1058 . The unit operates as follows:

a) Spatial point calculation based on photography matrix

For a group of cameras C _k , k=1, . . . , n, where n is the total number of cameras. The absolute positioning parameters of each camera are P _i (x, y, z), and the absolute orientation parameters are R _i (α, β, γ). The plane coordinates at time t in the acquired camera image plane trajectory coordinate sequence are s(x, y, t, i), where x, y are the image plane coordinates, t is the time, and i is the camera number. The projection matrix M _i is uniquely determined by the external parameters and internal parameters of the camera.

${\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\left[\begin{array}{cccc}{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 11</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}& {\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 12</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}& {\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 13</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}& {\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 14</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}\\ {\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; one</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}& {\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; two</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}& {\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; three</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}& {\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; four</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}\\ {\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 31</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}& {\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 32</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}& {\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 33</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}& {\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 34</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}\end{array}\right]<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 28</span>}<span\; class="notranslate">\; )</span>$

At the same moment, the projection coordinates s(x, y, t, i) of the spatial point P(X, Y, Z) on each image plane satisfy equations (29), (30):

$<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 31</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 11</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 32</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 12</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; Y</span>}<span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 33</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 13</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 14</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 34</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 29</span>}<span\; class="notranslate">\; )</span>$

$<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 31</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; one</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 31</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; two</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; Y</span>}<span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 33</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; three</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; four</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; 34</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 30</span>}<span\; class="notranslate">\; )</span>$

According to the respective projection matrices and image plane coordinates of n cameras, 2n above equations can be constructed, and the spatial point coordinates (X, Y, Z) can be obtained by least square method.

b) Calculation of spatial points based on triangulation

Under the condition of obtaining multi-camera and camera positioning and orientation parameters, the spatial position of the hand can be determined by the principle of triangulation and the method of least squares. The specific operation method may be: project the position and attitude of the camera in the three-dimensional space onto three orthogonal coordinate planes. On each plane, each coordinate component of the spatial point is calculated separately. This method does not need to calibrate the internal parameters of the camera.

c) Trajectory fitting

Trajectories were fitted using cubic basis splines. And use the smooth condition to determine the boundary condition of spline fitting.

V. Section calculation

As shown in FIG. 3 and FIG. 4 , the multi-video stream visual analysis unit 105 further includes a section calculation unit 1054 and a section profile storage unit 1057 . The interface calculation unit 1054 determines the normal plane at each image frame on the trajectory according to the continuous motion trajectory obtained by the trajectory fitting unit 1054 . The projection of the object on the normal plane is the cross-sectional contour line of the moving object at each frame position.

So far, the functions of the main data processing units 1051 , 1052 , 1053 , 1054 , and 1055 of the multi-video stream visual analysis unit 105 have been described. The unit 105 also includes three data storage units, namely the object model storage unit 1056 , the cross-sectional profile storage unit 1057 and the motion trajectory storage unit 1058 .

The object model storage unit 1056 is a system permanent storage unit, which stores the three-dimensional geometric model of the moving object. Here, the permanent storage unit means that after the system restarts, the content stored in the storage unit remains unchanged. In this example, the data structure of the object model is shown in Table 2. All object models established in the system are stored in the object model storage unit 1056 . In block 1056, each object model is stored in a model table data structure. Each model table stores the model number, model ID, model type ID, model type ID, model attribute table, model parameter table, the number of objects in the model, and the storage table structure of each object. For rigid objects, each object corresponds to a model table. For hands and constrained deformable objects, several model tables will be stored in unit 1056 for each object and its basic deformations. For simple objects, the model table contains only one geometric object; for complex objects, multiple geometric objects are used in the model table to store the various components of the complex object. The model attribute table describes the basic attributes and extensible attributes of the model. The model parameter table stores the basic parameters of the model.

The cross-sectional profile storage unit 1057 is a system working storage unit, which stores the time-discrete cross-sectional profile of the moving object in its moving direction. In this embodiment, a circular queue is used to store a limited time series object contour data table. For example, the object outline data table of hundreds of frames before the current working time can be stored.

The motion track storage unit 1058 is a system work storage unit, which stores the space-time discrete track of the moving object. In this embodiment, a circular queue is used to store a finite time series object trajectory data table, and the trajectory data includes the spatial coordinates and posture of the geometric center of the object. For example, a data table of object motion trajectory corresponding to the cross-sectional profile of the object can be stored.

●Real-time interactive semantic recognition

The real-time interactive semantic recognition is completed by the real-time interactive semantic recognition unit 106, as shown in FIG. 1 and FIG. 6 . The real-time interactive semantic recognition unit 106 of the 3D geometric modeling system 100 accepts the output of the multi-video stream visual analysis unit 105, and reads the semantic model from the defined semantic model storage unit 110, analyzes and interprets the motion semantics and outputs a three-dimensional styling commands. The real-time interactive semantic recognition unit 106 includes a collision detection unit 1061, which is used to read the motion track of the motion track storage unit 1058, and detect the three-dimensional geometric design model in the three-dimensional geometric model storage unit 111 and the object model storage unit 1056 by a collision detection method. Collisions between object models. The collision detection result will be used as the input of the operation semantic analysis unit 1062 and the interaction semantic analysis unit 1063 . The real-time interactive semantic recognition unit 106 also includes an interactive semantic analysis unit 1063 and an operation semantic analysis unit 1062, which are based on the collision detection result of the collision detection unit 1061 and the objects stored in the object model storage unit 1056, motion track storage unit 1058, and semantic model storage unit 110. The data obtains the operation semantics of the moving object for the 3D geometric model. The semantic analysis result outputs a 3D modeling command and stores it in a 3D modeling command storage unit 1065 .

The real-time interactive semantic recognition unit 106 includes performing the following basic operations:

I. Collision detection

The collision detection unit 1061 detects the collision result between the three-dimensional geometric design model and the object model through a collision detection method according to the object model, cross-sectional profile and motion trajectory established by the multi-video stream visual analysis unit 105 . As mentioned above, the object model, cross-sectional profile and motion track are stored in the object model storage unit 1056, the cross-sectional profile storage unit 1057, and the motion track storage unit 1058, respectively. The collision detection unit 1061 provides a context environment for interactive semantic analysis, and at the same time provides visual feedback for the designer's operation and interaction with the geometric design model.

The collision detection results will be directly provided to the semantic analysis unit, and the specific analysis process will be described in detail below. The real-time collision detection between two objects is realized by using the AABB tree algorithm known to those skilled in the art, and details will not be repeated here.

II. Operation Semantic Analysis Unit

The operational semantic analysis unit 1062 is used to interpret operational semantics. For example, "select" operation semantics, "cancel" operation semantics, etc., in one embodiment of the present invention, when the hand model collides with the 3D design model and lasts for a certain period of time, it is judged as "select" operation semantics; When the hand model and the 3D design model are already in contact, if they continue to be in contact and remain stationary for a period of time, it is determined as the "cancel selection" operation semantics. The result of semantic parsing depends on the pose of the current object and the predetermined semantic model. In order to ensure the flexibility of the operation, the operation mode can include several types: the operation on the generated model and the operation on the interface menu and tool bar.

a) Operations on interface menus and toolbars

There are two ways to operate this type of interface: mouse and keyboard mode and virtual hand mode. The mouse and keyboard mode is the traditional plane graphic interface mode. The virtual hand mode is similar to the operation of the touch screen in the prior art, and the menu and tool bar of the interface are operated through the movement and click of the virtual hand. When the virtual hand moves to the operating area of the system graphical interface, the system automatically switches to the interface command operation mode, and the mouse and keyboard mode and the virtual hand mode are automatically switched by responding to instant input.

b) Operations on the generative model

Commonly used interaction semantics are set as floating balls in virtual 3D space. These floating balls are called operation balls, and each ball is defined as an operation. When the virtual hand catches an operation ball, it means that the virtual hand will start the operation. According to the operation context, the operation ball automatically disappears, emerges and changes its depth in the virtual three-dimensional space. The ball most likely to be selected will be in the easiest position for the virtual hand to grasp.

The three operations of virtual space operation, GUI interface operation and voice command operation are automatically switched through the context environment. When the virtual hand moves to the menu command area of the system interface, it will automatically switch to the virtual mouse mode. When the virtual hand moves to the drawing area, it transforms into a virtual three-dimensional space modeling operation mode.

III. Interactive Semantic Analysis Unit

The interaction semantic analysis unit 1063 determines the interaction semantics according to the collision detection result, the object model storage unit 1056 , the motion trajectory storage unit 1058 and the semantic model storage unit 110 . In this embodiment, interactive semantics are expressed through static interactive gestures. The interactive semantic analysis unit 1063 performs semantic analysis on static gestures using an appearance-based recognition method. That is, according to the predefined gesture template, the gesture semantic analysis is performed according to the method of template matching. In this embodiment, the semantic analysis based on template matching is carried out according to the following method:

First, the distance transformation is performed on the edge image, that is, the distance transformation is performed on the binary image to obtain an equal-sized distance map corresponding to the original edge image. The new value of each "pixel" in the distance map is the distance value, and the distance The transformation is defined as follows:

D(p)=min( _de (p,q)) q∈O (31)

Among them, d _e (p, q) represents the Euclidean distance between pixel points p and q, and O is the element set of the target object. The Euclidean distance transformation d _e (p, q) is defined as follows:

${\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; e</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\sqrt{{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 32</span>}<span\; class="notranslate">\; )</span>$

In order to reduce the amount of calculation, the square root operation is omitted, that is, formula (33) is used instead of formula (32)

d _e (p, q)=(p _x -q _x ) ² +(p _y -q _y ) ² (33)

After the above-mentioned distance transformation, the new value of each point in the formed distance map is the distance of the closest object pixel to the point in the original image.

In this embodiment, the one-way Hausdorff distance h(M, I) is used for model matching. M is the set of edge pixels of the selected gesture template, and I is the set of edge pixels of the image after edge extraction. When matching and recognizing, first implement Euclidean distance transformation on the image to be recognized after edge extraction to obtain a distance map, and then in the distance space, the template is translated and matched on the distance map. Correspondingly, h _j (M, I) (the subscript j is the number of translations) is taken as the maximum among several values of the edge pixel in the template at the corresponding position on the current distance map, which measures the current translation of the template The maximum mismatch between the position and the corresponding pixel on the edge image. The discriminant rule of the basic form of Hausdorff distance template matching is: take the minimum value of the above h _j (M, I) values obtained in all translation matching as the similarity between this template and the corresponding object of the template that may exist in the image. measure of degree. During the translation matching process, if several models appear to be very similar to the current image to be recognized, then the edge direction information is added to the translation matching judgment. At this time, it is judged to be at a certain translation point q (with a certain translation The pixel point at the lower left corner of the middle template is the reference point), the conditions for whether the template matches the corresponding position of the edge image are:

① When the j-th translation matching is performed, if the ratio of the points meeting the matching requirements in the template to the total points of the template pixels is R _j

${\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; Q</span>}}{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; \&Exists;</span>}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; \&tau;</span>}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; Ang</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; Ang</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; \&theta;</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \}</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 34</span>}<span\; class="notranslate">\; )</span>$

In the above formula, Q is the point set of the template formed by the pixels in the lower left corner of the template in several translations on the edge image; n( ) is the operation of taking the number of elements in the set; the function Ang(x) is the obtained edge The angle value of the pixel point x; τ is the given distance difference threshold; θ is the given direction radian difference threshold.

② Take P(k)=max(R _j ), where: k=1, 2, .

③ Take the gesture indicated by the template corresponding to max P(k) as the final recognition result.

In this embodiment, the modified Hausdorff distance is used for template matching, that is, the similarity of the template to the image to be recognized is obtained by averaging the h _i (M, I) obtained in several translations,

${\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 35</span>}<span\; class="notranslate">\; )</span>$

Wherein, N is the number of edge pixels in the template.

According to the method described above, the basic semantics of several interactive gestures can be determined. Figure 15 shows several examples of interactive semantic gestures.

●3D geometric modeling

As shown in Figure 7, the 3D set modeling system 100 also includes a 3D geometric modeling unit 107 for motion and gesture recognition based on the multi-video stream visual analysis unit 105, and semantic analysis based on the real-time interactive semantic recognition unit 106. As a result, the 3D modeling command creates a new 3D geometric design body, edits and replaces the existing 3D geometric body, and The three-dimensional geometric model storage unit 111 is read and stored in real time. The three-dimensional geometric modeling unit 107 includes the following processing steps:

I. Duplicate processing

The repetitive processing unit 1071 processes the repetitive actions generated during the movement of the object to eliminate repetitive and overlapping movements of the object.

II. Jitter processing

The shake processing unit 1072 performs smoothing and smoothing processing on the motion of the object, and eliminates the slight shake of the object's trajectory and posture.

III. Envelope calculation unit

The basic function of the envelope calculation unit 1073 is to solve the envelope differential equation based on the jitter-eliminated and repeated motion trajectory and cross-sectional profile, and then use the Runge-Kutta algorithm to calculate the envelope surface generated by the motion of the object. This unit outputs the envelope surface generated by the motion of the object.

IV. Shape editing unit

The modeling editing unit 1074 replaces the existing 3D geometric modeling model with the motion envelope generated by the envelope computing unit 1073, and performs smooth connection processing of the new and old meshes. The replacement process processes the connection and smoothing of joint points and joint surfaces according to constraints and modification rules, and modifies the established three-dimensional geometric model.

●Model drawing and display

Modifications to the three-dimensional geometric model will activate the model rendering process and output the rendering results to the display device 109 .

Thus, the technical solution according to the first specific embodiment of the present invention has been described. In addition, as shown in FIG. 8 , the three-dimensional geometric modeling system 100 of this embodiment may be composed of three general-purpose digital computers, two front-end computers 1201 and 1202 and one back-end computer 1203 . Every two digital cameras are connected to a front-end computer (1201, 1202), and the two front-end computers 1201 and 1202 are connected to the back-end computer 1203, and the back-end computer 1203 is connected to the video output device 109. Both the front-end computers 1201 and 1202 are provided with the single-video visual analysis unit 104 corresponding to the connected video input device 101 and the stereo matching unit 1051 of the multi-video visual analysis unit 105 . The back-end computer 1203 is provided with various components of the multi-video visual analysis unit 105 except the stereo matching unit 1051 , a real-time interactive semantic recognition unit 106 , a 3D geometric modeling unit 107 , and a 3D model rendering unit 108 . The storage system of the computer 1203 is provided with a semantic model storage unit 110 and a three-dimensional geometric model storage unit 111 . The above-mentioned components of the present invention can be realized by software, firmware, integrated circuits and the like.

Second specific embodiment

As shown in FIG. 9 , the three-dimensional geometric modeling system 200 according to the second embodiment of the present invention further includes an audio input device 202 and an audio recognition unit 203 . The audio input device 202 may be a general-purpose microphone and a sound card device for converting natural speech into digital speech signals. In this embodiment, the audio input device is used as an auxiliary input device to provide auxiliary interactive input in a multi-channel user interaction manner. After obtaining the audio input, the speech recognition unit 203 performs speech recognition, that is, realizes the function of converting the audio input into a restricted language. The speech recognition unit 203 only recognizes the pre-defined speech patterns, and the undefined speech input will be discarded. For example, basic speech can be recognized using the Microsoft Speech 5.X Microsoft Speech Recognition Engine. By setting the limited grammar settings based on XML files in the speech recognition engine, the system can only recognize the voice commands given in the grammar, effectively improve the recognition rate, and limit other voice inputs that may cause abnormal behavior of the system.

The speech recognized by the speech recognition unit 203 is provided to the real-time interactive semantic recognition unit 206 . The real-time interactive semantic recognition unit 206 includes a speech semantic analysis unit 2064 to perform semantic recognition on the speech recognized by the speech recognition unit 203 in addition to the collision detection, operation and interactive semantic analysis of the first embodiment. Whenever a recognized speech input is obtained from the speech recognition unit 203, the speech semantic analysis unit 2064 performs semantic interpretation on the obtained speech according to the current context of the system. An example of a semantic analysis file is shown below, wherein the semantic analysis grammar marked with <O></O> is an optional grammar. Users can modify the content of this part according to specific needs, reflecting the idea of personalized human-computer interaction.

<GRAMMAR LANGID="804">

<RULE NAME="WithPara" TOPLEVEL="ACTIVE">

<P>

<O>

<L>

<P>Please</P>

<P>I think</P>

</L>

</O>

<L>

<P PROPNAME="TYPE_RULEREF" VALSTR="CrePoint">Create Point</P>

<P PROPNAME="TYPE_RULEREF" VALSTR="CreLine">Create Line</P>

<P PROPNAME="TYPE_RULEREF" VALSTR="CreSurface">Create surface</P>

<P PROPNAME="TYPE_RULEREF" VALSTR="Delete">Delete</P>

<P PROPNAME="TYPE_RULEREF" VALSTR="Cancel">Cancel</P>

<P PROPNAME="TYPE_RULEREF" VALSTR="Done">End</P>

<P PROPNAME="TYPE_RULEREF" VALSTR="Edit">Edit</P>

<P PROPNAME="TYPE_RULEREF" VALSTR="SelePoint">Select Point</P>

<P PROPNAME="TYPE_RULEREF" VALSTR="SeleLine">Select Line</P>

<P PROPNAME="TYPE_RULEREF" VALSTR="SeleSurface">Select Surface</P>

<P PROPNAME="TYPE_RULEREF" VALSTR="zin">zoom out</P>

<P PROPNAME="TYPE_RULEREF" VALSTR="zout">zoom in</P>

<P PROPNAME="TYPE_RULEREF" VALSTR="Translation">Translation</P>

<P PROPNAME="TYPE_RULEREF" VALSTR="X">X coordinate value</P>

<P PROPNAME="TYPE_RULEREF" VALSTR="Y">Y coordinate value</P>

<P PROPNAME="TYPE_RULEREF" VALSTR="Z">Z coordinate value</P>

</L>

<L>

<P VALSTR="value">Value</P>

</L>

</p>

</RULE>

</GRAMMAR>

Meanwhile, the operations performed in the real-time interactive semantic recognition unit 206 are shown in FIG. 10 . The real-time interactive semantic recognition unit 206 in this embodiment includes a collision detection unit 2061 , an operation semantic analysis unit 2062 , an interactive semantic analysis unit 2063 , a voice semantic analysis unit 2064 and a 3D modeling command storage unit 2065 . The main working steps of the real-time interactive semantic recognition unit 206 are as follows:

The collision detection unit 2061 reads the motion trajectory of the motion trajectory storage unit 2058, and detects the collision between the 3D geometric design model of the 3D geometric design storage unit 211 and the object model of the object model storage unit 2056 through a collision detection method. The collision detection result will be used as the input of the operation semantic analysis unit 2062 and the interaction semantic analysis unit 2063 . The interactive semantic analysis unit 2063 and the operation semantic analysis unit 2062 obtain the data stored in the three-dimensional geometric model of the moving object according to the collision detection result of the collision detection unit 2061 and the data stored in the object model storage unit 2056, the motion track storage unit 2058, and the semantic model storage unit 210. The operation semantics and the generated 3D modeling commands are stored in the 3D modeling command storage unit 2065 . The speech semantic analysis unit 2064 obtains the semantic recognition result of the restricted language from the speech recognition result unit 203, obtains the interpretation of the speech semantics according to the semantic analysis definition of the semantic model storage unit 210, and generates a 3D modeling command to store in the 3D modeling command storage unit 2065 middle.

Here, the voice operation, as an auxiliary interaction channel, can be used as a command operation and a control operation in the drawing process during the human-computer interaction process of the system.

Except for the above description, according to the second embodiment of the present invention, the technical solutions implemented by the same or similar parts as the first embodiment are the same.

Third specific embodiment

FIG. 11 shows a structural diagram of a third specific embodiment of a three-dimensional geometric modeling system 300 applying the present invention. The video input device 301 shown in FIG. 11 is a digital video camera. In the present embodiment, the video input device 301 is made up of six digital cameras; corresponding to each digital camera, there is a single video stream visual analysis unit 304, and each digital camera is directly connected to the general-purpose computer equipment by connecting wires according to the usual connection mode The interface is processed by the single video stream visual analysis unit 304. The audio input device 302 is a general-purpose microphone and a sound card device for converting natural speech into digital speech signals. Different from the first specific embodiment, this embodiment not only adopts more camera devices, but also has different connection methods. Specifically, the cameras C01, C02, and C03 are located at the same height as the designer's hand movement, and the cameras C04, C05, and C06 are located above the designer. Due to the increased number of cameras in this configuration, the occlusion problem that may be caused by the motion process can be better avoided. Another difference is the change in the way the cameras are connected. As shown in Figure 13, the 6 cameras are divided into two groups on average, and the three cameras in each group are connected to form two image input pairs for dual-view matching, which are provided to the multi-video stream visual analysis unit 305 for depth acquisition and stereo match.

The six camera devices are placed in six different positions in front of the concept designer, front left, front right, top, top left and top right. The designer's design actions should be fully recorded by the camera without affecting the designer's operation and other activities. A specific layout implementation of a digital camera is shown in FIG. 12 .

In this embodiment, the computer system is composed of three general-purpose digital computers, as shown in FIG. 14 . Every two cameras are connected to a front-end computer 1901, 1902, and the two front-end computers are connected to a back-end computer 1903, and the back-end computer is connected to an audio input device and a display device.

The running process of this embodiment is basically the same as that of the first specific embodiment. When the three-dimensional modeling designer turns on the computer system of the device, the system initialization process is first started. A 3D geometric model of the simple object is then generated. The difference is that, compared with the visual analysis of multiple video streams in the first specific embodiment, the visual analysis of multiple video streams in this embodiment accepts the outputs of all six visual analysis of single video streams. Its processing process processes the video input according to the combination shown in Figure 14 .

The implementation of the three-dimensional geometric modeling system and method of the present invention has been described above. While the foregoing description has been made with reference to particular embodiments of the invention, it should be understood that various modifications may be made without departing from the spirit thereof. Therefore, the embodiments of the present disclosure are illustrative rather than restrictive in all respects, and the scope of the present invention is represented by the appended claims rather than the foregoing description, so all changes belonging to the equivalent meaning and scope of the claims are included in in.

Claims (14)

1. a 3 d geometric modeling method comprises

Video input step: gather the staff video flowing from being distributed in designer's a plurality of video input devices (101) on every side;

Single video stream visual analysis step: a plurality of video flowings that above-mentioned collection video flowing step is gathered are handled through a single video stream visual analysis separately, to detect moving region and the non-moving region in the video flowing, estimate moving object travel direction and speed and predict next movement position and calculate the edge contour of moving object and estimate contour feature;

Multiple video strems visual analysis step: the result that receives above-mentioned some single video stream visual analysises, carry out the binocular solid coupling, and carry out three-dimensional reconstruction and movement locus of object match and calculate the moving object cross section, and the cross section profile of the object model, movement locus of object and the object that are obtained is offered the semantic identification of real-time, interactive treatment step based on the profile that is obtained and feature;

The semantic identification step of real-time, interactive: object model, cross section profile and movement locus according to multiple video strems visual analysis step is set up, detect the collision between 3-D geometric model and the object model, determine position and mode that collision takes place; And determine operational semantics such as semantic operand, action type according to predetermined semantic model according to the result of collision detection; And, determine interaction semantics according to the semantic model that collision detection result, the object model of being stored and the movement locus of being stored and semantic model memory cell are stored in advance;

The 3 d geometric modeling step: the output to the semantic identification step of object model, movement locus of object, object cross section profile and real-time, interactive of many videos visual analysis step output is handled, thereby obtain the three-dimensional geometric design moulding, and result is stored in the 3-D geometric model memory cell;

Threedimensional model plot step: the 3-D geometric model of real-time storage in the 3-D geometric model memory cell is plotted on the video output device;

Video output step: the designed THREE DIMENSION GEOMETRIC MODELING of display design teacher on video output device.

2. 3 d geometric modeling method according to claim 1 is characterized in that, described single video stream visual analysis step also comprises:

The image analysing computer step: the vision signal to its pairing video input device collection is handled, and obtains to have the characteristic video stream of different resolution yardstick and different characteristic element;

Real time kinematics detects step: the characteristic video stream that is obtained according to described image analysing computer step detects moving region in the video flowing and the direction of motion to be cut apart and the edge, moving region to obtain accurately the moving region;

The profile calculation procedure: by obtain the zone of hand based on Face Detection, the zone of removing hand obtains the zone of hand-held object, thereby calculates the edge contour in zone on the moving object zone;

Estimation and prediction steps: estimate moving object travel direction and speed and predict next movement position.

3. 3 d geometric modeling method according to claim 1 is characterized in that, described multiple video strems visual analysis step comprises:

Three-dimensional coupling step: receive motion detection result, calculate Region Segmentation and the moving object depth information that obtains moving object through three-dimensional coupling and parallax from the motion detection step output in two described single video stream visual analysis steps;

SFS﹠amp; The FBR step: the profile according to the profile calculation procedure output in the described single video stream visual analysis step, calculate and recover the object surfaces shape;

The track fitting step: a plurality of video camera photographic planes and video camera relative orientation parameter that usage space distributes estimate space continuous motion track by space coordinates intersection, curve fit;

The cross section calculation procedure: the continuous motion track according to the outline data and the above-mentioned track fitting step of the profile calculation procedure in described single vision video analysis step output obtained, calculate the cross section profile of moving object;

The object model establishment step: the indicated depth information data of result by above-mentioned three-dimensional coupling step are set up object model.

4. 3 d geometric modeling method according to claim 1 is characterized in that, described 3 d geometric modeling step comprises:

Reprocessing step: the repetitive operation that produces in the object of which movement process is handled, eliminated repetition, the plyability motion of object;

The dithering process step: to the motion of object carry out smoothly, fairing processing, eliminate the small shake of movement locus of object and attitude;

Envelope calculation procedure:, calculate the enveloping surface that object of which movement produces according to movement locus and the cross section profile eliminating shake and repeat;

Moulding edit step: the 3-D geometric model of being set up is made amendment.

5. 3 d geometric modeling method according to claim 1 is characterized in that, described method also comprises:

Step from speech input device input voice command;

Speech recognition steps: the voice command of importing from speech input device is discerned according to predetermined speech model; Wherein

The semantic identification step of described real-time, interactive also comprises a voice semantic analysis step: the output of many videos visual analysis step and the output of speech recognition steps are handled, acquisition man-machine interaction semanteme, and the man-machine interaction semantic results that is obtained offered the 3 d geometric modeling step.

6. 3 d geometric modeling method according to claim 1 is characterized in that, the quantity of described video input device is 4, is distributed in designer's right-hand, right front, left front and left respectively and places to be fit to the position that designer's gesture is expressed.

7. 3 d geometric modeling method according to claim 1, it is characterized in that, the quantity of described video input device is 6, is distributed in designer's the place ahead, left front, right front, top, upper left side and upper right side respectively and places to be fit to the position that designer's gesture is expressed.

8. three-dimensional geometric mode building system comprises:

A plurality of video input devices: be distributed in the staff video flowing that the designer is used to gather designer's design action on every side;

A plurality of single video stream visual analysises unit: corresponding to each video input device, make a plurality of video flowings of gathering by above-mentioned video input device flow the processing of visual analysis unit separately through a described single video, to detect moving region and the non-moving region in the video flowing, estimate moving object travel direction and speed and predict next movement position and calculate the edge contour of moving object and estimate contour feature;

Multiple video strems visual analysis unit: the result that is used to receive above-mentioned a plurality of single video stream visual analysises unit, carry out the binocular solid coupling, and carry out three-dimensional reconstruction and movement locus of object match based on profile that is obtained and feature, calculate the moving object cross section, and the cross section profile of the object model, movement locus of object and the object that are obtained is offered the semantic recognition unit of real-time, interactive;

The semantic recognition unit of real-time, interactive: be used for object model, cross section profile and the movement locus set up according to multiple video strems visual analysis unit, detect the collision between 3-D geometric model and the object model, determine position and mode that collision takes place; And determine operational semantics such as semantic operand, action type according to predetermined semantic model according to the result of collision detection; And, determine interaction semantics according to the semantic model that collision detection, the object model of being stored and the movement locus of being stored and semantic model memory cell are stored in advance;

3 d geometric modeling unit: be used for the output result of many videos visual analysis unit and the output of the semantic recognition unit of real-time, interactive are carried out integrated treatment, thereby obtain the three-dimensional geometric design moulding, and result is stored in the 3-D geometric model memory cell;

Threedimensional model drawing unit: be used for the 3-D geometric model of 3-D geometric model memory cell real-time storage is plotted to video output device;

Video output device: be used for the designed THREE DIMENSION GEOMETRIC MODELING of display design teacher.

9. three-dimensional geometric mode building system according to claim 8 is characterized in that, described single video stream visual analysis unit also comprises:

Image analysing computer unit: be used for the vision signal of its pairing video input device collection is handled, obtain to have the characteristic video stream of different resolution yardstick and different characteristic element;

Real time kinematics detecting unit: be used for the characteristic video stream that obtained according to described image analysing computer unit and detect the moving region of video flowing and the direction of motion and cut apart and the edge, moving region to obtain accurately the moving region;

The profile computing unit: be used for passing through to obtain based on Face Detection on the moving object zone zone of hand, the zone of removing hand obtains the zone of hand-held object, thereby calculates the edge contour in zone;

Estimation and predicting unit: be used to estimate moving object travel direction and speed and predict next movement position.

10. three-dimensional geometric mode building system according to claim 8 is characterized in that, described multiple video strems visual analysis unit comprises:

Three-dimensional matching unit: be used for receiving the motion detection result that two described single video flow the motion detection unit output of visual analysis unit, three-dimensional coupling of process and parallax calculating obtain the Region Segmentation and the moving object depth information of moving object;

SFS﹠amp; FBR unit: be used for profile result of calculation, recover the object surfaces shape according to the profile computing unit output of described single video stream visual analysis unit;

Track fitting unit: be used for a plurality of video camera photographic planes and video camera relative orientation parameter that usage space distributes, estimate space continuous motion track by space coordinates intersection, curve fit;

Cross section computing unit: be used for calculating the cross section profile of moving object according to the outline data of the profile computing unit output of described single video stream visual analysis unit and the continuous motion track that above-mentioned track fitting unit is obtained;

Object model is set up the unit: be used for setting up object model by the indicated depth information data of the result of above-mentioned three-dimensional matching unit.

11. three-dimensional geometric mode building system according to claim 8 is characterized in that, described 3 d geometric modeling unit comprises:

Reprocessing unit: be used for the repetitive operation that the object of which movement process produces is handled, eliminate repetition, the plyability motion of object;

The dithering process unit: be used for to the motion of object carry out smoothly, fairing processing, eliminate the small shake of movement locus of object and attitude;

Envelope computing unit: be used for calculating the enveloping surface that object of which movement produces according to movement locus and the cross section profile eliminating shake and repeat;

Moulding edit cell: be used for the 3-D geometric model of being set up is made amendment.

12. three-dimensional geometric mode building system according to claim 8 is characterized in that, described system also comprises:

Speech input device: be used to import voice command;

Voice recognition unit: be used for according to predetermined speech model to discerning from the voice command of speech input device input; Wherein

The semantic recognition unit of described real-time, interactive is used for the output of many videos visual analysis unit and the output of voice recognition unit are handled, and obtains the man-machine interaction semanteme, and the man-machine interaction semantic results that is obtained is offered the 3 d geometric modeling unit.

13. three-dimensional geometric mode building system according to claim 8 is characterized in that, the quantity of described video input device is 4, is distributed in designer's right-hand, right front, left front and left respectively and places to be fit to the position that designer's gesture is expressed.

14. three-dimensional geometric mode building system according to claim 8, it is characterized in that, the quantity of described video input device is 6, is distributed in designer's the place ahead, left front, right front, top, upper left side and upper right side respectively and places to be fit to the position that designer's gesture is expressed.

Images