CN101976463A - Manufacturing method of virtual reality interactive stereoscopic book - Google Patents

Abstract

The invention provides a manufacturing method of a virtual reality interactive stereoscopic book, which comprises the following steps: dividing the book into three areas: (1) a traditional word and picture area; (2) a mark area; and (3) an actual operation scene capture area; capturing a specific mark in the mark area by a common web camera, and calculating the spatial location and direction of the mark; and mapping the mark into a virtual three-dimensional model, and combining the virtual three-dimensional model with the captured real scenes to obtain a virtual reality interactive image. In the method, flicker and jitter of the virtual three-dimensional model are reduced by the modes of capture delay and matrix smoothing. The manufacturing method is applied to the book publishing field and has the advantages of simple equipment and natural interbehavior, thus enhancing sensory experiences of readers without adding operation complexity.

Description

A virtual and real fusion interactive three-dimensional book production method

technical field

The invention belongs to the fields of book publishing and computer graphics, and specifically relates to an interactive three-dimensional book production method based on virtual reality fusion of augmented reality technology, which is applied to the publishing of educational books with rich multimedia information.

Background technique

Augmented Reality (AR) is an extension of Virtual Reality (VR) technology. Different from virtual reality technology, augmented reality has two characteristics: first, conceptually, the video environment in the digital space is considered to be the real environment; second, technically, the fusion of virtual and real is realized through the seamless combination of images and geometric models. Augmented reality technology has been researched for more than ten years. At the earliest, it is generally necessary to wear a see-through head-mounted display, so that users can directly superimpose virtual objects when observing real physical objects. With the development of technology, in recent years, augmented reality technology has begun to use two-dimensional code-like markers to locate. After the camera captures the marker image, the software calculates the content.

At present, researchers in the field of education are paying more and more attention to the application of augmented reality technology in the field of education, and have designed many systems or platforms. Kaufmann (2003) used augmented reality technology to establish a cooperative teaching interaction between teachers and students. The system can present simple points, lines, planes and geometry, and has the function of Boolean operations. Teachers can easily express changes in geometric shapes and The relationship in space, and it is easier for students to grasp complex space concepts in this three-dimensional environment that combines virtual and real. Dünser and Horneker (2007) used fables as content, adding 3D characters, sounds and interactive props to observe how children aged 5-7 use augmented reality systems for interaction and cooperative learning. Participating children use the labeled interactive props to complete small tasks related to the content of the story while reading the story. Studies have found that augmented reality environments attract children's attention and make them willing to keep trying to complete tasks when they are active. Mulloni (2008) designed a "cow and UFO" game, through the bluetooth device on the mobile device and the tag card in the space to conduct a group confrontation game. Priestnall (2009) is a methodological article on the use of AR in education. It uses aerial photos or radar maps, superimposed contour lines for 3D mapping, and reproduces the appearance of glaciers before retreat, so that students' understanding of knowledge can be transformed from abstract Concepts are transformed into real concepts. Domestic research on augmented reality has gradually become a hot spot in recent years, but there is little involvement in AR technology in the field of book publishing. The domestic book publishing and educational technology fields have not systematically discussed the feasibility and empirical cases of AR virtual learning environment. There is no AR virtual learning environment with independent intellectual property rights for in-depth research.

[1]. Kaufmann, H., Schmalstieg, D. (2003). Mathematics and geometry education with collaborative augmented reality [J]. Computers & Graphics, 27(3): 339-345.

[2]. Dünser, A., Hornecker, E. (2007). An observational study of children interacting with an augmented story book [C]. Proceeding of Edutainment: 305-315

[3].Mulloni, A., Wagner, D., & Schmalstieg, D.(2008).Mobility and Social Interaction as Core Gameplay Elements in Multi-Player Augmented Reality[C].The 3rd international conference on Digital Interactive Media inEntertainment and Arts: 472-478

[4].Priestnall, G.(2009).Landscape Visualization in Fieldwork.Journal of Geography in Higher Education[J].33(S1):104-112.

Contents of the invention

The technical problem to be solved by the present invention is to overcome the disadvantages of weak interactivity and weak expressiveness of current books, and provide a method for making interactive three-dimensional books with fusion of virtual and real. The production process of this kind of book is simple, the required equipment is not complicated, and the operation is also very convenient. It is suitable for displaying three-dimensional interactive effects that are difficult to present on flat books.

The technical solution adopted by the present invention to solve the technical problem is: a method for producing an interactive three-dimensional book combining virtual and real, which is characterized in that it includes the following steps:

(1) Divide the book into three areas: ①Traditional text and picture area, ②Mark area, ③Practical operation scene screenshot area;

(2) Carry out three-dimensional modeling according to the content in the traditional text picture area, make virtual three-dimensional model;

(3) Capture the specific mark printed in the marked area on the book in real time through the camera, calculate the spatial position and direction of the specific mark, and map it to the established virtual three-dimensional model, so that the position and direction of the virtual three-dimensional model Generate corresponding changes, and combine the changed virtual 3D model with the original scene captured by the camera in real time to obtain an interactive image of virtual and real fusion;

(4) Place the image obtained in step (3) in the screenshot area of the actual operation scene.

Further, the camera used in the virtual-real scene superposition method in the step (3) is an ordinary network camera; a plurality of different specific markers are simultaneously captured, a plurality of corresponding virtual three-dimensional models are mapped, and the same virtual-real fusion It is displayed in the screen scene; set up a delay mechanism, and use a delay variable timeDelay to record the duration that the marker cannot be captured by the camera. Only when the value exceeds the preset threshold value will the marker be considered not captured by the camera, reducing the rate due to the low refresh rate of the camera. The impact of high or light changes on the inaccurate capture of markers, so that the composite image does not flicker; set a smooth matrix list, calculate the average value of the matrix in the list as the change value of the virtual 3D model position and direction, and reduce the reader's hand shake Slight changes cause virtual objects to have the effect of shaking, making composite images look more stable and natural.

The beneficial effect of the present invention compared with prior art:

(1) The method of the present invention has simple equipment, and only needs to be equipped with a notebook computer and a webcam for ordinary video chatting. The reader does not need additional operation command buttons, and can see it on the computer screen as long as the specific mark on the book is aimed at the camera. The three-dimensional interactive effect of the fusion of virtual and real, this kind of interaction is close to the natural reading behavior, while improving the reader's sensory experience without increasing the complexity of the operation.

(2) The present invention can recognize multiple marks at the same time, generate multiple virtual three-dimensional models in one scene, and enable readers to obtain a richer virtual-real fusion scene experience.

(3) The present invention reduces the flickering and jittering of the virtual three-dimensional model through the method of capturing delay and matrix smoothing, so that readers can obtain a more natural and real reading experience.

(4) The mark recognition, three-dimensional graphics display and interactive effects of the present invention are all realized by using Java language. Due to the cross-platform characteristics of Java language itself, the background program of this book can be transplanted in different operating systems.

Description of drawings

Fig. 1 is the making process figure of the present invention;

Fig. 2 is the schematic diagram of mark and additional logo of the present invention;

Fig. 3 is a layout structure diagram of books of the present invention;

Fig. 4 is a flow chart of marker acquisition, calculation and fusion of virtual and real scenes in the present invention.

Detailed ways

The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

An interactive three-dimensional book with virtual and real fusion of the present invention, the production process is as follows:

Figure 1 shows. The specific steps are as follows:

(1) Make the mark pattern and print the book

First of all, different and unique marking patterns are made according to the number of contents that the book needs to use for virtual three-dimensional display. In order to ensure that the camera can capture the marks sensitively, the marks should be made simple, the pattern color contrast is high, and the mutual distinction between different marks is obvious. A mark with a black background on the outside, a white background on the inside, and a black font color can be used, and the surrounding can be supplemented with a decorative logo pattern, as shown in Figure 2.

The layout of the book content is shown in Figure 3. The traditional text and pictures are on the left, accounting for 2/3 of the entire page. Readers can intuitively see the difference between this book and traditional books in an environment without a camera.

(2) Capture markers, make a virtual 3D model and superimpose it on the video captured in real time

The system first displays the real scene image acquired by the camera as the bottom layer, and then calculates the 3D position of the marker in the real space according to the internal and external parameters of the camera and the 3D registration algorithm obtained through calibration, and then finds out in the pre-built virtual 3D model library. One-to-one corresponding virtual model of the marked block, and project it on the projection plane of the camera according to the projection matrix of the camera at the three-dimensional position of the mark, and compare the image of the virtual three-dimensional model on the projection plane with the image of the real space on the projection plane The images are synthesized to form the final combined virtual and real scene for output, such as

Figure 4 shows. Specific steps are as follows:

1) Use the camera to capture the real scene and obtain color image information;

2) Convert the collected color image into a binary black-and-white image according to the set threshold, and perform connected domain analysis on the binary image;

3) Find out all the quadrangular regions as candidate matching regions, match each candidate region with the template in the template library, if a match is generated, it is considered that a mark has been captured;

4) Calculate the position and attitude of the camera relative to known marks according to the position information of the camera and the positioning marks in the real scene;

5) Draw the virtual object on the viewing plane according to the affine transformation matrix, and finally merge it with the video of the real scene, and project it on the common display screen.

(3) Set the delay mechanism

Virtual objects in the composite video stream can be seen using the method described above. However, in actual operation, if the reader operates too fast but the camera refresh rate is not high, or the ambient light is poor, or even sometimes the reader accidentally covers the small edge of the mark, the camera will not be able to capture the mark instantaneously. In this case, the virtual three-dimensional model on the synthesized video image will temporarily disappear from time to time, that is, the virtual three-dimensional model will "flicker", which affects the viewing effect of readers. The present invention adopts the method of setting a delay mechanism, by setting a delay variable timeDelay to record the continuous time that the mark cannot be captured by the camera, only when the value exceeds the preset threshold value is it considered that the mark has not been captured by the camera, and the virtual 3D is not displayed at this time Model. This measure reduces the flickering of the virtual 3D model, allowing readers to obtain a more natural viewing effect.

(4) Set smooth movement

When the reader operates the book, the position and orientation of the mark changes slightly due to hand shaking and other behaviors, and this change is captured by the camera to recalculate the position and orientation of the virtual 3D model, so that the reader can see it on the screen. The virtual three-dimensional model of the website keeps shaking slightly, making readers feel unreal. The present invention sets a matrix list smoothingMatrixList, stores the change matrix of the position and orientation of the marker captured by the camera into the smoothingMatrixList in order, and calculates the average value of the matrix stored in the list when the number of matrices exceeds the preset threshold matrixavg is used as the change matrix of the virtual 3D model, and at the same time, the matrix added first in the smoothingMatrixList is removed. This measure smoothes the translation or rotation of the virtual 3D model. The algorithm looks like this:

ArrayList<Matrix4d>smoothingMatrixList //Set the list type to Matrix4d

MATRIX_MAX_NUM＝10 //Set the maximum number of matrices contained in the matrix list

matNum=0

...

if(matNum==MATRIX_MAX_NUM){

smoothingMatrixList.remove(0) //Remove the first matrix added in the matrix list

matNum=matNum-1

}

smoothingMatrixList.add(currentMat)//Add the current matrix to the matrix list

matNum=matNum+1

...

public Matrix4d get(){//Get the call method function of the smooth moving matrix

for(Matrix4d mat: smoothingMatrixList)

matrixavg.add(mat);//Get all matrix sums in the matrix list

matrixavg.mul(1.0/matNum);//Get the average matrix

return matrixavg;

}

The smooth change matrix of the virtual three-dimensional model can be obtained by calling the get() method function in the above algorithm.

Claims (5)

1. an interactive three-dimensional book production method of virtual and real fusion, is characterized in that comprising the following steps: (1) Divide the book into three areas: ①Traditional text and picture area, ②Mark area, ③Practical operation scene screenshot area; (2) Carry out three-dimensional modeling according to the content in the traditional text picture area, make virtual three-dimensional model; (3) Capture the specific mark printed in the marked area on the book in real time through the camera, calculate the spatial position and direction of the specific mark, and map it to the established virtual three-dimensional model, so that the position and direction of the virtual three-dimensional model Generate corresponding changes, and combine the changed virtual 3D model with the original scene captured by the camera in real time to obtain an interactive image of virtual and real fusion; (4) Place the image obtained in step (3) in the screenshot area of the actual operation scene. 2. The interactive three-dimensional book production method of virtual and real fusion according to claim 1, characterized in that, in said step (3), the camera used to capture the specific mark is a common network camera. 3. the interactive three-dimensional book making method of virtual-real fusion according to claim 1, is characterized in that, in described step (3), captures a plurality of different specific marks simultaneously, maps a plurality of corresponding virtual three-dimensional models, and Displayed in the same virtual-real fusion screen scene. 4. the interactive three-dimensional book making method of virtual and real fusion according to claim 1, is characterized in that, in described step (3), a delay variable timeDelay record mark can not be captured by camera and is set to the continuous time of recording, only this value A marker is considered not captured by the camera if it exceeds a pre-set threshold. 5. the interactive three-dimensional book making method of virtual reality fusion according to claim 1, is characterized in that, in described step (3), a smooth matrix list is set, calculates the average value of matrix in the list as virtual three-dimensional model position and The value of the change in direction.

Images