CN106249607A - Virtual Intelligent household analogue system and method - Google Patents

Abstract

The invention discloses a virtual smart home simulation system, which includes a home equipment layer, a storage service layer and an application layer. Applying virtual reality technology to the design of smart home not only satisfies the immersive experience and interaction of smart home users, but also realizes real-time control of smart devices by virtual reality experiencers. The smart device control interface provided in the virtual scene enables the virtual scene in the 3D model to respond synchronously with the real home scene, meeting the basic functional requirements of traditional smart home control and display. At the same time, users can roam and walk in the 3D scene from multiple perspectives. Such a control method is more realistic and intuitive, and the operation interface is also more interactive. The platform can provide Internet of Things users with a more natural way of obtaining object information and virtual interactive experience, meeting the requirements of Internet of Things related applications for real-time, accuracy and stability.

Description

Virtual smart home simulation system and method

technical field

The invention relates to the technical field of the Internet of Things, in particular to a virtual smart home simulation system and method.

Background technique

With the successive release of various virtual reality devices, 2016 is designated as the first year of virtual reality. Virtual reality technology has also quickly gained the attention of various enterprises and businesses, and has gradually entered the public life. Among the released virtual reality devices, VR glasses are the most common. VR glasses can provide a 360-degree immersive game experience. It has a better 3D IMAX effect and can watch 3D videos and scenes. It is limited to the fields of games and entertainment, and has not been widely used in other fields.

At the same time, as a brand-new life concept, smart home has also developed rapidly in recent years, but in home design, it mainly displays related equipment and information in a static way through two-dimensional plane renderings Smart home products have added visual 3D animation effects, but they do not have the function of scene roaming, and still lack the sense of immersion. Moreover, the traditional operation management platform is only limited to satisfying the functional requirements, and the user interface is too monotonous, lacking in interactivity and realism, and it is difficult to meet the requirements of the user's immersive experience.

Contents of the invention

The technical problem to be solved by the present invention is that the existing smart home lacks interactivity and sense of reality, and it is difficult to meet the requirements of users' immersive experience. A virtual smart home simulation system and method are provided.

In order to solve the above problems, the present invention is achieved through the following technical solutions:

A virtual smart home simulation system includes a home equipment layer, a storage service layer and an application layer. The home device layer consists of device terminals, smart sensor devices and smart home devices; the device terminals are connected to smart sensor devices and smart home devices; smart sensor devices are used to collect home environment data; device terminals complete the transmission of smart sensor devices and smart home devices control; smart home devices execute the user's control commands. The storage service layer includes servers; the servers of the storage service layer are respectively connected to the device terminals of the home device layer and the user terminals of the application layer, which are bridges between the application layer and the home device layer; the status data of smart home devices are directly stored in the server, Models, textures, video and/or sound files are packaged into a format that unity3D can recognize and stored in the server; at the same time, the server also provides a corresponding control interface for the application layer, and processes background data by responding to requests from the application layer. The application layer includes the user terminal; the user terminal provides a real-time immersive three-dimensional control user interface for the user, and directly interacts with the user; the virtual scene of the user interface is synchronized with the real scene of the home, so that the user can be immersed in the virtual reality environment. At the same time complete a variety of remote monitoring and control.

In the above scheme, a device interface is reserved on the device terminal of the home device layer, and the device interface can add or delete smart home devices connected to it according to the actual situation.

A virtual smart home simulation method, comprising the steps of:

Step 1, using a three-dimensional modeling method to establish a three-dimensional model of the scene;

Step 2. Perform lighting rendering and baking for the static models in the scene, and add appropriate colliders according to the external shape of each model, so that the collision detection can better reflect the physical state of the real world, and effectively remove the artifacts generated in the virtual process. unreality;

Step 3: Generate a resource file library and complete the packaging and deployment of resources;

Step 4: Initialize the software, use the resource bundling and packaging method to realize the bundling and packaging and dynamic loading of resources, and complete the layout of the home scene;

Step 5, device initialization, associating the virtual 3D home device in the application layer, that is, the virtual device, with the smart home device in the home device layer, that is, the real device; and adding the real device associated with the virtual device in the client software of the user terminal, Set the parameters of the corresponding equipment and save the settings;

Step 6. Establish the connection between the client software of the user terminal and the distributed database of the server. The data status of the smart home device is stored in the corresponding table of the distributed database. After the client software connects to the distributed database, it reads the data in the corresponding table Data, and real-time update in the user interface of the client software, so that users can view the status information of the device in real time;

Step 7: Run the client software on the user terminal, and the user can obtain a virtual immersive experience through the VR glasses, and realize roaming in the virtual scene and remote device control through the control handle.

Above-mentioned step 1 specifically comprises the following steps:

First, obtain the original CAD data of the home environment, and determine the hierarchical structure of the model;

Then, collect the material of the texture map, and use the image processing software as an auxiliary tool to modify and beautify the texture map;

Finally, use 3D animation rendering and production software as the main modeling tool to model buildings, equipment, vegetation and terrain respectively, and add materials and textures to the models in the scene.

In the above step seven, the user terminal is a smart phone, which is inserted into the head-mounted VR glasses box to complete the immersive virtual experience, and the visual effect is further improved, giving people a more realistic sense of immersion and interaction .

Compared with the prior art, the present invention applies virtual reality technology to the design of smart home, which not only satisfies the immersive experience and interaction of smart home users, but also realizes real-time control of smart devices by virtual reality experiencers. The smart device control interface provided in the virtual scene enables the virtual scene in the 3D model to respond synchronously with the real home scene, meeting the basic functional requirements of traditional smart home control and display. At the same time, users can roam and walk in the 3D scene from multiple perspectives. Such a control method is more realistic and intuitive, and the operation interface is also more interactive. The platform can provide Internet of Things users with a more natural way of obtaining object information and virtual interactive experience, meeting the requirements of Internet of Things related applications for real-time, accuracy and stability.

Description of drawings

FIG. 1 is a schematic diagram of a hardware platform of a virtual smart home simulation system.

Fig. 2 is a flowchart of the construction of the three-dimensional model.

detailed description

A virtual smart home simulation system adopts a three-layer independent layered model, which is respectively a home equipment layer, a storage service layer and an application layer.

The home device layer is mainly composed of various smart sensor devices and WEB-based wired or wireless devices, which are used to collect home environment data, execute user control commands, and complete the control and transmission of various home devices. Household equipment adopts TCP/UDP protocol, and uses socket programming to access the virtual smart home simulation system.

The storage service layer is mainly responsible for storing various data and providing services for the application layer. The service layer is the bridge between the application layer and the home device layer. The status data of household equipment is stored in the database, and the models, textures, videos, and sound files are packaged into a format that unity3D can recognize and stored in the server of the storage service layer. The server provides a corresponding control interface for the application layer, and processes background data by responding to requests from the application layer.

The application layer provides users with a real-time immersive three-dimensional control interface to directly interact with users. The virtual scene of the user interface is synchronized with the real home scene, enabling users to complete various remote monitoring and control while being immersed in the virtual reality environment.

The working process of this system is as follows: the user's remote operation is transmitted to the server through the HTTP/TCP protocol of the Web, and the server responds to the request and sends a command to the home device. The smart home devices installed at home receive and analyze the command, and directly execute various instructions from the user. After the home device executes the command, it feeds the data back to the server, and the server then transmits the data to the client at the application layer, thereby realizing remote operation.

In the preferred embodiment of the present invention, the virtual smart home simulation system uses a hardware platform composed of several typical network devices and sensor acquisition terminals to complete the demonstration of remote real-time monitoring and control functions of various home devices in the virtual reality model. See Figure 1 for a schematic diagram of the hardware platform. The sensor acquisition terminal transmits data through the wireless radio frequency module and connects to the system using the TCP protocol. The temperature and humidity sensor can collect the temperature, humidity, and light data captured by the photosensitive sensor in the home environment. The human body infrared sensor detects whether someone enters, smoke sensor, gas leakage The sensor uploads alarm information when a fire or natural gas leakage is detected. The access controller uses the UDP protocol to establish a connection with the system, and executes the user's control command by receiving a specific command frame. The network camera is connected to the system through the local area network, and the video screen is displayed on the client by calling the interface function on the client, which can realize video recording and screenshot of video surveillance. Interfaces for other devices are also reserved in the platform, and household devices can be added or deleted according to the actual situation.

The virtual smart home simulation method based on the above system comprises the following steps:

Step 1. Using 3D modeling technology to build a 3D model of the scene.

First of all, it is necessary to obtain the original CAD data of the home environment and determine the hierarchical structure of the model. Then collect the material of the texture map, and use Photoshop as an auxiliary tool to modify and beautify the texture map. Finally, using 3dsMax as the main modeling tool, the buildings, equipment, vegetation and terrain were modeled separately. Add materials and textures to the models in the scene.

Step 2. In order to draw a more realistic scene model and bring more realistic visual effects to users, perform lighting rendering and baking for the static model in the scene. Add appropriate colliders according to the external shape of each model, so that the collision detection can better reflect the physical state of the real world, and effectively remove the unreality generated in the virtual process.

Step 3: Generate a resource file library and complete the packaging and deployment of resources.

Add animation to the dynamic home equipment model in the scene, use AssetBundle technology to bundle the interface function (or script) that controls the smart device and the virtual model of the smart device, generate a resource file in the format of .assetbundle, and save it in the root directory of the Apache server Resource files for users to load and use.

Step 4: Initialize the software, use the AssetBundle technology to realize the bundling and dynamic loading of resources, and complete the layout of the home scene. The bundling and packaging of 3D models and interface functions simplifies the operation process and makes the operation faster and more convenient. And dynamic loading can enable users to load some resources first when entering a scene, and then load other resources when entering other scenes, which can not only improve the speed, but also reduce the consumption of memory resources.

Initialize the client software, load/update/destroy the indoor scenes and furniture models according to the user's own needs, and place the virtual furniture models in the appropriate positions in the virtual scene according to the real indoor layout, and then load the virtual furniture models with appropriate textures and materials to determine the overall style of the virtual scene.

Step five, device initialization, associating the virtual 3D device with the real controllable device. Add the real smart home device associated with the virtual home device in the client software, set the parameters (IP address, port number) of the corresponding device, and save the settings.

Step 6. Establish the connection between the client and the distributed database. The data status of the household equipment is stored in the corresponding table of the database. After the client connects to the database, it reads the data in the corresponding table and updates it in the client user interface in real time. Enable users to view device status information in real time.

Step 7. Run the client software on the smartphone, insert the smartphone into the VR glasses case, and start the virtual immersive experience and remote control of smart home devices.

Step 8: Use the Bluetooth handle to realize roaming and remote device control in the virtual scene.

Users can realize 360-degree omnidirectional roaming inspection in the virtual home scene without dead ends, and can observe the status and data of smart home devices in real time, and complete remote control, and keep the 3D model and the status of real controllable devices in sync.

Key technology of the present invention is further described below:

1. 3D modeling technology

The construction of the 3D model is an important part of the virtual smart home simulation system. A good simulation effect is the key to making users feel immersive. This system uses 3dsMax as the main modeling tool to model buildings, equipment, vegetation and terrain respectively. . In order to draw a more realistic scene model and bring more realistic visual effects to users, key technologies such as lighting rendering, baking, and animation production are used in the established scene model. Photoshop is used as an auxiliary tool to retouch and beautify the texture map. After the model is completed, export the model into FBX file format to unity3D. See Figure 2 for the flow chart of the construction of the 3D model.

2. Visual simulation technology

2.1. Scene interaction

If a realistic 3D model is the key to creating a sense of immersion, then the visual simulation of actions that simulate the physical world and natural behavior is the key to creating a sense of interaction. The virtual simulation system developed by the Unity3D 3D engine has good dynamic interaction performance. It can realize the dynamic interaction of the 3D scene through the script attached to the GameObject. The script used in this system is implemented by C#, and the original static scene Added dynamic real-time interactive features.

Table 1 User 3D interactive function table

2.2. Device status synchronization

The system uses the Sql Server distributed database as a bridge to keep the state of virtual equipment and real equipment synchronized, so that the home model changes according to the state of the actual equipment in the virtual environment. In the virtual home environment, a fixed ID is assigned to each controllable home device, which corresponds to the device address of the real device. And add the trigger control of various real devices in the most intuitive and fast form, send control commands to home devices, and the home devices will make corresponding responses after receiving the control commands, and send the status data frame containing the device to the server, and the server will analyze it The data frame stores the status information of the device in the database. The home model in the virtual scene detects the change of the device status in the database, plays a synchronous animation, and keeps in sync with the real device status.

3. Three-dimensional roaming technology

In order to enhance the user experience, a first-person roaming method is adopted. This system provides a total of two roaming modes: first-person free-view roaming and first-person fixed-view switching. The first-person roaming uses the CharacterController component as the character controller, accepts input operations from the Bluetooth control handle, and controls the actions of the character.

4. Active database technology

Active Database technology is a new development branch of modern database technology, and it is the product of the combination of database technology and artificial intelligence technology. Its core idea is to make the database system have various active service functions, and to realize various active requirements with a unified and convenient mechanism, that is, to use these active functions with a unified method and principle Some database functions are integrated in a database system. In this system, the user controls and operates the terminal equipment, and the cascading between the control commands sent to the equipment and the SQL Server 2008 data tables is mainly realized through the active database technology. The approach to proactiveness is primarily to create event monitors and triggers. This mechanism is mainly realized by embedding some rules into the database system, and the active database can actively respond to events.

In this system, when the user controls the device, by clicking the model in the virtual scene, the model sends the device code to the data table specified by the database by binding the click event, and the trigger mechanism is established in the database to monitor the updata of the data table in real time , insert, delete, when the above changes occur in the data table, use the CLR function of the SQL Server 2008 trigger to call the method in the external function definition class, and send the corresponding control protocol to the device to control the terminal device. Compared with the method of continuously querying the state of the database with an external program, this method of sending control commands in an interrupt mode has better stability, reduces the coupling between the server side and the hardware protocol, and can occupy less resources to a greater extent .

Cascading operations on data tables. When there are many devices connected to the platform, the device information and logical relationship stored in the database will be quite complicated. Modifying the status of a data table may require modifying the information of multiple related data tables at the same time, relying only on manual management Or the traditional database technology is very inefficient, and the logic is complex, so it is easy to miss and report errors. This paper studies the application of active database technology in the augmented reality virtual monitoring platform, and establishes an efficient, safe and dynamic data table information management mechanism. Improve information management efficiency and system stability.

The present invention simulates the real home scene, uses the three-dimensional engine Unity3D in combination with the real-time data acquisition terminal, the SQLSERVER database and the Apache server, and uses multiple key technologies such as three-dimensional modeling technology, visual simulation technology, active database technology, and node control methods to establish a real-time The virtual smart home simulation system completed the interaction between the virtual world and real devices, and designed the system architecture and functional modules of the smart home virtual simulation system. In the virtual smart home simulation system, the virtual scene is synchronized with the real home scene. The control interface of the smart device in the real home scene is provided in the virtual scene, so that users can complete various remote monitoring and control while being immersed in the virtual reality environment. Humanized The three-dimensional human-computer interaction interface allows users to obtain better roaming immersion and interaction, and provides a more natural way to obtain object information and virtual interaction experience. The virtual smart home simulation system based on Unity3D supports multiple platforms and can run on multiple platforms such as Windows, Android, and IOS, and has wider practical applications under the trend of platform diversification.

Claims (5)

1. A virtual intelligent home simulation system, characterized in that: comprising a household equipment layer, a storage service layer and an application layer; The home device layer consists of device terminals, smart sensor devices and smart home devices; the device terminals are connected to smart sensor devices and smart home devices; smart sensor devices are used to collect home environment data; device terminals complete the transmission of smart sensor devices and smart home devices control; smart home devices execute the user's control commands; The storage service layer includes servers; the servers of the storage service layer are respectively connected to the device terminals of the home device layer and the user terminals of the application layer, which are bridges between the application layer and the home device layer; the status data of smart home devices are directly stored in the server, Models, textures, video and/or sound files are packaged into a format that unity3D can recognize and stored in the server; at the same time, the server also provides a corresponding control interface for the application layer, and processes background data by responding to requests from the application layer; The application layer includes the user terminal; the user terminal provides a real-time immersive three-dimensional control user interface for the user, and directly interacts with the user; the virtual scene of the user interface is synchronized with the real scene of the home, so that the user can be immersed in the virtual reality environment. At the same time complete a variety of remote monitoring and control. 2. A virtual smart home simulation system according to claim 1, characterized in that: a device interface is reserved on the device terminal of the home device layer, and the device interface can add or delete smart home devices connected to it according to the actual situation . 3. A virtual intelligent home simulation method is characterized in that, comprising the steps: Step 1, using a three-dimensional modeling method to establish a three-dimensional model of the scene; Step 2. Perform lighting rendering and baking for the static models in the scene, and add appropriate colliders according to the external shape of each model, so that the collision detection can reflect the physical state of the real world, and effectively remove the unreality generated in the virtual process ; Step 3: Generate a resource file library and complete the packaging and deployment of resources; Step 4: Initialize the software, use the resource bundling and packaging method to realize the bundling and packaging and dynamic loading of resources, and complete the layout of the home scene; Step 5, device initialization, associating the virtual 3D home device in the application layer, that is, the virtual device, with the smart home device in the home device layer, that is, the real device; and adding the real device associated with the virtual device in the client software of the user terminal, Set the parameters of the corresponding equipment and save the settings; Step 6. Establish the connection between the client software of the user terminal and the distributed database of the server. The data status of the smart home device is stored in the corresponding table of the distributed database. After the client software connects to the distributed database, it reads the data in the corresponding table Data, and real-time update in the user interface of the client software, so that users can view the status information of the device in real time; Step 7: Run the client software on the user terminal, and the user can obtain a virtual immersive experience through the VR glasses, and realize roaming in the virtual scene and remote device control through the control handle. 4. A virtual smart home simulation method according to claim 3, wherein step 1 specifically comprises the following steps: First, obtain the original CAD data of the home environment, and determine the hierarchical structure of the model; Then, collect the material of the texture map, and use the image processing software as an auxiliary tool to modify and beautify the texture map; Finally, use 3D animation rendering and production software as the main modeling tool to model buildings, equipment, vegetation and terrain respectively, and add materials and textures to the models in the scene. 5. A virtual smart home simulation method according to claim 3, wherein in step 7, the user terminal is a smart phone, and the smart phone is inserted into a head-mounted VR glasses box to complete the immersive virtual Experience and visual effects have been further improved, giving people a more realistic sense of immersion and interaction.