KR20220004156A - Car cabin interaction method, device and vehicle based on digital human - Google Patents

Abstract

An embodiment of the present application provides a method, apparatus and vehicle for car cabin interaction based on a digital human, obtains state information of a living body in a car cabin, determines operation information matching the state information, and based on the operation information to create an animation in which the digital human performs the corresponding action and also displays it on a display device in the car cabin.

Description

Car cabin interaction method, device and vehicle based on digital human

[Cross-Reference to Related Applications]

This application is filed with the Intellectual Property Office of China on March 30, 2020, the application number is CN202010239259.7, the title of the invention is "Child state detection method and device, electronic device, storage medium" and June 23, 2020 Claims the priority of the Chinese patent application filed with the Chinese Patent Office at CN202010583637.3, titled "Digital Human-Based Car Cabin Interaction Method, Device and Vehicle" Included in this application.

[Technical field]

The present application relates to the field of computer vision technology, and more particularly to a method, apparatus and vehicle for car cabin interaction based on a digital human.

Currently, many vehicles are equipped with monitoring and control products for performing interactions with living organisms in the vehicle. However, the existing monitoring and control products have a hard interaction method and lack toughness.

The present application provides an automobile cabin interaction method, apparatus, and vehicle based on a digital human.

According to a first aspect of an embodiment of the present application, there is provided a method for interacting with a car cabin based on a digital human, the method comprising: obtaining status information of a living body riding in a car cabin; determining operation information matching the state information; and generating an animation in which the digital human performs a corresponding motion based on the motion information and displaying the animation on a display device in the automobile cabin.

In some embodiments, the step of generating an animation in which a digital human performs a corresponding action based on the motion information and displaying the animation on a display device in the car cabin includes: determining voice information matching the status information ; obtaining a corresponding voice including a timestamp based on the voice information; and generating and displaying an animation in which the digital human executes the motion at a time corresponding to the timestamp based on the motion information while reproducing the voice.

In some embodiments, the operation includes a plurality of sub-operations, each sub-operation matches one phoneme in the voice, and the timestamp includes a timestamp of each phoneme; The step of generating and displaying an animation in which the digital human executes the motion at a time corresponding to the timestamp based on the motion information includes: based on the timestamp of each phoneme, a sub-action matching each phoneme. determining an execution time; and generating and displaying, based on the motion information, an animation in which the digital human executes a sub-action matching the corresponding phoneme at a timestamp of each phoneme.

In some embodiments, the step of generating an animation in which the digital human executes a corresponding motion based on the motion information and displaying the animation on a display device in the automobile cabin includes at least one corresponding to the motion information in a motion model base calling a segment of the digital human of; and sequentially displaying each frame of the frame of the motion image of the at least one digital human on the display device.

In some embodiments, the state information of the living body includes the first state information of the living body, and the step of obtaining the state information of the living body riding in the car cabin includes collecting a surveillance control video of a rear row in the car cabin. step; and performing biometric detection on the monitoring control video and performing state analysis on the detected living body to obtain first state information of the living body.

In some embodiments, the surveillance control video is captured via a video acquisition device mounted on a rearview mirror in the car cabin and with a lens facing the back row of the car cabin.

In some embodiments, the first state information includes at least one of classification information of the living body, identification information, attribute information, emotion information, facial expression information, gesture information, seating information, and seat belt wearing information; and/or the living body includes at least one of a driver, a co-pilot, a child, an elderly person, a pet, and a back-row passenger.

In some embodiments, the state information of the living body includes the first state information and second state information of the living body, wherein the first state information is obtained based on a surveillance control video in a car cabin; The acquiring of the status information of the living body in the car cabin further includes acquiring second status information sent from a smart device carried by the living body; The determining of the operation information matching the state information includes determining operation information matching both the first state information and the second state information.

In some embodiments, the second state information includes at least one of health state information and nervous system state information.

In some embodiments, the obtaining of the state information of the living body riding in the car cabin includes: inputting a surveillance control video in the car cabin into a pre-trained neural network; and determining the state information of the living body based on the output result of the neural network.

In some embodiments, the method includes generating an animation of a digital human performing a corresponding action based on the motion information and generating the shape of the digital human before displaying it on a display device in the car cabin include more

In some embodiments, generating the shape of the digital human comprises: generating the shape of the digital human based on the state information of the living body; or generating a shape of the digital human based on a preset shape template of the digital human.

In some embodiments, the attribute information of the living body includes at least one of age, gender, appearance, body type, clothes and accessories, hair style, and skin color.

In some embodiments, the method further includes controlling an execution state of the on-vehicle device based on the state information.

In some embodiments, the determining of the operation information matching the state information includes: obtaining a driving state of the vehicle; and determining the driving state of the vehicle and operation information respectively matched with the state information.

In some embodiments, the obtaining of the state information of the living body riding in the automobile cabin includes: recognizing the living body riding in the automobile cabin based on a target image in the automobile cabin; and determining whether the living body is located in a seat in the vehicle cabin based on the location information of the living body.

In some embodiments, the obtaining of the state information of the living body riding in the car cabin comprises: determining object information of each object from among the target images in the car cabin, wherein, with respect to each of the objects, the object the object information includes location information of the center point of the object and object type information corresponding to the center point of the object; selecting the living body and the seat in the car cabin from the object based on object type information of each of the objects; and determining whether the living body is located in the seat based on the center point position of the living body and the center point position of the seat.

In some embodiments, the step of generating an animation in which the digital human executes the corresponding motion based on the motion information and displaying it on a display device in the car cabin comprises: a digital human corresponding to a living body not located in the seat generating an animation of performing a demonstration motion of sitting in the seat and wearing a seat belt and displaying the animation on a display device in the automobile cabin.

In some embodiments, the method further comprises, in response to determining that the living body is not located in the seat, sending a prompt message.

In some embodiments, determining the location information of the center point of the object in the target image includes: performing feature extraction on the target image to obtain a first feature map corresponding to the target image; acquiring each feature point in the first feature map from among first preset channels of the first feature map as a response value of an object center point; dividing the first feature map into a plurality of sub-regions and determining a maximum response value and a feature point corresponding to the maximum response value in each sub-region; and using a feature point with a maximum response value greater than a preset threshold as the center point of the object, and determining the location information of the center point of the object based on the location index of the center point of the object in the first feature map. do.

In some embodiments, determining the object type information corresponding to the center point of the object in the target image includes: performing feature extraction on the target image to obtain a second feature map corresponding to the target image; determining a location index of the center point of the object in the second feature map based on the location index of the center point of the object in the first feature map; and acquiring object type information corresponding to the center point of the object at a location corresponding to the location index of the center point of the object in the second feature map.

According to a second aspect of the embodiment of the present application, there is provided an automobile cabin interaction device based on a digital human, the device comprising: an acquisition module configured to acquire status information of a living body riding in an automobile cabin; a determining module, configured to determine operation information matching the state information; and a display module configured to generate an animation in which a digital human performs a corresponding operation based on the operation information and display the animation on a display device in the automobile cabin.

In some embodiments, the display module includes: a first determining unit, configured to determine voice information matching the state information; a first acquiring unit, configured to acquire a corresponding voice including a timestamp based on the voice information; and a first display unit configured to reproduce the voice and simultaneously generate and display an animation in which the digital human executes the operation at a time corresponding to the timestamp based on the operation information.

In some embodiments, the operation includes a plurality of sub-operations, each sub-operation matches one phoneme in the voice, and the timestamp includes a timestamp of each phoneme; The first display unit may include: a determining subunit configured to determine an execution time of a sub-operation matching each phoneme based on a timestamp of each phoneme; and a display sub-unit configured to generate and display an animation of a sub-action matching each phoneme that the digital human executes at a timestamp of each phoneme, based on the motion information.

In some embodiments, the display module comprises: a calling unit, configured to call a motion segment of at least one digital human corresponding to the motion information in a motion model base; and a second display unit configured to display each frame of the frame of the motion image of the at least one digital human on the display device in order.

In some embodiments, the state information of the living body includes the first state information of the living body, and the acquiring module includes: a collecting unit configured to collect a surveillance control video of a rear row in a car cabin; and a detection and analysis unit, configured to execute biometric detection on the monitoring control video and perform state analysis on the detected living body to obtain first state information of the living body.

In some embodiments, the surveillance control video is captured via a video acquisition device mounted on a rearview mirror in the car cabin and with a lens facing the back row of the car cabin.

In some embodiments, the first state information includes at least one of classification information of the living body, identification information, attribute information, emotion information, facial expression information, gesture information, seating information, and seat belt wearing information; and/or the living body includes at least one of a driver, a co-pilot, a child, an elderly person, a pet, and a back-row passenger.

In some embodiments, the state information of the living body includes the first state information and second state information of the living body, wherein the first state information is obtained based on a surveillance control video in a car cabin; the acquiring module is further configured to acquire second status information sent from the smart device carried by the living body; The determining module is configured to determine operation information matching both the first state information and the second state information.

In some embodiments, the second state information includes at least one of health state information and nervous system state information.

In some embodiments, the acquiring module comprises: an input unit, configured to input the surveillance control video in the car cabin into a pre-trained neural network; and a second determining unit configured to determine the state information of the living body based on an output result of the neural network.

In some embodiments, the apparatus is configured to generate an animation in which a digital human performs a corresponding operation based on the operation information and to generate the shape of the digital human before displaying it on a display device in the car cabin It further includes a creation module.

In some embodiments, the generating module generates the shape of the digital human based on the state information of the living body; Alternatively, it is configured to generate the shape of the digital human based on a preset shape template of the digital human.

In some embodiments, the attribute information of the living body includes at least one of age, gender, appearance, body type, clothes and accessories, hair style, and skin color.

In some embodiments, the apparatus further includes a control module configured to control an execution state of the on-vehicle device based on the state information.

In some embodiments, the determining module is configured to obtain a driving state of the vehicle; It is set to determine the driving state of the vehicle and operation information respectively matched with the state information.

In some embodiments, the acquisition module is configured to recognize a living body in the car cabin based on the target image in the car cabin; It is further configured to determine whether the living body is located in a seat in the automobile cabin based on the location information of the living body.

In an embodiment, the acquiring module is configured to determine object information of each object in the target image in the vehicle cabin; selecting the living body and the seat in the car cabin in the object based on object type information of each of the objects; further configured to determine whether the living body is located in the seat based on the center point position of the living body and the center point position of the seat, wherein, with respect to each of the objects, the object information of the object is the center point of the object location information and object type information corresponding to the center point of the object.

In some embodiments, the display module generates an animation in which a digital human corresponding to a living body not located in the seat sits on a seat and performs a demonstration operation to wear a seat belt, and is displayed on a display device in the car cabin. additionally set to display.

In some embodiments, the device further comprises a notification module configured to send a prompt message in response to determining that the living body is not located in the seat.

In some embodiments, the acquiring module includes: performing feature extraction on the target image to obtain a first feature map corresponding to the target image; acquiring each feature point in the first feature map from among first preset channels of the first feature map as a response value of an object center point; dividing the first feature map into a plurality of sub-regions and determining a maximum response value and a feature point corresponding to the maximum response value in each sub-region; and using a feature point with a maximum response value greater than a preset threshold as the center point of the object, and determining the location information of the center point of the object based on the location index of the center point of the object in the first feature map. It is set to determine position information of the center point of the corresponding object in the target image.

In an embodiment, the acquiring module performs feature extraction on the target image to obtain a second feature map corresponding to the target image; determining a location index of the center point of the object in the second feature map based on the location index of the center point of the object in the first feature map; and acquiring object type information corresponding to the center point of the object at a location corresponding to the location index of the center point of the object in the second feature map, and the object type corresponding to the center point of the object in the target image set to determine information.

According to a third aspect of the embodiments of the present application, there is provided a computer-readable storage medium, wherein the computer program is stored in the computer-readable storage medium, and when the program is executed by a processor, the method described in any of the embodiments is provided. come true

According to a fourth aspect of an embodiment of the present application, there is provided a computer device, wherein the computer device includes a memory, a processor, and a computer program stored in the memory, executable in the processor, and when the program is executed by the processor, any The method described in the examples is realized.

According to a fifth aspect of the embodiment of the present application, there is provided a vehicle, and a display device in an automobile cabin of the vehicle, a monitoring control system and a digital human-based automobile cabin interaction apparatus described in any embodiment of the present application or the present application A computer device as described in any of the embodiments is installed.

According to a sixth aspect of an embodiment of the present application, there is provided a computer program product, the computer program product comprising computer instructions, and when the computer instructions are executed by a processor, the method described in any embodiment of the present application is provided. come true

An embodiment of the present application obtains status information of a living body in a car cabin, determines motion information matching the status information, and generates an animation in which a digital human executes a corresponding motion based on the motion information, and By displaying on the display device in the car cabin, based on the state information of the living body in the car cabin, it is possible to display the digital human of different actions, realizing a personified interaction, the interaction process is more natural, and feedback information during the interaction process By improving the degree of bioacceptance to the biomaterial, biosafety in the process of driving a vehicle is improved.

It should be understood that the above general description and the detailed description given below are merely exemplary and interpretative and are not intended to limit the present application.

The accompanying drawings form a part of the specification in the specification, and the accompanying drawings correspond to embodiments of the present application, and together with the specification, describe the technical solution of the present application.
1 is a flowchart of a vehicle cabin interaction method based on a digital human according to an embodiment of the present application.
2A is a schematic diagram of a digital human according to an embodiment of the present application.
2B is a schematic diagram of a digital human according to another embodiment of the present application.
3 is a schematic diagram of a vehicle cabin interaction method based on a digital human according to an embodiment of the present application.
4 is a schematic diagram of a vehicle cabin interaction device based on a digital human according to an embodiment of the present application.
5 is a structural schematic diagram of a computer device according to an embodiment of the present application.
6A and 6B are schematic views of a vehicle according to an embodiment of the present application, respectively.

Embodiments will be described in detail herein, examples of which are shown in the drawings. When the following description includes drawings, like numbers in different drawings refer to similar or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

The terms used in the present disclosure are for the purpose of describing specific embodiments only, and are not intended to limit the present disclosure. As used in this disclosure and the appended claims, the singular forms "a", "said" and "the" also refer to the plural forms, unless the context clearly indicates otherwise. It is intended to include It should further be understood that the term “and/or” as used herein refers to and includes any or all possible combinations of one or more associated listed items. Also, as used herein, the term “at least one” means any one of a plurality of types or any combination of at least two of a plurality of types.

Although the terms first, second, third, etc. may be used in this disclosure to describe various pieces of information, it should be understood that the information should not be limited to these terms. These terms are only used to distinguish between the same types of information. For example, without departing from the scope of the present disclosure, first information may be referred to as second information; Similarly, the second information may be referred to as first information. Depending on the context, the word "if" as used herein means "upon" or "when" or "in response to determination." " can be interpreted as

To enable a person skilled in the art to better understand the technical solutions in the embodiments of the present disclosure, and to make the described objects, features and advantages of the embodiments of the present disclosure more clear, Technical solutions in embodiments of the disclosure will be further described in detail below with reference to the accompanying drawings.

With the spread of private cars and the rapid development of smart car cabins, a wide range of interest is being raised in safety during the driving process. One way to improve safety is to install a monitoring and control system in a car cabin to execute monitoring and control on the living body in the car cabin, and output interaction information based on the monitoring control result to quickly notify the living body in the car cabin, if necessary. method. The conventional interaction method generally outputs a voice prompt message through an audio reproducing device in a car cabin, or outputs a voice prompt message and a text prompt message to a display device of the vehicle at the same time. However, this interaction method gives people a hard feeling of interacting with the machine, and sometimes the degree of acceptance of the living body for the interaction information is relatively low, so the biosafety in the vehicle driving process is deteriorated.

In view of this, the embodiment of the present application provides a method for car cabin interaction based on a digital human, and as shown in FIG. 1, the method is

Step 101 of obtaining status information of the living body riding in the car cabin;

determining operation information matching the state information 102; and

and generating an animation in which the digital human performs a corresponding motion based on the motion information and displaying the animation on a display device in the car cabin.

In step 101, the vehicle may be various types of vehicles, such as a private car, a school bus, and a large bus. For convenience of description, a method according to an embodiment of the present application will be described below with an example of a private vehicle. The living body may include at least one of a driver, a co-pilot, a child, an elderly person, a pet, and a passenger in the back row, but is not limited thereto. According to actual needs, interaction with other living bodies can be performed, satisfying the interaction demands of various scenes.

For example, during the boarding process, certain safety issues such as a child unfastening a seat belt or a pet leaving the seat are given to a living body (for example, a back-row passenger such as a child or a pet) seated in the back row. there is a risk of However, since the driver has to concentrate on driving during the driving process, it is not possible to combine the situation in the back row in the car cabin. In order to improve the safety of the rear row living body during the riding process, it is possible to interact with the living body riding in the back row of the car cabin through a digital human. Accordingly, in some embodiments, the living body is a living body seated in a back row of a car cabin. By performing an interaction with the living body seated in the back row through the digital human, the degree of acceptance of the riding body to the feedback information during the interaction process can be improved, improving the safety of the riding body, and the driver applying excessive energy to the situation in the back row It allows you to focus on driving in the course of driving without putting yourself on both sides of the road.

In some embodiments, the state information may include first state information of the living body, for example, the first state information of the living body includes classification information of the living body, identity information, attribute information, emotion information, It includes at least one of facial expression information, gesture information, seat seating information, and seat belt wearing information.

Among them, the classification information is used to indicate the classification of the living body, and the classification may include humans and/or animals. The identification information may include biometric recognition information for uniquely recognizing each living body. The attribute information is used to indicate the characteristic attribute of the living body, and may include, but is not limited to, at least one of age, gender, appearance, body type, clothes and accessories, hair style, and skin color. The emotion information is used to indicate the emotion classification of the living body, and the emotion classification may include, but is not limited to, at least one classification such as joy, sadness, anger, shame, surprise, excitement, fear, anger, and peace. The expression information is used to represent the facial expression of the living body, and the facial expression may include at least one of a smile, a pouting mouth, crying, squinting eyes, a humorous expression, and the like, but is not limited thereto. The gesture information is used to indicate the motion of the living body, and may include, for example, at least one of clapping, rolling a foot, opening a car door, or putting a head or hand out a window. The seat seating information is used to indicate whether the living body leaves a seat (eg, a child seat). The seat belt wearing information is used to indicate whether the living body wears the seat belt.

Further, the state information of the living body may also include the second state information of the living body, for example, the second state information of the living body includes at least one of health state information and nervous system state information. Among them, the health state information is used to indicate the health state of the living body, and may include, for example, at least one of heart rate, blood pressure, blood fat, blood sugar, and the like, but is not limited thereto. The nervous system state information is used to indicate the degree of excitement of the nervous system of the living body, and includes, for example, whether the living body is tired or asleep. When the first state information and the second state information are simultaneously acquired, the operation information is operation information that matches both the first state information and the second state information.

By acquiring other status information, it is possible to determine the status of various aspects of the living body, and the more the obtained status information is classified, the more comprehensive and accurate the determined status of the living body is, so that the operation performed by the digital human can be performed by the living body. It is more suitable to the state of the human being, and furthermore, the degree of personification of digital human can be improved.

The state information of the living body can be acquired through various methods, for example, it can be acquired based on a video surveillance control in the car cabin, or it can be acquired based on the results of audio surveillance control in the car cabin, Alternatively, the living body can obtain it from a smart device carried. In addition, the state information of the living body may be acquired through a method in which several means are combined with each other. By acquiring the status information in various ways, the overallness and flexibility of the acquired status information can be improved. Different state information acquisition methods can be selected according to different application scenes, so that various types of state information are acquired according to different application scenes. For example, in a dimly lit scene, it is possible to jointly acquire the status information of the living body based on the video and audio monitoring control results of the monitoring control in the car cabin, thereby improving the accuracy of acquiring status information.

In some embodiments, a supervisory control system may be mounted within the vehicle cabin. According to the needs of the actual application scene, the monitoring and control system may execute monitoring control for a living body in an arbitrary area in the automobile cabin, for example, the monitoring and control system may execute monitoring control for the driver of the driver's seat; ; Also for example, the supervisory control system may execute supervisory control for a driver's assistant in the passenger seat; Also, for example, the supervisory control system may execute supervisory control for living bodies in the rear row in the automobile cabin. Alternatively, the monitoring and control system may monitor and control the living body in a plurality of areas (eg, a passenger seat and a rear row) in an automobile cabin.

The monitoring control system may include at least one of a visual monitoring control system and an audio monitoring control system, wherein the visual monitoring control system displays a visual monitoring control result (eg, monitoring control video or monitoring control image) of the living body. used to obtain In some embodiments, surveillance control video may be acquired via a video acquisition device mounted on a rearview mirror in the car cabin and with a lens facing the back row of the car cabin. By mounting the video collection device on the rearview mirror in the car cabin, the field of view of the video collection device is broadened, so that it is not easily obscured by objects in the car cabin, so that a relatively complete control video of the back row of the car cabin can be collected. The audio monitoring control system is used to obtain the audio monitoring control result of the living body, and the audio monitoring control result is the voice of the living body and/or the living body operation (eg, opening a car door or wearing a seat belt). or a sound generated when at least one action such as tapping on a car window) is executed. The first state information of the living body may be determined based on the monitoring control result.

When the state information of the living body is obtained based on the surveillance control video in the car cabin, the state information of the living body may be determined based on an output result of the neural network by inputting a neural network trained in advance for the surveillance control video. In some embodiments, the state information of the living body may be determined by combining a neural network and other algorithms (eg, a safety recognition algorithm and/or a skeletal key point detection algorithm, etc.). In some embodiments, the state information of the living body may be determined by other methods, and will not be described further herein. By acquiring the state information of the living body through the neural network, it is possible to improve the accuracy of obtaining the state information.

The neural network may include an input layer, at least one intermediate layer, and an output layer, wherein the input layer, the at least one intermediate layer and the output layer all include one or a plurality of neurons. Here, the intermediate layer usually refers to a layer located between the input layer and the output layer, for example a hiding layer. In some examples, the intermediate layer of the neural network may include, but is not limited to, at least one of a convolutional layer, a Rectified Linear Units (ReLU) layer, and the like, and the more the number of layers of the intermediate layer included in the neural network, the more the network is deeper The deep neural network may specifically be a deep neural network or a convolutional neural network.

or, collect surveillance control video of the back row in the car cabin; The first state information of the living body may be obtained by performing biometric detection on the monitoring control video and performing state analysis on the detected living body. Based on the surveillance control video, on the one hand, it is possible to obtain relatively comprehensive status information, and on the other hand, since there is a temporal correlation in the target image of a plurality of frames included in the surveillance control video, the temporal correlation is used to obtain the status information Acquisition accuracy can be improved. For example, by acquiring the monitoring control video of the living body, performing safety recognition on the monitoring control video, and determining the identification information of the living body based on the safety recognition result.

Also, for example, the emotional information of the living body may be recognized based on the surveillance control video. Specifically, acquiring a target image of at least one frame including the living body in the surveillance control video; The facial sub-image of the living body may be cut out from the target image. An operation of each of the at least two organs of the face represented by the facial sub-image, for example, a frown, a frown, or a raised corner of the mouth is recognized. Based on the recognized motion of each organ, the facial emotion information represented by the facial sub-image is determined. Before recognizing the motion of each organ, image preprocessing may be performed on the facial sub-image. Specifically, determining location information of key points (eg, around the eyes, around the mouth, between the eyebrows, at the end of the eyebrows, on the nose, etc.) among the facial sub-images; based on the location information of the key point, performing affine transformation on the facial sub-image to convert the facial sub-image in another direction among the facial sub-images into a front-facing facial sub-image; Normalization processing is performed on the face sub-image facing the front, so that the facial sub-image after the processing can be obtained.

Also, for example, a facial sub-image of a living body may be cut out in the target image. Based on the facial sub-image, state information on whether the left eye of the living body is closed or opened and information about whether the right eye is closed or opened is determined. Specifically, in implementation, the facial sub-image is input to a neural network that has undergone one training, and based on the output result of the neural network, information on whether the left eye of the living body is closed or not and information about whether the right eye is closed or not is determined. Thereafter, based on the state information on whether the left eye of the living body is closed and open and the state information on whether the right eye is closed or opened, it is determined whether the living body is tired or asleep. Specifically, determining the cumulative time for closing the eyes of the living body based on the left eye closing/opening status information and the right eye closing/opening status information corresponding to the target image of the living body in a plurality of consecutive frames; determining that the living body has already fallen asleep when the cumulative time for closing the eyes is greater than a preset threshold; When the accumulated time for closing the eyes is equal to or less than or equal to a preset threshold, it may be determined that the living body has not yet fallen asleep.

When the state information of the living body is acquired based on the audio monitoring control result in the car cabin, the voice made by the living body is acquired, the voice is recognized for voice recognition, and the identification information of the living body is based on the voice recognition result. and/or determine sentiment information.

When the state information of the living body is obtained from the smart device carried by the living body, the second state information collected in advance by the related smart device may be received. The smart device may be a wearable device such as a smart bracelet or smart glasses, or a portable terminal such as a mobile phone or a tablet PC.

In step 102, operation information matching the state information may be determined. In some embodiments, a degree of matching between each type of motion information and the status information may be determined, and motion information having the highest degree of matching may be determined as motion information matching the status information. When multiple pieces of information are included in the status information, the degree of matching between the same motion information and multiple pieces of the status information is determined, and then the degree of matching between the motion information and the status information based on the matching degree corresponding to the multiple pieces of information can be decided For example, a weighted average is performed on the degree of matching corresponding to various pieces of information. In some other embodiments, a mapping relationship between other state information and matching action information may be established in advance, and the matching action information may be determined based on the mapping relationship.

In step 103, after determining the matching motion information, an animation is generated in which the digital human performs the corresponding motion based on the matching motion information, and a display device in the car cabin (eg, a control panel or a display device). The digital human may be a virtual shape that is generated through software and can be displayed on a display device in an automobile cabin. The software-based method of creating and displaying a digital human has a relatively low cost, a relatively fast response speed of the digital human, a relatively low post-maintenance cost, and is convenient for updating and upgrading.

2A and 2B , the shape of the digital human may be a cartoon shape, a 3D shape generated based on the actual shape of the living body, or other types of shapes. In some embodiments, the shape of the digital human may be a fixed shape, that is, the shape of the digital human displayed on the device every time is the same, for example, the shape of a boy, or a cartoon character (eg, , the shape of Doraemon), and the like. In some other embodiments, the shape of the digital human may be dynamically generated according to an actual situation, that is, the shape of the digital human displayed on the device may be different.

Another digital human shape may be displayed based on the state information of the living body (eg, at least one of identity information, emotion information, and attribute information). For example, a first digital human shape corresponding to A may be displayed to A, and a second digital human shape corresponding to B may be displayed to A. Also, for example, when the emotion of the living body is joy, a smiling expression and/or a digital human shape of bright clothes and accessories may be displayed. Also, for example, when the living body is a child, a child digital human shape is displayed; When the living body is an adult, an adult digital human shape may be displayed. Also, for example, when the living body is long-haired, a long-haired digital human shape is displayed; When the living body is single-shot, a single-shot digital human shape may be displayed.

In actual application, the digital human shape may be generated based on a preset digital human shape template. Here, the shape template may be a template previously created by a user or received from a server. For example, when a living body is riding, an image of the living body may be photographed with a camera of a vehicle or an image of the living body sent from a user terminal may be received, and a shape template of the living body may be generated based on the image of the living body. Specifically, by performing attribute detection on the image of the living body, the attributes of the living body may be acquired, and a digital human corresponding to the living body may be generated based on the attributes of the living body. Regeneration (eg, replacing an existing shape template with a new shape template) or partial editing (eg, changing the hairstyle of a shape template) may be performed on an already created shape template. When generating a shape template, the degree of cartoonization of the shape template can be customized.

When displaying the animation of the digital human, the shape of the digital human may be created by calling the corresponding template according to the actual situation. The template may be an adult shape template, a child shape template, or a pet shape template. When the living body is an adult, the adult shape template can be called; When the living body is a child, the child's shape template can be called. Further, the state information of the living body when riding may not match the template, so after calling the corresponding template, the attribute information of the digital human template is adjusted based on the state information of the living body, so that the digital The shape of the human and the state information of the living body may be matched. For example, according to the emotion of the living body, the facial expression of the digital human template and clothes and accessories are adjusted. Further, a display interface when displaying the digital human on the display device may be adjusted based on the state information of the living body. For example, when the emotion of the living body is joy, the background color of the display interface may be set to a bright color and/or the effect of scattering flowers on the display interface may be displayed.

By creating and displaying other digital human shapes in the above-described way, the digital human shape conforms to the living body to a certain extent, so that the living body feels intimate and gentle in the interaction process, and feedback information during the interaction process with the digital human To improve the degree of acceptance of the living body.

In addition to displaying an animation in which a digital human performs a corresponding action on a display device in the car cabin, to further enhance the interaction effect, voice information matching the state information is determined, and the animation is displayed, at the same time, the animation is displayed. Based on the voice information, the corresponding voice may be synchronously reproduced. For example, if the matching motion information is motion information corresponding to an action to unfasten the seat belt, an animation in which the digital human executes the corresponding action is displayed on the display device, and at the same time, "Little friend, seat belt in the course of driving" It is also possible to play the voice of one paragraph, "It is very dangerous to solve . In a practical application, a neural network is used to determine voice information matching the state information, or a mapping relationship between other state information and voice information is established in advance, and the matched voice information is also based on the mapping relationship. may decide Further, while the audio is reproduced, caption information corresponding to the audio may be displayed on the display interface.

Synchronization of the reproduced voice and the displayed animation indicates that the reproduced voice content matches an action executed by a digital human during the animation, wherein the action includes at least one of a gesture, a mouth action, an eye action, etc. can do. For example, when the digital human performs a motion of waving a hand during animation, and a mouth motion is a motion corresponding to “hello”, the voice content of “hello” is played. Specifically, determining voice information matching the state information; obtain a corresponding voice including a timestamp based on the voice information; While reproducing the voice, an animation in which the digital human executes the motion at a time corresponding to the timestamp may be generated and displayed based on the motion information. In the above-described manner, by synchronizing and reproducing the operation of the digital human among the reproduced voice and animation, the degree of personification of the digital human can be further improved, and the interaction between the digital human and the living body is made more natural.

The voice is pulled from a voice database, the fetched voice includes a timestamp of the voice, and the time during which the digital human performs a corresponding action during the animation is used to synchronize the voice with the voice. When retrieving a voice, it also acquires the state information of the living body (for example, at least one of attribute information and emotion information), and sends the state information of the living body to a voice database, You can even have the voice dragged in. For example, when the living body is a child, a voice matching the tone of the child is drawn.

A voice in a paragraph often includes a plurality of phonemes, a phoneme is a minimum phonetic unit divided according to the natural properties of a voice, and is analyzed according to the pronunciation operation of a syllable, and one pronunciation operation constitutes one phoneme. For example, "hello" includes two phonemes "an" and "hyeong". When a voice includes a plurality of phonemes, a timestamp of each phoneme may be included in the timestamp. In general, a plurality of sub-motions are included in one motion. For example, the motion of waving a hand may include a sub motion of waving an arm to the left and a sub motion of waving an arm to the right. In order to make the displayed digital human more lively and figurative, it is possible to match each sub-action and one phoneme of the voice, respectively. Specifically, based on the timestamp of each phoneme, determine an execution time of a sub-operation matching each phoneme; Based on the motion information, the digital human may generate and display an animation in which the digital human executes a sub-action matching the corresponding phoneme at the timestamp of the phoneme. For example, while playing the phoneme "an", at the same time displaying a mouth motion matching "an", and also displaying the motion of the digital human waving its arm to the left, while playing the phoneme "nyeong", " It displays a mouth-shaped motion that matches "nyeong", and also displays the digital human's right arm waving motion. By synchronizing each phoneme and the operation of the digital human, the accuracy of the synchronization process was improved, the operation and voice reproduction of the digital human more vivid, and the degree of personification of the digital human was further improved.

An operation corresponding to the above-described operation information may be called from the operation model base. Specifically, call a motion segment of at least one digital human corresponding to the motion information from the motion model base; Each frame among the frame of the motion image of the at least one digital human may be sequentially displayed on the display device. At least one of a gesture, a facial expression action, a mouth action, an eye action, etc. of the digital human corresponding to the different motion image frames is different, and by calling and sequentially displaying the corresponding motion image frames, the digital human is displayed on the display device. An animation of the corresponding action being executed may be displayed. In the above manner, the animation of the digital human can be displayed by calling only the corresponding frame, and the display efficiency is high and the cost is low.

In order to add more voice material and motion material to the corresponding voice database and motion model base, both the above-mentioned voice database and motion model base can be updated. The updating method of the voice database and the motion model base is similar, and here, taking the voice database as an example, the updating method of the database will be described, and the updating method of the motion model base may refer to the voice database, Here, duplicate descriptions will be omitted. When a voice database is updated, an update package is downloaded from the server, data in the update package is analyzed, and data in the update package is added to the voice database or unique in the voice database as data in the update package. data can be replaced.

In one embodiment, the voice database may be updated at regular time intervals; In another embodiment, receive an update prompt message pushed by the server, and update the voice database in response to the update prompt message; In another embodiment, after receiving the update command sent from the living body, sending an update request to the server in response to the update command, and receiving the update package returned by the server in response to the update request, The voice database may be updated based on the update package. In addition, the voice database may be updated in other ways, and the redundant description will be omitted here.

In some embodiments, it is also possible to obtain the driving state of the vehicle, and determine the driving state of the vehicle and operation information respectively matched with the state information. For example, when the vehicle is driving, when a gesture of the living body is detected as an operation to unfasten the seat belt, the matching operation information includes operation information corresponding to the movement of the living body not to unfasten the seat belt. When the vehicle is stopped and the engine is turned off, if a body gesture is detected as a motion to loosen the seat belt, it is determined that the matching motion information includes motion information corresponding to the motion of waving a hand to say goodbye. In this way, it is possible to more accurately determine the matching motion information, thereby reducing the probability of misjudgment.

In some embodiments, the execution state of 89 may be controlled based on the state information. The in-vehicle device includes at least one of a lighting device in an automobile cabin, an air conditioner, a window, an audio playback device, and a seat. For example, when the living body is tired or has already fallen asleep, the seat may be laid so that the living body may lie flat on the available seat. Also, for example, when the living body is hot, a car window is opened or an air conditioner is turned on to adjust the car cabin environment in the car cabin, thereby providing a more comfortable and safe riding environment to the living body.

In practical applications, through the embodiments of the present application, an action of a living body entering a car cabin or leaving a seat in the car cabin, opening a car door or closing a car door, fastening or unfastening a seat belt, etc. is detected. Also, based on the state information of the living body, an animation in which the digital human executes a corresponding action may be displayed, so that each of the above-described actions may be interacted with the living body.

For example, detecting that the living body enters the car cabin or leaves the seat in the car cabin, in some embodiments, acquiring the status information of the living body riding in the car cabin includes: acquiring a target image in the car cabin; and recognizing the living body in the target image, and determining whether the living body is located in a seat in a car cabin based on the location information of the living body. The target image may be acquired based on a surveillance control video in a car cabin. Specifically, object information (including position information of the center point of the object and object type information corresponding to the center point of the object) of each object in the target image is determined, and each of the target images is determined based on the object type information The living body and the seat are selected from among the objects of , and it can be determined whether the living body is located in the seat based on the position of the center point of the living body and the position of the center point of the seat. In response to determining that the living body is not located in the seat, a prompt message is sent. The object in the target image may include a human face, a human body, a rear seat, a car seat, and the like. For example, if it is detected that the living body is not in the seat while the vehicle is being driven, it may be determined that the living body is not wearing the seat belt, and a digital human corresponding to the living body sits on the seat on the control panel and the seat belt While displaying an animation of running a demonstration motion of wearing make it match

In some embodiments, location information of the center point of each object in the target image may be determined through steps 601 to 604 below.

In step 601, feature extraction is performed on the target image to obtain a first feature map corresponding to the target image. In some embodiments, first, the target image is input to the first neural network and image feature extraction is performed to obtain one initial feature map. Thereafter, the initial feature map is input to a second neural network for extracting object information to obtain the above-described first feature map. The above-described target image may be an image having one size of 640*480 pixels, and an initial feature map of 80*60*C may be obtained after first neural network processing. Among them, C represents the channel quantity. After the initial feature map is processed by the second neural network for extracting object information, a first feature map of 80*60*3 can be obtained.

In step 602, from among the first preset channels of the first feature map, each feature point in the first feature map is acquired as a response value of an object center point. The first preset channel may be channel 0 of the first feature map, the channel is a channel of an object center point, and a response value in the channel may indicate the possibility that each feature point in the first feature map is a center point of an object. . After obtaining a response value corresponding to each feature point among the first preset channels, the response value may be switched between 0 and 1 using a sigmoid.

In step 603, the first feature map is divided into a plurality of sub-regions, and a maximum response value and a feature point corresponding to the maximum response value in each sub-region are determined. In some embodiments, a max pooling operation in which a step of 3×3 is 1 is performed on the first feature map, and the maximum response value within 3×3 and the position index of the maximum response value in the first feature map are performed. can get It is possible to obtain 60×80 maximum response values and a position index corresponding to each of these maximum response values. Thereafter, by merging the same location indexes, N maximum response values, a location index corresponding to each maximum response value, and a feature point corresponding to each maximum response value may be obtained.

In step 604, position information of the center point of the object is determined based on the position index of the center point of the object in the first feature map by using a feature point with a maximum response value greater than a preset threshold as the center point of the object. A threshold value thrd may be preset, and when the maximum response value is greater than thrd, a feature point corresponding to the maximum response value is determined as the center point of the object.

In the above-described embodiment, the maximum pooling process is performed on the response values in the first feature map to find the feature point most likely to be the center point of the object within the local range, thereby effectively improving the accuracy of the determined center point. can

In some embodiments, the center point of the object and location information of the center point are used as center point information of the object. In some embodiments, the center point information may include length information and width information of the center point of the object. In this case, length information and width information of the center point can be determined through the following steps:

In the second preset channel of the first feature map, at a position corresponding to the position index of the center point of the object, information on the length of the center point of the object is obtained. In the third preset channel of the first feature map, at a position corresponding to the position index of the center point of the object, information on the width of the center point of the object is obtained.

The above-described second preset channel may be a first channel in the first feature map, and the third preset channel may be a second channel in the first feature map. In the first channel of the first feature map, information on the length of the center point is obtained at a position corresponding to the center point, and information on the width of the center point is obtained at a position corresponding to the center point among the second channels in the first feature map.

After determining the center point of the object, length information and width information of the center point of the object may be accurately obtained from among other preset channels of the first feature map by using the position index of the center point of the object.

Since the object may include a human face, a human body, a rear seat, a car seat, etc., when specifically implemented, a first feature map corresponding to another object is determined using another neural network, and then another first feature map is determined based on another first feature map. The center point of the object, location information of each center point, length information of each center point, and width information of each center point must be determined.

In some embodiments, object type information corresponding to the center point of the object in the target image is determined through the following steps 701-703:

In step 701, feature extraction is performed on the target image to obtain a second feature map corresponding to the target image. After inputting the target image into the third neural network and performing image feature extraction to obtain one initial feature map, the initial feature map is input and processed into the fourth neural network for object type recognition to obtain a second feature map, , determine object type information corresponding to the center point of the object based on the second feature map. The above-described second feature map may be a single 80*60*2 feature map. The above-described third neural network may be the same as the first neural network.

In the application scene for recognizing children, one two-dimensional feature vector corresponds to each feature point in the second feature map, and the center point of the object corresponds to the two-dimensional feature vector of the feature point in the second feature map described above. Execute classification processing to obtain a classification result, and if one classification result represents a child and the other classification result represents another object, an object corresponding to the center point of the object based on the above-described classification result It is possible to determine whether the type information is a child. In an application scene for recognizing a child, the above-described object may be a human body or a human face.

In the application scene for recognizing the car seat, one two-dimensional feature vector corresponds to each feature point in the second feature map, and the center point of the object corresponds to the two-dimensional feature vector of the feature point in the second feature map described above. Execute classification processing to obtain a classification result, and if one classification result represents a car seat and another classification result represents another object, the object corresponding to the center point of the object based on the above-described classification result It can be determined whether the type information is a car seat. Using the same method, it can be understood that the rear seat and the like can be recognized.

Since the object may include a human face, a human body, a rear seat, a car seat, etc., when specifically implemented, a second feature map corresponding to another object is determined using another neural network, and then another second feature map is determined based on another second feature map. It is necessary to determine the object type information of the object.

In step 702, the position index of the center point of the object in the second feature map is determined based on the position index of the center point of the object in the first feature map.

In step 703, object type information corresponding to the center point of the object is obtained at a location corresponding to the location index of the center point of the object in the second feature map. After determining the center point of the object as described above, object type information corresponding to the center point of the object may be accurately obtained by using the location index of the center point.

After determining object type information corresponding to the center point of each object, a specific group (eg, children, pets, etc.) in the target image is recognized through the following steps 801-803, and a digital human corresponding to the specific group and specific groups to implement the interaction. For convenience of explanation, the following description will be made with a child as an example, and the recognition methods of other groups are similar, so duplicate descriptions will be omitted here.

In step 801, based on the position offset information corresponding to the center point of each human body, the predicted position information of the center point of the human face matching each human body is determined, respectively; Here, the human body and the face belonging to the same person are matched with each other. When determining the above-described position offset information, an initial feature map may be obtained by first inputting a target image to the fifth neural network and performing image feature extraction. After that, the initial feature map is again input to the sixth neural network for determining the above-described position offset information to obtain one feature map, and position offset information corresponding to the center point of each human body can be determined based on the feature map. The feature map may be a single 80*60*2 feature map. The fifth neural network may be the same as the first neural network.

In step 802, a human face matching each human body is determined based on the determined predicted position information and the position information of the center point of each human face. In some embodiments, the human face corresponding to the position of the center point closest to the position corresponding to the predicted position information may be a human face matching the human body.

In step 803, for the human body and human face that has been successfully matched, the person to which the matching successful human body and human face belongs by using object type information corresponding to the center point of the human body and object type information corresponding to the center point of the human face Decide whether this is a child. When the object type information corresponding to the center point of the human body that has succeeded in matching indicates a child to which the corresponding human body belongs, and the object type information corresponding to the center point of the human face, the person to which the corresponding human face belongs points to a child; The person to which the human body and human face that has succeeded in the matching belong is determined as a child. For a human body that does not succeed in matching, it is determined whether the person to which the center point of the human body belongs is a child by using object type information corresponding to the center point of the human body. Specifically, if the object type information corresponding to the center point of the human body indicates a child, the person to which the human body belongs is determined as the child.

Hereinafter, in conjunction with a specific example, a method according to an embodiment of the present application will be described.

When a living body gets on and sits down, a monitoring control video of the living body is acquired through the monitoring and control system, and it is detected that the classification information of the living body is human, the identity information of the living body is A, and the attribute information is a child, and at the same time, the living body A When it is detected that the emotional information is joy, it creates and displays an animation of a child digital human with a smile on his face and raising his hand in bright clothing and accessories to say hello, while at the same time emitting a greeting voice saying "A, you look good today" 3, the mouth and gestures of the digital human during animation are matched with the reproduced voice. If the biometric identity is not recognized, a common name, for example, a little friend, a teacher, etc. may be used according to attribute information such as gender and age when greeting. After the living body A is seated, if the emotion of A is relatively calm at this time, the voice interaction may not be performed. When it is detected that living body A is not wearing a seat belt after starting the vehicle, an animation of a demonstration motion of a child-shaped digital human wearing a seat belt is displayed on the control panel, and at the same time, "A, wear a seat belt with me. Let's do it" is played, and the digital human's mouth and gestures during animation are matched with the reproduced voice.

After a certain period of time, when it is detected that A says "it's hot", the window is opened and the air conditioner is activated. Also, after a certain time has elapsed, A interacts with the digital human in a preset manner (for example, by touching the digital human, the gaze gazes at the digital human, or calls the digital human by voice, etc.), and the digital human Interaction with this A is executed, the interaction mode may be dialogue, game or music playback control, and upon interaction, a corresponding digital human animation is displayed according to the content of the interaction, and the voice is synchronized. When A gets off, the monitoring and control system detects that an object has been left on A's seat, and displays an animation of the digital human waving on the control panel while simultaneously displaying "A, object was dropped into the car, Come and take it," the voice is played in sync. After A gets off, he can close the car window and turn off the music.

According to the embodiment of the present application, based on the state information of the living body in the car cabin, an animation in which a digital human performs a corresponding action is generated and displayed on a display device in the car cabin, thereby realizing a personified interaction, the interaction The process is more natural, makes the human-machine interaction feel warm, and improves riding pleasure, comfort, and companionship, thereby improving the degree of acceptance of the living body for the feedback information during the interaction process, thereby improving the living body in the vehicle driving process. improve the safety of In addition, the embodiment of the present application generates an animation of a digital human in a software manner, so that the cost is relatively low, the response speed of the digital human is relatively fast, and it is also convenient for post-maintenance, update, and upgrade.

A person skilled in the art will not want to limit the implementation process by strictly executing the order in the writing order of each step in the above-described specific method, and the specific implementation order of each step may be based on its function or internal logic understand what can be determined with

As shown in FIG. 4 , the present application further provides an automobile cabin interaction device based on a digital human, the device comprising:

an acquisition module 401 configured to acquire status information of a living body riding in an automobile cabin;

a determining module (402) configured to determine operation information matching the state information; and

and a display module 403 configured to generate an animation in which a digital human performs a corresponding operation based on the operation information and display it on a display device in the automobile cabin.

In some embodiments, the display module 403 includes: a first determining unit, configured to determine voice information matching the state information; a first acquiring unit, configured to acquire a corresponding voice including a timestamp based on the voice information; and a first display unit configured to reproduce the voice and simultaneously generate and display an animation in which the digital human executes the operation at a time corresponding to the timestamp based on the operation information.

In some embodiments, the operation includes a plurality of sub-operations, each sub-operation matches one phoneme in the voice, and the timestamp includes a timestamp of each phoneme; The first display unit includes: a determining subunit configured to determine an execution time of a sub-operation matching each phoneme based on a timestamp of each phoneme; and a display sub-unit configured to generate and display an animation in which the digital human executes a sub-action matching the corresponding phoneme at the timestamp of each phoneme, based on the motion information.

In some embodiments, the display module 403 includes: a calling unit, configured to call a motion segment of at least one digital human corresponding to the motion information in a motion model base; and a second display unit configured to display each frame of the frame of the motion image of the at least one digital human on the display device in order.

In some embodiments, the biological status information includes the first status information of the living body, and the acquiring module 401 includes: a collection unit configured to collect a surveillance control video of a rear row in a car cabin; and a detection and analysis unit, configured to execute biometric detection on the monitoring control video and perform state analysis on the detected living body to obtain first state information of the living body.

In some embodiments, the surveillance control video is captured via a video acquisition device mounted on a rearview mirror in the car cabin and with a lens facing the back row of the car cabin.

In some embodiments, the first state information includes at least one of classification information of the living body, identification information, attribute information, emotion information, facial expression information, gesture information, seating information, and seat belt wearing information; and/or the living body includes at least one of a driver, a co-pilot, a child, an elderly person, a pet, and a back-row passenger.

In some embodiments, the state information of the living body includes the first state information and second state information of the living body, wherein the first state information is obtained based on a surveillance control video in a car cabin; the acquiring module 401 is further configured to acquire second status information sent from the smart device carried by the living body; The determining module 402 is configured to determine operation information matching both the first state information and the second state information.

In some embodiments, the second state information includes at least one of health state information and nervous system state information.

In some embodiments, the acquisition module (401) comprises: an input unit, configured to input the surveillance control video in the car cabin into a pre-trained neural network; and a second determining unit configured to determine the state information of the living body based on an output result of the neural network.

In some embodiments, the apparatus is configured to generate an animation in which a digital human performs a corresponding operation based on the operation information and to generate the shape of the digital human before displaying it on a display device in the car cabin It further includes a creation module.

In some embodiments, the generating module generates the shape of the digital human based on the state information of the living body; Alternatively, it is configured to generate the shape of the digital human based on a preset shape template of the digital human.

In some embodiments, the attribute information of the living body includes at least one of age, gender, appearance, body type, clothes and accessories, hair style, and skin color.

In some embodiments, the apparatus further includes a control module configured to control an execution state of the on-vehicle device based on the state information.

In some embodiments, the determining module 402 obtains the driving state of the vehicle; It is set to determine the driving state of the vehicle and operation information respectively matched with the state information.

In some embodiments, the acquiring module 401 is configured to recognize a living body riding in the car cabin based on the target image in the car cabin; It is further configured to determine whether the living body is located in a seat in the automobile cabin based on the location information of the living body.

In some embodiments, the acquiring module 401 is configured to determine object information of each object in the target image in the car cabin; selecting the living body and the seat in the car cabin in the object based on object type information of each of the objects; further configured to determine whether the living body is located in the seat based on the center point position of the living body and the center point position of the seat, wherein, with respect to each of the objects, the object information of the object is the center point of the object location information and object type information corresponding to the center point of the object.

In some embodiments, the display module 403 generates an animation in which a digital human corresponding to a living body not located in the seat sits on a seat and performs a demonstration operation of wearing a seat belt, and also in the car cabin. It is further set to display on a display device.

In some embodiments, the device further comprises a notification module configured to send a prompt message in response to determining that the living body is not located in the seat.

In some embodiments, the acquiring module 401 may further include: performing feature extraction on the target image to obtain a first feature map corresponding to the target image; acquiring each feature point in the first feature map from among first preset channels of the first feature map as a response value of an object center point; dividing the first feature map into a plurality of sub-regions and determining a maximum response value and a feature point corresponding to the maximum response value in each sub-region; and using a feature point with a maximum response value greater than a preset threshold as the center point of the object, and determining the location information of the center point of the object based on the location index of the center point of the object in the first feature map. It is set to determine position information of the center point of the object in the target image.

In some embodiments, the acquiring module 401 performs feature extraction on the target image to obtain a second feature map corresponding to the target image; determining a location index of the center point of the object in the second feature map based on the location index of the center point of the object in the first feature map; and acquiring object type information corresponding to the center point of the object at a location corresponding to the location index of the center point of the object in the second feature map, and the object type corresponding to the center point of the object in the target image set to determine information.

In some embodiments, functions provided by the apparatus provided in the embodiments of the present application or modules included may be used to execute the methods described in the method embodiments, and for the sake of concise description, redundant descriptions are omitted here. to do it

Embodiments of the present specification further provide a computer device, wherein the computer device includes at least a memory, a processor, and a computer program stored in the memory and executable in the processor, wherein the processor executes the program when the program executes any of the aforementioned implementations. Implement the method described in the example.

5 shows a schematic diagram of a computer device hardware structure provided in an embodiment of the present specification, wherein the device includes a processor 501 , a memory 502 , an input/output interface 503 , a communication interface 504 and a bus 505 . may include Among them, the processor 501 , the memory 502 , the input/output interface 503 , and the communication interface 504 realize interconnection communication within the device via the bus 505 .

The processor 501 is implemented in a general-purpose CPU (Central Processing Unit, Central Processing Unit), microprocessor, Application Specific Integrated Circuit (ASIC) or in one or a plurality of integrated circuits, etc., in the embodiment of the present specification A related program may be executed to realize the technical solution provided.

The memory 502 may be implemented in the form of a read only memory (ROM), a random access memory (RAM), a static storage device, a dynamic storage device, or the like. The memory 502 may store an operating system and other application programs, and when realizing the technical solution provided in the embodiment of the present specification in software or firmware, the related program code is stored in the memory 502 and executed by the processor 501 . called and executed.

The input/output interface 503 is used to connect input/output modules to implement input and output of information. The input/output module may be configured in the device as a component (not shown), or may be externally connected to the device to provide a corresponding function. Among them, the input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, and the like, and the output device may include a monitor, a speaker, a vibrator, an indicator, and the like.

The communication interface 504 is used to connect a communication module (not shown) to realize communication interaction between the device and another device. Among them, the communication module may realize communication in a wired method (eg, USB, LAN cable, etc.) or may realize communication in a wireless method (eg, mobile network, WIFI, Bluetooth, etc.).

Bus 505 includes one path and transfers information between each component of the device (eg, processor 501 , memory 502 , input/output interface 503 , and communication interface 504 ).

It should be noted that, in the above-described device, only the processor 501, the memory 502, the input/output interface 503, the communication interface 504, and the bus 505 are shown. It may include other components necessary for the realization of In addition, a person skilled in the art may understand that the above-described device may include only components essential for realizing the embodiments of the present specification, and need not include all illustrated components.

6A and 6B , the embodiment of the present application further provides a vehicle, wherein a display device 601, a supervisory control system 602, and a digital human-based automobile cabin interaction apparatus in the vehicle cabin of the vehicle ( 603) or a computer device 604 is installed.

The display device 601 is used to display an animation in which the digital human performs a corresponding operation. The display device 601 may include at least one of a control panel of the vehicle and a screen mounted on the back of the vehicle seat.

The monitoring control system 602 may include at least one of a visual monitoring control system and a voice monitoring control system, and the visual monitoring control system may include at least one camera, wherein the camera is located above an area to be monitored and controlled. It can be mounted on the monitor and is used to collect video or images of the area to be monitored and controlled. For example, the camera may be mounted on a windshield of a vehicle or mounted on a position such as above a seat. Also for example, the camera may be mounted on a rearview mirror in the car cabin, and the lens may be directed towards the back row of the car cabin. This mounting method widens the field of view of the camera, and it is convenient to acquire surveillance control video of the rear row of the car cabin. The voice monitoring control system, including at least one microphone, may be used to collect the audio signal of the monitoring control waiting area.

The digital human-based car cabin interaction apparatus 603 according to the embodiment of the present application may use the car cabin interaction apparatus according to any of the above-described embodiments, and the computer device 604 according to the embodiment of the present application is described above. A computer device according to any one embodiment may be used. The digital human-based car cabin interaction device 603 or computer device 604 may be integrated into a central management system of the vehicle. The supervisory control system 602 may communicate with the in-vehicle communication device 603 or the computer device 604 via an in-vehicle communication bus, for example, the in-vehicle communication bus may be connected to a controller area network (Controller Area Network). , CAN) bus.

In some embodiments, the digital human-based car cabin interaction apparatus 603 or computer device 604 also includes an on-vehicle device based on the status information, for example, a lighting device in a car cabin, an air conditioner, a car window, At least one of the audio reproducing device and the seat may be controlled.

161 In some embodiments, the digital human-based car cabin interaction device 603 or computer device 604 is also connected to a user's smart device or a voice database, an operation model database, etc. through a network, so that the smart device Alternatively, data interaction with the voice database and the motion model database may be performed.

An embodiment of the present application further provides a computer-readable storage medium in which a computer program is stored, and when the program is executed by a processor, the method described in any of the above-described embodiments is realized.

Computer-readable media includes permanent and non-persistent, removable and non-removable media, and storage of information may be realized by any method or technology. Information may be computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory ( RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape may be configured to store information that can be accessed by computing devices, including, but not limited to, storage or other magnetic storage devices or any other non-transmission medium. By the definition in this paper, computer readable media does not include transitory media such as modulated data signals and carrier waves.

From the description of the above implementations, it can be seen that those skilled in the art can clearly understand that the embodiments of the present specification can be implemented by software and a necessary general-purpose hardware platform. Based on this understanding, for the technical solutions of the embodiments of the present description, an essential part thereof, that is, a part contributing to the prior art, may be embodied in the form of a software product. The computer software product may be stored in a storage medium. For example, ROM/RAM, magnetic disk, optical disk, etc. A computer software product may include several instructions that enable a computer device (which may be a personal computer, server, or network device, etc.) to carry out the method described in each embodiment or some portion of an embodiment of this description.

An embodiment of the present application further provides a computer program product, wherein the computer program product includes computer instructions, and when the computer instructions are executed by a processor, the method described in any of the above-described embodiments can be realized.

The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or may be implemented by products having specific functions. A typical implementation is a computer, and certain types of computer are personal computers, laptop computers, cellular phones, camera phones, smart phones, personal digital assistants, media players, navigation devices, email transceiver devices, game consoles, It may be a tablet computer, a wearable device, or a combination of any of these devices.

Various embodiments in the present description are described in a progressive manner, and parts similar to each other may be referred to. The description of each embodiment is different from other embodiments. In particular, for the apparatus embodiments, since they are basically similar to the method embodiments, the description is simplified, and corresponding parts of the description of the method embodiments may be referred to. In the device embodiments described above, modules described as separate components may or may not be physically separated, and the functions of the modules may be implemented in one or more software and/or hardware when the embodiments of the present description are implemented. These are just schematics. Some or all of the modules may be selected according to actual requirements to implement the objectives of the solutions in the embodiments. Those skilled in the art can understand and implement the present disclosure without creative work.

Claims (24)

In the car cabin interaction method based on digital human,
acquiring status information of a living body riding in a car cabin;
determining operation information matching the state information; and
generating an animation in which a digital human performs a corresponding motion based on the motion information and displaying the animation on a display device in the car cabin
A car cabin interaction method based on a digital human, characterized in that. According to claim 1,
generating an animation in which a digital human performs a corresponding motion based on the motion information and displaying the animation on a display device in the car cabin,
determining voice information matching the state information;
obtaining a corresponding voice including a timestamp based on the voice information; and
generating and displaying an animation in which the digital human executes the motion at a time corresponding to the timestamp based on the motion information while reproducing the voice
A car cabin interaction method based on a digital human, characterized in that. 3. The method of claim 2,
the operation includes a plurality of sub-operations, each sub-operation matches one phoneme in the voice, and the timestamp includes a timestamp of each phoneme;
generating and displaying an animation in which the digital human executes the motion at a time corresponding to the timestamp based on the motion information,
determining an execution time of a sub-operation matching each phoneme based on the timestamp of each phoneme; and
based on the motion information, generating and displaying an animation in which the digital human executes a sub-action matching the corresponding phoneme at a timestamp of each phoneme;
A car cabin interaction method based on a digital human, characterized in that. 4. The method according to any one of claims 1 to 3,
generating an animation in which a digital human performs a corresponding motion based on the motion information and displaying the animation on a display device in the car cabin,
calling a motion segment of at least one digital human corresponding to the motion information from a motion model base; and
displaying on the display device each frame of the frame of the motion image of the at least one digital human in order
A car cabin interaction method based on a digital human, characterized in that. 5. The method according to any one of claims 1 to 4,
The state information of the living body includes the first state information of the living body, and the step of obtaining the state information of the living body riding in a car cabin includes:
collecting surveillance control video of a rear row in a car cabin; and
performing biometric detection on the monitoring control video and performing state analysis on the detected living body to obtain first state information of the living body
A car cabin interaction method based on a digital human, characterized in that. 6. The method of claim 5,
The surveillance control video is mounted on a rearview mirror in the car cabin, and also acquired through a video collecting device with a lens facing the rear row of the car cabin.
A car cabin interaction method based on a digital human, characterized in that. 7. The method of claim 5 or 6,
the first state information includes at least one of classification information of the living body, identification information, attribute information, emotion information, facial expression information, gesture information, seating information, and seat belt wearing information; and/or
The living body includes at least one of a driver, a co-pilot, a child, the elderly, a pet, and a passenger in the back row
A car cabin interaction method based on a digital human, characterized in that. 8. The method according to any one of claims 1 to 7,
the biological status information includes first status information and second status information of the living body, wherein the first status information is obtained based on a surveillance control video in a car cabin;
The acquiring of the status information of the living body in the car cabin further includes acquiring second status information sent from a smart device carried by the living body;
The step of determining operation information matching the state information includes:
Further comprising the step of determining operation information that matches both the first state information and the second state information
A car cabin interaction method based on a digital human, characterized in that. 9. The method according to any one of claims 1 to 8,
The step of obtaining the status information of the living body riding in the car cabin,
inputting surveillance control video in the car cabin into a pre-trained neural network; and
Determining the state information of the living body based on the output result of the neural network
A car cabin interaction method based on a digital human, characterized in that. 10. The method according to any one of claims 1 to 9,
Generating an animation in which a digital human performs a corresponding motion based on the motion information and further comprising generating a shape of the digital human before displaying it on a display device in the car cabin
A car cabin interaction method based on a digital human, characterized in that. 11. The method of claim 10,
The step of creating the shape of the digital human comprises:
generating the shape of the digital human based on the state information of the living body;
or
Creating a shape of the digital human based on a preset shape template of the digital human
A car cabin interaction method based on a digital human, characterized in that. 12. The method according to any one of claims 1 to 11,
Further comprising the step of controlling the execution state of the vehicle-mounted device based on the state information
A car cabin interaction method based on a digital human, characterized in that. 13. The method according to any one of claims 1 to 12,
The step of determining operation information matching the state information includes:
acquiring a driving state of the vehicle; and
Comprising the step of determining the driving state of the vehicle and operation information that is respectively matched with the state information
A car cabin interaction method based on a digital human, characterized in that. According to claim 1,
The step of obtaining the status information of the living body riding in the car cabin,
recognizing a living body riding in the car cabin based on the target image in the car cabin; and
Based on the location information of the living body, further comprising the step of determining whether the living body is located in a seat in the car cabin
A car cabin interaction method based on a digital human, characterized in that. According to claim 1,
The step of obtaining the status information of the living body riding in the car cabin is
determining object information of each object in the target image in the vehicle cabin, wherein, with respect to each of the objects, the object information of the object includes location information of a center point of the object and an object type corresponding to the center point of the object contains information -;
selecting the living body and the seat in the car cabin from the object based on object type information of each of the objects; and
Determining whether the living body is located in the seat based on the center point position of the living body and the center point position of the seat
A car cabin interaction method based on a digital human, characterized in that. 16. The method of claim 15,
generating an animation in which a digital human performs a corresponding motion based on the motion information and displaying the animation on a display device in the car cabin,
generating an animation in which a digital human corresponding to a living body not located in the seat sits on a seat and performs a demonstration operation of wearing a seat belt, and displaying the animation on a display device in the car cabin
A car cabin interaction method based on a digital human, characterized in that. 17. The method of claim 16,
in response to determining that the living body is not located in the seat, sending a prompt message
A car cabin interaction method based on a digital human, characterized in that. 16. The method of claim 15,
Determining the location information of the center point of the object in the target image,
performing feature extraction on the target image to obtain a first feature map corresponding to the target image;
acquiring each feature point in the first feature map from among first preset channels of the first feature map as a response value of an object center point;
dividing the first feature map into a plurality of sub-regions and determining a maximum response value and a feature point corresponding to the maximum response value in each sub-region; and
Using a feature point with a maximum response value greater than a preset threshold as the center point of the object, and determining the location information of the center point of the object based on the location index of the center point of the object in the first feature map.
A car cabin interaction method based on a digital human, characterized in that. 19. The method of claim 18,
Determining object type information corresponding to the center point of the object in the target image includes:
performing feature extraction on the target image to obtain a second feature map corresponding to the target image;
determining a location index of the center point of the object in the second feature map based on the location index of the center point of the object in the first feature map; and
obtaining object type information corresponding to the center point of the object at a location corresponding to the location index of the center point of the object in the second feature map;
A car cabin interaction method based on a digital human, characterized in that. In the car cabin interaction device based on digital human,
an acquisition module configured to acquire status information of a living body riding in an automobile cabin;
a determining module, configured to determine operation information matching the state information; and
a display module configured to generate an animation in which a digital human executes a corresponding operation based on the operation information and display it on a display device in the automobile cabin
A car cabin interaction device based on a digital human, characterized in that. A computer program is stored, and when the program is executed by a processor, the method according to any one of claims 1 to 19 is realized.
A computer-readable storage medium, characterized in that. 20. A method comprising a memory, a processor and a computer program stored in the memory, executable by the processor, and when the processor executes the program, the method according to any one of claims 1 to 19 is realized.
Computer device, characterized in that. A display device, a monitoring control system, and the digital human-based automobile cabin interaction apparatus according to claim 20 or the computer device according to claim 22 are installed in an automobile cabin.
Vehicle characterized in that. 20. A method comprising computer instructions, which when executed by a processor realizes the method according to any one of claims 1 to 19.
A computer program product, characterized in that

Images