CN112735440A

CN112735440A - Vehicle-mounted intelligent robot interaction method, robot and vehicle

Info

Publication number: CN112735440A
Application number: CN202011629125.2A
Authority: CN
Inventors: 陈万慧; 胡泽钢
Original assignee: Beijing Overlooking Technology Co ltd
Current assignee: Beijing Yanyan Zhiyu Technology Co ltd
Priority date: 2020-12-30
Filing date: 2020-12-30
Publication date: 2021-04-30

Abstract

The invention provides a vehicle-mounted intelligent robot interaction method, which comprises the following steps: acquiring user voice information and calculating a sound source position; controlling the robot to rotate towards the sound source position to acquire user image information; judging the identity of the user according to the sound source position; when the user identity is a driver, recognizing the emotion and driving state of the user according to the image information and the voice information of the user, and entering a first interaction mode; and when the user identity is a passenger, recognizing the emotion of the user according to the image information and the voice information of the user, and entering a second interaction mode. In addition, the invention also provides a robot and a vehicle.

Description

Vehicle-mounted intelligent robot interaction method, robot and vehicle

Technical Field

The invention relates to the technical field of vehicle-mounted robots, in particular to a vehicle-mounted intelligent robot interaction method, a robot and a vehicle.

Background

With the increase of the number of global automobiles, the automobiles are the most important transportation tools for people, the use range of the road vehicles is on the rise at present, and traffic jam becomes a common problem of urban traffic, so that a driver needs to drive the vehicles for a long time. Long-term driving can lead to poor mood and fatigue driving for the driver. Passengers can feel uninteresting for a long time when riding the vehicle.

With the development of robotics, more and more vehicle-mounted robot apparatuses are emerging. However, the existing vehicle-mounted robot has single function, poor interaction capability and complex operation, and most of the existing vehicle-mounted robots are limited to the aspects of in-vehicle entertainment, information, navigation, electronic maps based on position solutions and the like. Therefore, the robot which can soothe the driver and passengers in the vehicle and assist the driver in driving has important social and practical significance.

Disclosure of Invention

The invention provides an interaction method of an on-vehicle intelligent robot, the robot and a vehicle, which can identify the emotion of a user and perform good interaction with the user.

In a first aspect, an embodiment of the present invention provides a vehicle-mounted intelligent robot interaction method, where the vehicle-mounted intelligent robot interaction method includes:

acquiring user voice information and calculating a sound source position;

controlling the robot to rotate towards the sound source position to acquire user image information;

judging the identity of the user according to the sound source position;

when the user identity is a driver, recognizing the emotion and driving state of the user according to the image information and the voice information of the user, and entering a first interaction mode;

and when the user identity is a passenger, recognizing the emotion of the user according to the image information and the voice information of the user, and entering a second interaction mode.

In a second aspect, embodiments of the present invention provide a robot, comprising:

the voice information module is used for acquiring the voice information of the user and calculating the position of a sound source;

the action control module is used for controlling the robot to rotate towards the sound source position;

the recognition module is used for recognizing the emotion and the driving state of the user according to the image information and the voice information of the user when the identity of the user is a driver, and recognizing the emotion of the user according to the image information and the voice information of the user when the identity of the user is a passenger, and entering a second interaction mode;

and the processing module is used for judging the identity of the user, entering a first interaction mode and entering a second interaction mode.

And the image collection module is used for acquiring the image information of the user.

In a third aspect, an embodiment of the present invention provides a robot, where the robot includes a processor and a memory, where the memory is used to store program instructions of the vehicle-mounted intelligent robot interaction method, and the processor is used to execute the program instructions of the vehicle-mounted intelligent robot interaction method, so as to implement the vehicle-mounted intelligent robot interaction method.

In a fourth aspect, an embodiment of the present invention provides a vehicle including a vehicle body and the robot provided to the vehicle body.

According to the vehicle-mounted intelligent robot interaction method, the robot and the vehicle, the user voice message is obtained, the sound source position is identified, the user image information is obtained, the semantic identification and the face identification are carried out according to different user attributes, the user emotion is obtained, the pacification is carried out in a targeted manner, the accompanying can be given to the user in a long-time driving journey, the long-distance driving is not boring and tasteless, and the driving experience is improved. Meanwhile, the driving safety of the driver can be monitored or the road condition and objects around the vehicle can be identified so as to remind the driver of paying attention to the driving safety, so that the driving safety is improved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the structures shown in the drawings without creative efforts.

Fig. 1 is a flowchart of an interaction method of a vehicle-mounted intelligent robot according to an embodiment of the present invention.

Fig. 2 is a flowchart illustrating a first embodiment of a first interaction mode according to the present invention.

Fig. 3 is a flowchart illustrating a second embodiment of the first interaction mode according to the present invention.

Fig. 4 is a flowchart illustrating a third embodiment of the first interaction mode according to the present invention.

FIG. 5 is a flow chart illustrating a second interaction mode provided by the present invention.

FIG. 6 is a schematic flow chart illustrating a first embodiment of the present invention.

FIG. 7 is a flow chart illustrating a second embodiment of the present invention.

Fig. 8 is a schematic diagram of a robot module according to the present invention.

Fig. 9 is a schematic diagram of the internal structure of the robot provided by the present invention.

The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. The drawings illustrate examples of embodiments of the invention, in which like numerals represent like elements. It is to be understood that the drawings are not to scale as the invention may be practiced in practice, but are for illustrative purposes and are not to scale. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The terms "first," "second," "third," "fourth," and the like in the description and in the claims of the present application and in the drawings described above, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances, in other words that the embodiments described are to be practiced in sequences other than those illustrated or described herein. Moreover, the terms "comprises," "comprising," and any other variation thereof, may also include other things, such as processes, methods, systems, articles, or apparatus that comprise a list of steps or elements is not necessarily limited to only those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such processes, methods, articles, or apparatus.

It should be noted that the description relating to "first", "second", etc. in the present invention is for descriptive purposes only and is not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present invention.

Please refer to fig. 1 in combination, which is a flowchart of an interaction method of an intelligent robot mounted on a vehicle according to an embodiment of the present invention.

Step S100, acquiring user voice information and calculating the position of a sound source. Specifically, the method obtains the user voice information through a sound sensor array disposed on the robot, where the sound sensor array includes a plurality of sound sensors, and calculates the sound source position through a sound correlation algorithm, such as music algorithm, GCC-phot algorithm, and the like, which is not limited herein. In some possible embodiments, a wake-up condition may be set, and only the user speaks a specific sentence, for example, when the user speaks "Hi, bear", the robot will wake up, keep on, and analyze the sound source position according to the user voice information.

And S200, controlling the robot to rotate towards the sound source position, and acquiring user image information. Specifically, the robot faces the user through an electric device according to the sound source position to give the user a feeling of communicating face to face. And the robot faces the user, and the image information of the user can be conveniently acquired through the image sensor.

And step S300, judging the identity of the user according to the sound source position. Specifically, based on the sound source information, it is possible to analyze whether the user is in the driver's seat or the passenger's seat or in the rear seat, thereby determining whether the speaking user is the driver or the passenger.

And step S400, when the user identity is a driver, recognizing the emotion and driving state of the user according to the user image and the voice information, and entering a first interaction mode. Details will be described later.

And S500, when the user identity is a passenger, recognizing the emotion of the user according to the user image and the voice information, and entering a second interaction mode. Details will be described later.

Please refer to fig. 2, which is a flowchart illustrating a first embodiment of a first interaction mode according to the present invention.

And step S411, acquiring the facial micro-expression of the user according to the user image information. Specifically, from the user image acquired by the image sensor, facial micro-expressions of the user in the user image, such as an eyebrow direction, a distance between two eyebrows, an eye size, a pupil size, two mouth corner angles, upper and lower lip shapes, and the like, are recognized.

And step S412, judging the emotion of the user according to the facial micro-expression and the voice information of the user. Specifically, the emotion of the user is comprehensively judged according to the change of the facial micro-expression of the user, the tone, the speed of speech and the decibel of sound in the voice information and the semanteme calculated through deep learning. For example, when the facial expression of the user is small in opening and closing of both eyes, tight in closing of eyebrows, small in distance between two eyebrows, and pulling down the mouth corner, the speech rate is not fast, and "qi-word" frequently appears, the user is judged to be unhappy or worried.

Step S413, if the user is in a bad mood, playing a corresponding soothing voice. Specifically, the soothing voice is played through the loudspeaker according to the bad mood of the user. In some possible embodiments, the robot stores the user's data and is provided with a face recognition device, reads the user's data through face recognition, and analyzes an optimal soothing scheme according to the user's data, such as joke, playing happy songs, and playing gasoline and gas.

According to the embodiment, the identity of the user is judged as the driver, the emotion of the driver is analyzed according to the facial micro-expression and the voice information of the driver, and the optimal soothing scheme is obtained according to the emotion of the driver and the preference of the user in the data; the mood of the driver is helped to be pacified, so that the driver is not easy to generate impatient mood which is not beneficial to driving, and dangerous driving is caused.

Please refer to fig. 3, which is a flowchart illustrating a second embodiment of the first interaction mode according to the present invention.

Step S421, according to the user image information, identifying user portrait information, wherein the user portrait information comprises the catch of eyes, the opening degree of eyelids and the head lowering frequency. Specifically, the portrait information of the user is recognized based on the image information of the user acquired by the image sensor, for example, information identifying the size, position, shape, mutual distance, scale size, catch of eye, degree of opening of eyelid, and frequency of lowering head of the user.

And step S422, judging the driving state according to the portrait information of the user. Specifically, the frequency of eye closure, the eye closing time per time, and the head lowering frequency are analyzed to determine whether or not the driver is tired. The image sensor scans the face of the driver at a high speed, for example, when the eye closing frequency of the driver reaches 100 times in 1 minute, the fatigue driving of the driver is judged; for example, if the eye closing time of the driver is more than two seconds within 1 minute for five times, the fatigue driving of the driver is judged; for example, if the driver lowers the head 3 times within 30 seconds, it is determined that the driver is tired.

Step S423, if the user is driving fatigue, playing a corresponding warning voice. Specifically, the voice can be played through the loudspeaker to remind the user of fatigue driving. In some possible embodiments, the robot is connected with an on-board computer, and can control the driver seat to vibrate to remind the driver or control the automobile horn to give out a warning sound. In a specific situation, for example, fatigue driving of the driver is monitored in a short time, or the fatigue degree of the driver is high, wherein the fatigue degree comprises more than 5 seconds per eye closing time, the robot can control the vehicle to decelerate and stop at the roadside.

The embodiment analyzes the driving state of the driver by judging the identity of the user as the driver according to the facial micro-expression of the driver, and assists the driver in driving safely by effective reminding measures when the driver is in fatigue driving.

Please refer to fig. 4, which is a flowchart illustrating a third embodiment of the first interaction mode according to the present invention.

In step S431, road information is acquired, and objects around the vehicle are identified. Specifically, the image sensor may be a plurality of cameras, and when the user is recognized as the driver, the camera that captures the road information is turned on to acquire the road information.

And step S432, playing corresponding reminding voice. Specifically, when the road conditions are relatively complicated, the driver may not be able to consider obstacles around the vehicle, and at this time, the robot is required to assist the driver in identifying the road conditions and reminding the driver.

According to the embodiment, the user identity is judged to be the driver, the obstacles around the vehicle are obtained according to the road information, the driver is reminded, and the driving of the driver is assisted.

Please refer to fig. 5, which is a flowchart illustrating a second interaction mode according to the present invention.

And step S511, acquiring the facial micro-expression of the user according to the user image information. Specifically, when the identity of the user is identified as a passenger, facial micro-expressions of the user in the user image, such as an eyebrow direction, a distance between two eyebrows, an eye size, a pupil size, two mouth corner angles, upper and lower lip shapes, and the like, are identified according to the user image acquired by the image sensor.

And S512, judging the emotion of the user according to the facial micro-expression and the voice information of the user. Specifically, the emotion of the user is comprehensively judged according to the change of the facial micro-expression of the user, the tone, the speed of speech and the decibel of sound in the voice information and the semanteme obtained through deep learning. For example, when the facial expression of the user is that the head is slightly tilted upwards, the eyebrows are raised, the eyes are almost closed, the mouth is opened, the upper teeth and the lower teeth are separated, the lower teeth are visible, and the tone is light, the emotion of the user is judged to be happy.

Step S513, if the user is bad, playing corresponding soothing voice. Specifically, a soothing voice is played through a speaker of the robot according to the bad mood of the user. In some possible embodiments, the robot stores data of a user, and is provided with a face recognition module, reads the data of the user through face recognition, and analyzes an optimal soothing scheme according to the data of the user, for example, the optimal soothing scheme can be a joke, a happy song and a joke of filling oil and gas.

In some embodiments, when the user is a passenger, when it is determined that the passenger is sad, the mood of the passenger can be adjusted in a plurality of ways, for example, playing a video according to the preference of the user, finding a topic of interest to the user and communicating with the user. When the user is a driver, too much interaction of the robot with the driver can distract the driver, possibly leading to dangerous driving.

Please refer to fig. 6, which is a flow chart illustrating a first embodiment of the present invention.

Step S601, receiving the user voice in real time, and acquiring the user voice information. Specifically, the voice information module of the robot receives the voice of the user in real time to acquire the voice information of the user.

Step S602, semantic analysis is carried out according to the user voice information, and the user intention is obtained. The recognition module head analyzes the voice of the user through deep learning to obtain the intention of the user.

Step S603, generating a response to the user intention according to the user intention. Specifically, for example, when the user says "chat well, want to listen to a song" is received, through semantic analysis, it is obtained that the emotion of the user is that the user is chat and needs to listen to music for relaxation, and the music is played immediately; when the fact that the user says 'on duty, waist soreness and back pain' is received, the user is informed of discomfort and needs to be massaged through semantic analysis, the seat massage function is turned on, and the user is pacified through voice.

According to the embodiment, the user intention is obtained through analysis according to the voice information of the user, and the user intention is automatically responded, so that the complicated operation of the user is saved, and the robot is humanized.

Please refer to fig. 7, which is a flow chart illustrating a second embodiment of the present invention.

And step S701, acquiring the temperature in the vehicle and the body temperature of the user. Specifically, the robot acquires the temperature inside the vehicle and the body temperature of the user through the temperature sensor.

And S702, regulating and controlling the temperature in the vehicle according to the temperature in the vehicle and the body temperature of the user. The robot is in communication connection with the vehicle-mounted computer, and automatically opens the air conditioner to adjust the temperature in the vehicle to the optimum temperature according to historical body temperature data and real-time body temperature data of a user and the temperature in the vehicle.

Please refer to fig. 8, which is a schematic diagram of a robot module according to the present invention. The robot 800 is installed in a vehicle, and includes a voice information module 501, a motion control module 502, a recognition module 503, a processing module 504, and an image collection module 505.

A voice information module 501, configured to obtain user voice information and calculate a sound source position;

a motion control module 502 for controlling the robot to rotate towards the sound source position;

the recognition module 503 is configured to recognize the emotion and the driving state of the user according to the image information and the voice information of the user when the identity of the user is a driver, and recognize the emotion of the user according to the image information and the voice information of the user when the identity of the user is a passenger, and enter a second interaction mode;

the processing module 504 is configured to determine the identity of the user, enter a first interaction mode, and enter a second interaction mode.

And an image collecting module 505 for acquiring user image information.

Please refer to fig. 9, which is a schematic diagram of the internal structure of the robot according to the present invention. The robot 800 includes at least one processor 802, memory 801, and a communication component 804. The memory 801 is used for storing vehicle-mounted intelligent robot interaction method program instructions. The processor 802 is configured to execute the in-vehicle intelligent robot interaction method program instructions to implement robot-based interaction between the robot and the user. In some embodiments, the communication component 804 includes, but is not limited to, a mobile network, satellite communication, bluetooth, etc. wireless communication component.

Processor 802 may be, in some embodiments, a Central Processing Unit (CPU), controller, microcontroller, microprocessor or other data Processing chip for executing the robot-based autonomous vehicle emergency braking method stored in memory 801. The memory 801 includes at least one type of readable storage medium including flash memory, hard disks, multi-media cards, card type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disks, optical disks, and the like. The memory 801 may in some embodiments be an internal storage unit of the computer device, such as a hard disk of the computer device. The memory 801 may also be a storage device of an external computer device in other embodiments, such as a plug-in hard disk provided on the computer device, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like. Further, the memory 801 may also include both internal and external storage units of the computer device. The memory 801 may be used for storing various data and application software installed in the computer device, such as: codes or the like based on the robot-implemented autonomous vehicle emergency braking method may also be used to temporarily store data that has been output or is to be output.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product.

The computer program product includes one or more computer instructions. The procedures or functions according to the embodiments of the invention are brought about in whole or in part when the computer program instructions are loaded and executed on a computer. The computer apparatus may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device, such as a server, a data center, etc., that is integrated with one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the unit is only one logical functional division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone, or two or more units may be integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method of the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

It should be noted that the above-mentioned numbers of the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments. And the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, apparatus, article, or method that includes the element.

The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.

Claims

1. The vehicle-mounted intelligent robot interaction method is characterized by comprising the following steps:

acquiring user voice information and calculating a sound source position;

judging the identity of the user according to the sound source position;

2. The vehicle-mounted intelligent robot interaction method according to claim 1, wherein when the user identity is a driver, the user emotion is recognized according to the user image and the voice information, and entering the first interaction mode comprises:

acquiring a user facial micro-expression according to the user image information;

judging the emotion of the user according to the facial micro-expression of the user and the voice information;

and if the user is in bad mood, playing corresponding soothing voice.

3. The vehicle-mounted intelligent robot interaction method according to claim 1, wherein when the user identity is a driver, the driving state is recognized according to the user image and the voice information, and entering the first interaction mode comprises:

identifying user portrait information according to the user image information, wherein the user portrait information comprises eye spirit, eyelid opening degree and head lowering frequency;

judging the driving state according to the user portrait information

And if the user is in fatigue driving, playing corresponding warning voice.

4. The in-vehicle intelligent robot interaction method according to claim 1, wherein when the user identity is a driver, the in-vehicle intelligent robot interaction method further comprises:

acquiring road information and identifying objects around the vehicle;

and playing corresponding reminding voice.

5. The vehicle-mounted intelligent robot interaction method according to claim 1, wherein when the user identity is a passenger, the user emotion is recognized according to the user image and the voice information, and the entering into the second interaction mode comprises:

and if the user is in bad mood, playing corresponding soothing voice.

6. The in-vehicle smart robot interaction method of claim 1, further comprising:

receiving user voice in real time and acquiring user voice information;

performing semantic analysis according to the user voice information to obtain the user intention;

responding to the user intent.

7. The in-vehicle smart robot interaction method of claim 1, further comprising:

acquiring the temperature in the vehicle and the body temperature of a user;

and regulating and controlling the temperature in the vehicle according to the temperature in the vehicle and the body temperature of the user.

8. A robot, characterized in that the robot:

9. A robot, characterized in that the robot comprises a processor and a memory, the memory is used for storing program instructions of the vehicle-mounted intelligent robot interaction method, and the processor is used for executing the program instructions of the vehicle-mounted intelligent robot interaction method so as to realize the vehicle-mounted intelligent robot interaction method according to any one of claims 1-7.

10. A vehicle characterized by comprising a vehicle body and the robot of claim 9 provided to the vehicle body.