CN115312061A - Voice question-answer method and device in driving scene and vehicle-mounted terminal - Google Patents

Abstract

The embodiments of the present application disclose a voice question answering method, device and vehicle terminal in a driving scene, which belong to the technical field of human-computer interaction. The method includes: in the case of receiving a voice question and answer instruction, acquiring external environment information, the external environment information is collected by an environmental information acquisition component during the driving of the vehicle, and the external environment information is used to represent the information of the vehicle. based on the external environment information and the voice question and answer command, obtain the question and answer result corresponding to the voice question and answer command, and perform a voice broadcast based on the question and answer result; using the solution provided by this embodiment, the user can When asking questions in the outdoor environment, the in-vehicle terminal can answer according to the environment, which improves the intelligence of the human-vehicle interactive question-and-answer system.

Description

Voice question answering method, device and vehicle-mounted terminal in driving scene

technical field

The present application relates to the technical field of human-computer interaction, in particular to a voice question-and-answer method, device, and vehicle-mounted terminal in a driving scene.

Background technique

With the rapid development of the Internet of Vehicles system, the human-vehicle interaction function in the driving scene is also becoming more and more popular. Among them, voice, as a convenient interaction method, has been widely used in the human-vehicle interaction. Therefore, the question and answer in the driving scene The system is also constantly improving.

In related technologies, the smart car is equipped with a voice interaction system, which can obtain the user's voice in the driving scene to realize device control or voice question and answer. Among them, the voice question and answer function is first converted into text through voice recognition technology, and then finds the relevant information through data query. The text-matched answers are finally returned to the user in the form of screen display or voice broadcast, so as to achieve the purpose of intelligent question answering.

However, the question answering system in the above method is only applicable to the user's questions about the state of the smart car equipment or the answers can be directly searched on the Internet, and can only realize simple vehicle control and navigation functions, and the degree of intelligence is low.

Contents of the invention

Embodiments of the present application provide a voice question answering method, device, and vehicle-mounted terminal in a driving scene. Described technical scheme is as follows:

On the one hand, an embodiment of the present application provides a voice question answering method in a driving scene, the method comprising:

In the case of receiving a voice question and answer instruction, acquire external environment information, the external environment information is collected by the environmental information collection component during the vehicle's driving process, and the external environment information is used to characterize the location of the vehicle external environment;

Acquiring a question and answer result corresponding to the voice question and answer instruction based on the external environment information and the voice question and answer instruction;

Perform voice broadcast based on the question and answer results.

On the other hand, an embodiment of the present application provides a voice question answering device in a driving scene, and the device includes:

An information acquisition module, configured to acquire external environment information when a voice question-and-answer instruction is received, the external environment information is collected by the environment information collection component during vehicle driving, and the external environment information is used to represent the The external environment in which the vehicle is located;

A result acquisition module, configured to acquire a question and answer result corresponding to the voice question and answer instruction based on the external environment information and the voice question and answer instruction;

The voice broadcast module is used to perform voice broadcast based on the question and answer results.

On the other hand, an embodiment of the present application provides a terminal, the terminal includes a processor and a memory; the memory stores at least one instruction, and the at least one instruction is used to be executed by the processor to implement the above aspects Voice question answering method in the described driving scene.

On the other hand, an embodiment of the present application provides a computer-readable storage medium, where at least one piece of program code is stored in the computer-readable storage medium, and the program code is loaded and executed by a processor to implement the above aspects. A Voice Question Answering Method in Driving Scenarios.

On the other hand, an embodiment of the present application provides a computer program product, where the computer program product includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the voice question-and-answer method in the driving scene provided in various optional implementation manners of the above aspects.

In the embodiment of the present application, when a voice question and answer instruction is received, the vehicle-mounted terminal can determine the question and answer result of the voice question and answer instruction based on the external environment information that characterizes the external environment, and broadcast the question and answer result by voice to realize driving. During the process, the intelligent question and answer for the real-time external environment improves the success rate of human-computer interaction and the degree of intelligence in the driving process.

Description of drawings

FIG. 1 is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

Fig. 2 is a block diagram of the main components of the voice question answering system provided by an exemplary embodiment of the present application;

Fig. 3 is a method flowchart of a voice question-and-answer method in a driving scene provided by an exemplary embodiment of the present application;

Fig. 4 is a method flowchart of a voice question-and-answer method in a driving scene provided by another exemplary embodiment of the present application;

Fig. 5 is a method flowchart of a voice question-and-answer method in a driving scene provided by another exemplary embodiment of the present application;

Fig. 6 is a schematic diagram of the process of acquiring an external environment image provided by an exemplary embodiment of the present application;

Fig. 7 is a schematic diagram of the process of obtaining the question and answer text corresponding to the voice question and answer instruction provided by an exemplary embodiment of the present application;

Fig. 8 is a method flowchart of a voice question-and-answer method in a driving scene provided by another exemplary embodiment of the present application;

FIG. 9 is a schematic diagram of a process of determining a first collection period provided by an exemplary embodiment of the present application;

Fig. 10 is a flow chart of obtaining the question and answer results corresponding to voice question and answer instructions provided by an exemplary embodiment of the present application;

FIG. 11 is a schematic diagram of a question-and-answer analysis algorithm provided by an embodiment of the present application;

Fig. 12 is a flowchart of a method for extracting features of external environment information to obtain external environment features provided by an exemplary embodiment of the present application;

Fig. 13 is a schematic diagram of the difference between the observer's angle of view and the shooting angle of view shown in an exemplary embodiment of the present application;

Fig. 14 is a schematic diagram of a voice question answering application scenario provided by an exemplary embodiment of the present application;

Fig. 15 is a schematic diagram of the state transition of the vehicle terminal proposed by an exemplary embodiment of the present application;

Fig. 16 is a structural block diagram of a voice question answering device in a driving scene provided by an exemplary embodiment of the present application;

Fig. 17 is a structural block diagram of a vehicle-mounted terminal provided by an exemplary embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the implementation manners of the present application will be further described in detail below in conjunction with the accompanying drawings.

FIG. 1 is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application. The implementation environment may include: a vehicle 110 , a vehicle-mounted terminal 120 and a server 130 .

The vehicle 110 may be a vehicle, a ship, an aircraft, etc. The following embodiments are described by taking a vehicle as an example, but it is not limited thereto.

An environmental information collection component is provided outside the carrier 110 , and the environmental information collection component may include an image collection component 140 and an audio collection component 150 . Wherein, the image acquisition component is used to acquire external environment images, specifically referring to scenes that can be seen by human eyes, for example, buildings and vehicles in the external environment. The audio collection component is used to collect audio from the external environment, and specifically refers to sounds that can be heard by human ears, for example, the sound of whistles and birdsong in the external environment.

In the embodiment of the present application, the vehicle-mounted terminal 120 is set in the vehicle 110, and the vehicle-mounted terminal 120 can be the vehicle machine 121, or a mobile terminal 122 that establishes a communication connection with the vehicle machine, such as a smart phone, a notebook computer, a wearable For electronic equipment such as equipment, in FIG. 1 , the mobile terminal 122 is taken as an example of a smart phone for illustration. The communication connection between the car machine 121 and the mobile terminal 122 can be established in a wired or wireless manner, for example, a Bluetooth connection, a Universal Serial Bus (Universal Serial Bus, USB), a Wireless Fidelity (Wireless Fidelity, WiFi) connection or a mobile data network connection Etc., this embodiment does not limit it.

The vehicle-mounted terminal 120 is used to process external environment information and user voice commands, specifically extracting target external environment information from the external environment information, converting voice question and answer instructions into corresponding voice question and answer texts, and based on the target external environment information and voice question and answer commands The corresponding voice question-and-answer text generates a question-and-answer result.

In the embodiment of the present application, the vehicle-mounted terminal 120 has the function of performing data communication with the server 130, establishes a connection in a wireless communication manner, and then performs data communication through the connection. The communication connection may be a wireless fidelity connection or a mobile data network connection, etc., which is not limited in this embodiment of the present application.

In the embodiment of the present application, when the vehicle-mounted terminal generates a question-and-answer result based on the voice question-and-answer instruction and external environment information, it may be processed by the local vehicle-mounted terminal or a mobile terminal, or the server 130 may be used to generate the question-and-answer result.

It should be noted that the follow-up steps can only be performed after the voice question-answer program of the vehicle-mounted terminal is awakened. The wake-up instruction is preset. There is no limitation on the way of waking up the voice question answering program.

Schematically, as shown in FIG. 1 , environmental information collection components are provided in the four directions of front, rear, left, and right sides of the vehicle 110, and each environmental information collection component includes an image collection component 140 and an audio collection component 150. The image collection component 140 can It is a vehicle-mounted external perception camera, a driving recorder, etc., and the audio collection component 150 may be a microphone or the like.

The image acquisition component 140 and the audio acquisition component 150 work together, and each of the image acquisition component 140 and the audio acquisition component 150 is connected with the vehicle terminal 120 for acquiring external environment information.

Optionally, an auxiliary imaging component is set in the image acquisition component for auxiliary imaging, and the auxiliary imaging component may be a millimeter radar or an infrared imaging instrument.

Optionally, the image acquisition component and the audio acquisition component pass through Low-Voltage Differential Signaling (Low-Voltage Differential Signaling, LVDS), Composite Video Broadcast Signal (Composite Video Broadcast Signal, CVBS), Controller Area Network (Controller Area Network, CAN), LAN A connection is established with the vehicle-mounted terminal 120 by means of Local Interconnect Network (LIN), etc., and the corresponding vehicle-mounted terminal 120 can read the external environment information acquired by the environment collection component through the connection.

In addition, the vehicle-mounted terminal 120 is also provided with a voice broadcast component and an image display component for broadcasting or displaying the question and answer results corresponding to the voice question and answer instructions.

Schematically, as shown in FIG. 1 , during driving, the image collection component 140 and the audio collection component 150 collect external environment information and cache the collected external environment information in the built-in memory of the vehicle terminal. Passengers in the vehicle see the scene outside the car window, and send a voice question and answer command to a certain scene they see. After receiving the voice question and answer command, the vehicle-mounted terminal extracts the cached external environment information from the built-in memory of the vehicle-mounted terminal, and passes the vehicle-mounted terminal 120 Or the server 130 performs data processing, obtains the result of the voice question and answer, and broadcasts the result of the voice question and answer.

In a schematic example, the main components of the voice question answering system are shown in FIG. The analysis component 240 and the remote information analysis component 230 constitute. The remote information analysis component 230 is mainly a remote server that establishes a communication connection with the vehicle terminal, and the near-field information analysis component 240 is mainly a near-field mobile terminal that establishes a connection with the vehicle terminal.

The direction of the arrow in FIG. 2 indicates the direction of information flow. The environment information collection component 220 collects the external environment of the vehicle in real time, and sends the external environment information data to the vehicle terminal 210 for buffering. After the voice collection component 250 collects the voice question-and-answer command issued by the user, it sends the voice question-and-answer command to the vehicle-mounted terminal 210, and the vehicle-mounted terminal 210 extracts external environment information for a certain period of time from the cached external environment information. Then the vehicle-mounted terminal 210 performs calculation and analysis on the external environment information and voice question-and-answer instructions, or, the vehicle-mounted terminal 210 sends the extracted external environment information and voice question-and-answer instructions to the near-field information analysis component 240 or the remote information analysis component 230 for calculation and analysis, Then, the generated question and answer results are returned to the vehicle-mounted terminal 210, wherein the near-field information analysis component 240 can also perform joint distributed calculations with the remote information analysis component 230 . After receiving the question and answer results, the vehicle-mounted terminal 210 performs certain processing on the results, and then sends the voice question and answer results to the image display component 270 and the voice broadcast component 260 respectively, and the image display component 270 and the voice broadcast component 260 feed back the question and answer results to the user.

Fig. 3 is a method flowchart of a voice question-and-answer method in a driving scene provided by an exemplary embodiment of the present application. In this embodiment, the method is used in the vehicle-mounted terminal shown in Fig. 1 as an example for illustration. The method includes the following steps:

Step 301, when a voice question and answer instruction is received, acquire external environment information.

The external environment information is collected by the environment information collection component during the driving of the vehicle, and the external environment information is used to characterize the external environment where the vehicle is located.

In a possible implementation manner, the external environment information may characterize the external environment of the vehicle from at least two dimensions, and the at least two dimensions may include an image dimension and a sound dimension.

In step 302, based on the external environment information and the voice question and answer instruction, the question and answer result corresponding to the voice question and answer instruction is obtained.

Based on external environment information and voice question-and-answer instructions means that the vehicle-mounted terminal selects the external environment information corresponding to the voice question-and-answer instruction from the cached external environment information data according to the voice question-and-answer instruction proposed by the user, and then responds to the external environment information and voice question-and-answer instruction. The instruction is processed, and the question and answer result corresponding to the voice question and answer instruction is obtained.

Optionally, the question-and-answer results corresponding to the voice question-and-answer instructions can be obtained through the local vehicle-mounted terminal, that is, the vehicle machine or mobile terminal, or the question-and-answer results can be obtained after data processing by the server.

In step 303, voice broadcast is performed based on the question and answer results.

After getting the question and answer results, the vehicle terminal will automatically fill the analysis results into the broadcast template according to the preset language broadcast template. The language broadcast template will contain polite words that are more in line with the human interaction language and some safe driving reminders .

In some embodiments, while the vehicle-mounted terminal obtains the question-and-answer results corresponding to the voice question-and-answer instructions, it also extracts the location information of the current vehicle, and searches for additional information that may exist at the current location in the preset navigation map knowledge map. Such as nearby landmarks, nearby restaurants, gas stations, etc. When the result is broadcast, it can be filled into the broadcast template together with the question and answer results for voice broadcast.

To sum up, in the voice question-and-answer method in the driving scene provided in the embodiment of the present application, the vehicle-mounted terminal acquires external environment information through the received voice question-and-answer instruction, and generates a corresponding question-and-answer according to the acquired external environment information and the voice question-and-answer instruction results and broadcast. It solves the problem that the question answering system cannot interact according to the current driving environment during driving, and achieves the effect that users can ask and answer questions visible in the driving scene.

After the voice question-and-answer program of the vehicle-mounted terminal is awakened, it is necessary to classify the voice commands issued by the user, and the vehicle-mounted terminal implements different functions according to different needs of the user. This application implements subsequent steps.

Fig. 4 is a method flowchart of a voice question answering method in a driving scene provided by another exemplary embodiment of the present application. The method includes the following steps:

Step 401, in the case of receiving a voice command, perform command type recognition on the voice command.

The types of voice commands are divided into two types in terms of function, voice question and answer commands and non-voice question and answer commands. Non-voice question and answer commands include device control commands and navigation commands, among which device control commands are used to adjust the equipment carried inside the vehicle through voice interaction. Such as air conditioners, car TVs, car audio, etc., to adjust the working status of the equipment, and navigation instructions are used to realize voice control of the car navigation system through human-computer interaction.

Voice question-and-answer instructions are question-and-answer instructions, for example, "Will it rain today?", "What's the temperature outside?" and "What is that white car?" are all voice question-and-answer instructions. Subsequent steps in the embodiment of the present application are all performed when the voice question-and-answer instruction is directed at the external environment of the vehicle. For conventional voice question-and-answer instructions, the vehicle-mounted terminal can obtain the answer through a network database search, etc., and then perform voice broadcast, which will not be described here. repeat.

In a possible manner, the speech instruction type recognition can be implemented through an instruction classification model. A pre-trained instruction classification model that is used to compute instruction type probabilities for input speech instructions. The vehicle-mounted terminal inputs the voice text corresponding to the voice command into the command classification model to obtain an output result, which is used to represent the probability of the command type, and the command type whose probability is higher than the threshold is determined as the final command type. The instruction types include device control instructions, navigation instructions, voice question and answer instructions, and the like. The instruction classification model may be trained based on a large number of sample instructions and corresponding instruction labels (indicating instruction types).

In another possible manner, the vehicle-mounted terminal may determine the instruction type corresponding to the voice instruction based on keyword recognition. For example, if the voice command contains keywords such as "air conditioner temperature" and "speaker volume", the voice command is likely to be a device control command. The voice command is likely to be a voice question and answer command.

The embodiment of the present application does not limit the specific manner of classifying voice instructions.

Step 402, if the instruction type of the voice instruction is a question and answer instruction, determine that the voice question and answer instruction is received, and acquire external environment information.

After the vehicle-mounted terminal acquires the voice question-and-answer command, it extracts the external environment information, and performs the subsequent step of analyzing the voice question-and-answer command in this embodiment. In the case that the voice command is a non-voice question-and-answer command, the vehicle-mounted terminal does not execute the subsequent steps in the embodiment of the present application, but executes the program corresponding to the voice command. The non-voice question-and-answer instruction may be a parameter instruction or a navigation instruction for adjusting the vehicle-mounted device. At this time, the vehicle-mounted terminal also executes a corresponding device adjustment program or navigation program. For example, if the user issues an instruction: "increase the temperature of the air conditioner", which is a non-question-and-answer instruction at this time, the subsequent steps will not be executed, and the vehicle-mounted terminal controls the vehicle-mounted air conditioner to increase the temperature.

Step 403, based on the external environment information and the voice question and answer instruction, obtain the question and answer result corresponding to the voice question and answer instruction.

For the implementation manner of this step, reference may be made to the foregoing step 302, and details are not described here in this embodiment.

Step 404, perform voice broadcast based on the question and answer results.

For the implementation manner of this step, reference may be made to the foregoing step 303, and details are not described in this embodiment here.

To sum up, in the real scene, the vehicle-mounted terminal judges the type of instruction issued by the user, determines that the instruction is a voice question-and-answer instruction, and then performs the subsequent steps, which can prevent the vehicle-mounted terminal from processing non-voice question-and-answer instructions in combination with external environmental information. cause waste of processing resources.

In the embodiment of the present application, the vehicle-mounted terminal obtains the question-and-answer results corresponding to the voice question-and-answer instruction based on external environment information and voice instructions, wherein the external environment information includes all information in multiple dimensions or within a period of time, but the voice question-and-answer instruction may It is only proposed for one of the dimensions or a specific time period. If the question and answer results are obtained based on all the information content, it will not only cause unnecessary data processing by the vehicle terminal, but also affect the accuracy of the question and answer results. Therefore, , it is necessary to screen the external environment information first.

In a possible implementation manner, the vehicle-mounted terminal extracts target external environment information from external environment information based on the voice question-and-answer instruction, and the correlation between the target external environment information and the voice question-and-answer instruction is higher than that of other external environment information and the voice question-answer instruction. Correlation.

Wherein, the correlation may include at least one of dimension correlation or time correlation. Correspondingly, the target external environment information may be information of a specific dimension, or information collected in a specific period of time.

Therefore, the target external environment information can be extracted from the external environment information from the two aspects of identifying the problem dimension and judging the specific time period. The two manners of extracting target external environment information will be described respectively below through two exemplary embodiments.

Fig. 5 is a method flowchart of a voice question answering method in a driving scene provided by another exemplary embodiment of the present application. The method includes the following steps:

Step 501, when a voice question and answer instruction is received, acquire external environment information.

The external environment information includes information of multiple dimensions, wherein the multiple dimensions include at least an image dimension and a sound dimension. The image dimension corresponds to the external environment image in the external environment information, and the sound dimension corresponds to the external environment audio in the external environment information.

The process of obtaining external environment information will be described below. As shown in Figure 6, in a possible implementation, firstly, the external image acquisition component of the vehicle shoots the external environment, and then obtains a video image through imaging processing of the captured content, wherein the image acquisition component includes a vehicle-mounted Photographic equipment such as external perception cameras or driving recorders.

Optionally, auxiliary imaging equipment is used for auxiliary imaging processing when acquiring external environmental information, such as millimeter-wave radar or infrared imager, etc., orthodontic processing can be performed on the image frame according to the non-visible light band images collected by the auxiliary imaging equipment, so that the image The location information in is more accurate, and the final question and answer results are also more accurate. For example, when there are multiple vehicles in front of the user, if the user asks a question about one of the vehicles, it is difficult to accurately locate the target vehicle that the user is asking only by relying on the images captured by the camera or driving recorder. After adding auxiliary imaging equipment, it can Further confirm the target vehicle asked by the user according to factors such as distance and orientation. It can meet the user's targeted questions, such as "what is the second car in front?" Such questions.

After the image acquisition component acquires the external environment image, the vehicle terminal caches it for invocation. Since the image information occupies a large storage space, the cache time should not be too long. On the other hand, the user will ask questions according to the real-time environment of the vehicle during the driving process, so the image cache time can be set within two minutes. . When the image cache duration reaches the preset duration, the oldest image frame is deleted, and then the latest image frame is written.

The vehicle-mounted terminal reads the image frames from the camera equipment at a fixed frame rate. After reading the image frames, it uses the image filtering algorithm to quickly eliminate the noise of each frame of the image. If the auxiliary imaging equipment is used, the auxiliary imaging equipment will also be processed The collected non-visible light band images are subjected to orthodontic processing on the image frames.

Optionally, the frame rate is generally set to 20fps, that is, 20 frames of images are read per second, and it can also be adjusted according to different shooting devices and application scenarios, which is not limited in this embodiment of the present application.

The storage method of the external environment audio is similar to the image caching method, and the collected audio also needs to be denoised by means of noise reduction or processed by other algorithms before caching, which will not be repeated here.

Step 502: Perform question dimension recognition on the voice question and answer text corresponding to the voice question and answer instruction to obtain the question dimension corresponding to the voice question and answer text, and the question dimension includes at least one of image dimension and sound dimension.

Among them, the question and answer text corresponding to the voice question and answer command is obtained by the vehicle-mounted terminal by sequentially performing beamforming algorithm, front-end signal processing and ASR (Automatic Speech Recognition, automatic speech recognition) algorithm on the voice question and answer command, as shown in FIG. 7 .

Optionally, the front-end signal processing adopts the ANC (Active Noise Cancellation, active noise cancellation) algorithm to eliminate environmental noise; the AEC (Acoutic Echo Cancellation, acoustic echo cancellation) algorithm is used to eliminate the voice echo broadcast by the vehicle terminal; AGC ( Automatic Gain Control) algorithm, which is used to adjust the amplitude range of the speech signal so that the output signal amplitude after processing is stable.

In some usage scenarios, there may be a situation where the ASR recognition result is empty. At this time, the vehicle-mounted terminal does not perform subsequent steps, waits for a period of time, and returns to the standby state.

After the vehicle-mounted terminal obtains the voice question-and-answer text corresponding to the voice question-and-answer command, it identifies the question dimension. After identifying the question dimension, it extracts the corresponding target external environment information from the external environment information based on the question dimension and the type of external environment information.

In one possible manner, a question classification model is pre-trained, which is used to calculate probabilities representing question dimensions. The vehicle-mounted terminal inputs the voice question-and-answer text corresponding to the voice question-and-answer command into the question classification model to obtain an output result, which is used to represent the probability of the question dimension, and the dimension whose probability is higher than the threshold is determined as the final question dimension. The question classification model can be trained based on a large number of sample questions and corresponding question labels (indicating question dimensions).

In another possible manner, the vehicle-mounted terminal may determine the question dimension (image-related keywords, sound-related keywords) corresponding to the voice question-and-answer instruction based on keyword matching. Words that can characterize the appearance of the scene such as color and shape can be used as keywords related to the image, for example, red, green, spherical, the largest, etc. Sound, onomatopoeia and other sound-characterizing words can be used as sound-related keywords, for example, birdsong, beep, etc.

The embodiment of the present application does not limit the specific problem dimension identification method.

Step 503, if the dimension of the question is the dimension of the image, extract the external environment image from the external environment information as the target external environment information.

Among them, the image dimension means that the user asks questions from the perspective of the image, and the user's description of information such as shape, color, and volume that can be observed by the human eye belongs to the image dimension. For example, "What is that H-shaped building?". Obviously, the voice question and answer command issued by the user is a description of the appearance of the question target, so the question belongs to the image dimension, and the vehicle terminal extracts the external environment image from the external environment information as the target external environment information.

Step 504, if the question dimension is the sound dimension, extract the external environment audio from the external environment information as the target external environment information.

Among them, the sound dimension means that the user asks questions from the perspective of sound, and the user's description of the volume, sound characteristics, and presence or absence of sound that can be captured by the human ear belongs to the sound dimension. For example, "What kind of bird is singing?" Obviously, the object of the voice question and answer instruction is the sound in the external environment. Therefore, the vehicle-mounted terminal extracts the audio from the external environment information as the target external environment information.

In a possible implementation manner, the question dimension includes both the image dimension and the sound dimension. At this time, the external environment image and the external environment audio in the external environment information are simultaneously extracted as the target external environment information. For example, the voice question-and-answer command is "Which car is honking now?" Obviously, the target object of the voice question-and-answer command includes both the sound in the external environment and the image in the external environment. Therefore, at this time, the vehicle-mounted terminal It is necessary to simultaneously extract the external environment image and the external environment audio as the target external environment information.

Step 505, based on the target external environment information and the voice question and answer instruction, obtain the question and answer result corresponding to the voice question and answer instruction.

After the vehicle-mounted terminal extracts the target external environment information, it analyzes and processes the target external environment information and the voice question-and-answer command, and obtains a question-and-answer result corresponding to the voice question-and-answer command.

When the target external environment information is the external environment image, the vehicle-mounted terminal performs data processing on the external environment image and the voice question-and-answer command to obtain the question-and-answer result; and voice question-and-answer instructions for data processing to obtain question-and-answer results; when the target external environment information includes external environment images and external environment audio at the same time, the vehicle-mounted terminal performs data processing on the external environment images, external environment audio, and voice question-and-answer instructions to obtain The question and answer result corresponding to the voice question and answer instruction.

In step 504, the target external environment information has been extracted. In step 505, the range of external environment information to be analyzed by the vehicle-mounted terminal is reduced, thereby reducing the amount of computation when analyzing the external environment information and voice commands.

Step 506, if the external environment information includes the external environment image, determine the associated image frame corresponding to the question and answer result in the external environment image, and display the associated image frame.

In some application scenarios, it is difficult for the user to intuitively understand the question-and-answer results when the vehicle-mounted terminal only broadcasts the question-and-answer results for the voice question-and-answer instructions issued by the user. It is difficult for users to obtain enough information about the question and answer results of such voice question and answer instructions only through voice broadcast. More information is received.

Optionally, based on the question and answer result corresponding to the voice question and answer instruction, determine the image frame where the target object indicated by the question and answer result is located in the video, and then select a frame with the best image quality from several frames before and after the image frame as the associated image frame.

For example, the user issues a voice question-and-answer command "Where does the street sign that just passed point to?", and the vehicle-mounted terminal executes the steps in the embodiment of the application to obtain a question-and-answer result that is "the street sign just now points to the department store, food street and Central Park", and displays it on the vehicle Displaying the image of the road sign collected by the image acquisition component on the screen can enable the user to more intuitively see the various places pointed to by the road sign and the directions they point to, and obtain more information than the voice broadcast.

Step 507, perform voice broadcast based on the question and answer results.

For the implementation manner of this step, reference may be made to the foregoing step 303, and details are not described in this embodiment here.

To sum up, the question-and-answer method in the driving scene provided by this embodiment, by identifying the dimension of the question, and then extracting the target external environment information based on the dimension of the question, enables the vehicle-mounted terminal to extract part of the external environment information in a targeted manner. The data is used as the target external environment information, which reduces the pressure on the vehicle-mounted terminal to process data on the external environment information, and improves the efficiency of the vehicle-mounted terminal to deal with problems.

In addition, the method of displaying associated image frames provided in this embodiment enables users to know the question and answer results not only by listening to the voice broadcast, but also in a visual way, which further ensures the reliability of the question and answer results sex and accuracy.

Fig. 8 is a method flowchart of a voice question answering method in a driving scene provided by another exemplary embodiment of the present application, the method includes the following steps:

Step 801, when a voice question and answer command is received, time keyword recognition is performed on the voice command.

In step 802, if it is identified that the voice question and answer text corresponding to the voice question and answer instruction contains a time keyword, based on the time keyword and the receiving time, a first collection period is determined.

After the vehicle-mounted terminal receives the voice question and answer command, it needs to analyze and process the external environment information and the voice question and answer command. Due to the large amount of image and audio data, the overhead of data processing the external environment information within the entire preset cache time is also very high. big. Therefore, the vehicle-mounted terminal can first perform time keyword recognition on the voice question and answer text corresponding to the voice question and answer command, and then determine a specific time period according to the recognized time keyword, and then perform data analysis on the external environment information within the specific time period Processing greatly reduces the amount of computation and overhead.

When the vehicle-mounted terminal receives a voice question and answer instruction, it performs time keyword recognition on the question and answer text corresponding to the voice question and answer instruction, such as "just now", "five seconds ago", "within a minute", etc. all belong to the time keyword. After identifying the time keyword contained in the voice question-and-answer command, the time obtained by subtracting the time expressed by the time keyword from the moment when the vehicle-mounted terminal receives the voice question-and-answer command is used as the starting time of the first collection period. From time to time to receive this voice question and answer command as the first collection period.

As shown in Figure 9, assuming that _t2 is the moment when this voice question-and-answer command is received, and the time from _t1 to _t2 is the duration expressed by the time keyword, then the time from _t1 to _t2 is set as the first A collection period. For example, at 17:33, the vehicle-mounted terminal receives a voice question-and-answer command "How many convenience stores have we passed in the past minute?", and the vehicle-mounted terminal uses 17:32-17:33 as the first collection period.

Step 803, determining the external environment information whose collection time is within the first collection period as the target external environment information.

After determining the first collection period, the vehicle-mounted terminal extracts the external environment information within the first collection period from the external environment information as the target external environment information.

For example, after determining that the first collection period is 17:32-17:33, the vehicle-mounted terminal extracts the data in the period of 17:32-17:33 from the cached external information as the target external information.

Step 804, in a case where it is recognized that the voice question and answer text corresponding to the voice question and answer instruction does not contain a time keyword, determine a second collection period based on the receiving time.

The embodiment of the present application is applied in a driving scene, and the voice question-and-answer instruction is proposed by the user based on the implementation environment of the vehicle, usually for content within a short period of time. Therefore, in order to reduce the pressure of data processing, the vehicle-mounted terminal may determine a relatively short period of time as the second collection period according to the moment when the voice question-and-answer instruction is received. The second collection period is a short period of time before the moment when the voice question and answer instruction is received. For example, for example, at 17:50:30, the vehicle-mounted terminal receives the voice question and answer command "What is the blue building on the left?" from the user. Therefore, the 10 seconds before the moment when the vehicle-mounted terminal receives the voice question-and-answer command, that is, 17:50:20-17:50:30, can be used as the second collection period.

Step 805, determining the external environment information whose collection time is within the second collection period as the target external environment information.

After the vehicle-mounted terminal determines the second collection period, it extracts the external environment information within the second collection period from the external environment information as the target external environment information.

For example, after determining that the second collection period is 17:50:20-17:50:30, the vehicle-mounted terminal extracts data from the cached external information during the period of 17:50:20-17:50:30 as the target external information.

Step 806, based on the target external environment information and the voice question and answer instruction, obtain the question and answer result corresponding to the voice question and answer instruction.

In a possible implementation scenario, if the vehicle-mounted terminal does not obtain the question-and-answer result corresponding to the voice question-and-answer instruction according to the target external environment information in the first collection period or the second collection period, it is still necessary to All external environmental information is processed to obtain question and answer results.

Step 807, performing voice broadcast based on the question and answer results.

For the implementation manner of this step, reference may be made to the foregoing step 303, and details are not described in this embodiment here.

To sum up, in this embodiment, when the vehicle-mounted terminal recognizes that the voice question-and-answer instruction contains a time keyword, in the case that the voice question-and-answer instruction contains a time keyword, according to the receiving time of the voice question-and-answer instruction and the time keyword, determine If the first collection period does not contain the time keyword, the second collection period is determined. The external environment information of the corresponding collection period is extracted from the external environment information for analysis, which makes the vehicle-mounted terminal more targeted when extracting the target environment information, reduces the pressure of data processing of the vehicle-mounted terminal, and improves the efficiency of obtaining question-and-answer results.

In this embodiment of the present application, based on the external environment information and the voice question-and-answer command, the question-and-answer result corresponding to the voice question-and-answer command is obtained. Therefore, the vehicle-mounted terminal needs to analyze the target external environment information and the voice question-and-answer command. Due to the large amount of external environment information collected during driving, although extracting the target external environment information according to the receiving time reduces the data processing time to a certain extent, in most cases, only the local processor is used for image data processing, and there are still problems. Certain pressure. Therefore, this application provides the following three methods according to different driving scenarios and other factors, all of which can generate question and answer results.

1. The vehicle terminal generates a question and answer result corresponding to the voice question and answer instruction based on the external environment information and the voice question and answer instruction.

2. When the network status meets the transmission conditions, the vehicle-mounted terminal reports the external environment information and the voice question-and-answer command to the server, so that the server can generate a question-and-answer result corresponding to the voice question-and-answer command based on the external environment information and the voice question-and-answer command. Receive the question and answer results sent by the server.

3. When the network status does not meet the transmission conditions, the vehicle-mounted terminal determines the target near-field device from the near-field devices based on the computing power of the device; sends external environment information and voice question and answer instructions to the target near-field device, so that the target near-field device Based on the external environment information and the voice question and answer instruction, generate a question and answer result corresponding to the voice question and answer instruction; receive the question and answer result sent by the target near-field device. Wherein, the near-field device may be a smart phone or a tablet computer in the vehicle.

Among them, the near-field device is a mobile terminal that establishes a communication connection with the vehicle-mounted terminal in the vehicle. The communication connection can be established through Bluetooth, WiFi, etc., and the vehicle-mounted terminal can determine the near-field device through Bluetooth scanning and other methods.

After determining the near-field device, the vehicle-mounted terminal determines the target near-field device from the near-field devices according to the computing power of the device. The computing power of a device refers to the computing power of a device to achieve a specific output by processing data. The computing power can be measured by objective data. The computing power of different devices will be tested in advance through a dedicated test program to obtain the performance of the computing power.

Optionally, sort the computing power performance of different devices that have been tested in advance, and set them to different priorities. The highest priority is 1, which means that the device has the strongest computing power performance, and then store the computing power priority of the device in In the built-in memory of the vehicle-mounted terminal, when it is necessary to determine the target near-field device, the vehicle-mounted terminal selects the device with the relatively highest computing power priority from the near-field devices based on the device computing power priority, and determines it as the target near-field device. For example, under normal circumstances, the computing power of laptops is greater than that of smartphones and smart watches. Set the computing power priority of laptops to 1, and that of smartphones to 2. The computing power priority of the watch is set to 3, and the priority is stored in the built-in memory of the vehicle terminal. When the vehicle terminal needs to determine the target near-field device, if there is a notebook computer in the near-field device, determine the notebook computer. The computer is the target near-field device; otherwise, the device with the highest priority among the remaining near-field devices is determined to be the target near-field device.

Optionally, set network delay and transmission speed thresholds, and when the current network delay is greater than the set threshold, or, when the previous network transmission speed is lower than the set threshold, the vehicle terminal determines that the current network condition does not meet the transmission status. When the current network delay is less than the set threshold and the current network transmission speed is greater than the set threshold, the vehicle-mounted terminal determines that the current network state satisfies the transmission condition.

Of course, other parameters may also be used to determine whether the current network status meets the transmission condition, which is not limited in this embodiment of the present application.

In a possible implementation, when the vehicle-mounted terminal needs to perform data processing through the near-field device or the remote server, it can directly transmit the received voice question-and-answer instruction to the near-field device or the remote server, or send the received The voice question and answer command is converted into the corresponding voice question and answer text and then sent to the near-field device or the remote server. This embodiment does not limit it.

Fig. 10 is a flowchart of a process of obtaining a question and answer result corresponding to a voice question and answer instruction provided by an exemplary embodiment of the present application. Based on the above method 1, this embodiment describes the method steps for obtaining the question and answer result corresponding to the voice question and answer instruction. As shown in FIG. 10 , the method includes the following steps:

Step 1001, perform feature extraction on external environment information to obtain external environment features.

The vehicle-mounted terminal extracts the features in the external environment information. The external image features include color features, texture features, shape features, etc., and the external audio features include loudness, pitch, and timbre. When the vehicle-mounted terminal extracts external image features, it can use algorithm models such as Non-local model or Slow-fast model for feature extraction.

Step 1002, performing feature splicing on the external environment features and the text features of the voice question and answer text corresponding to the voice question and answer instruction to obtain fusion features.

After the vehicle-mounted terminal extracts the features of the external environment information, if the external environment information contains image information, it reduces the dimensionality of the three-dimensional tensor representing the environmental image information into a one-dimensional vector, and then combines the one-dimensional vector representing the image information with the voice question-answer text The corresponding text vectors are concatenated to obtain fusion features.

Step 1003, input the fusion feature into the question answering model, and obtain the question answering result output by the question answering model.

Among them, the input of the question answering model is the fusion feature vector of the external environment information and the voice question answering command, and the output is the question answering result corresponding to the voice question answering command. The question answering model can use algorithm models such as convolutional neural network, recurrent neural network or Transformer model.

Here, the above steps are described by taking the external environment information in the image dimension as an example. In this embodiment, the vehicle-mounted terminal implements the above steps by using a question-and-answer analysis algorithm, as shown in FIG. 11 . FIG. 11 is a schematic diagram of a question and answer analysis algorithm provided by an embodiment of the present application.

Among them, the voice question and answer text corresponding to the target external environment and the voice question and answer instruction is used as the input of the question and answer analysis algorithm, and the question and answer result is used as the output of the question and answer analysis algorithm.

First, use the Slow-fast model 1102 to extract feature information in the target external environment information 1101 to obtain external environment features. Among them, the fast branch network 11021 has a small computational cost and is used to analyze the dynamic change information in the video sequence, while the slow branch network 11022 has a large computational cost and a slightly larger number of parameters and is used to analyze information such as color, texture, and illumination changes in the video sequence.

After the feature information is extracted by the fast and slow branch network respectively, it is fused through the feature fusion network 11023 to obtain a three-dimensional tensor representing image information, and then through a dimensionality reduction network 1104 to generate a one-dimensional vector 1106 representing image information.

Simultaneously, the voice question and answer text 1103 corresponding to the voice question and answer instruction generates the text vector corresponding to the voice question and answer instruction through the text vector generation 1105, and then performs word segmentation, and then queries the word vector of each word wherein, and weights and averages the word vectors of these words, Obtain the one-dimensional text vector 1107 of the voice question and answer text corresponding to the voice question and answer instruction.

Finally, the one-dimensional vector representing the image information and the one-dimensional text vector of the voice question-answer text corresponding to the voice question-answer instruction are subjected to feature concatenation to obtain a fusion feature vector 1108 . Input the fused feature vector into the Transformer model 1109 to generate a question answering result.

When the sound dimension is used as external environment information to obtain the Q&A results, it is also necessary to extract the features of the external environment audio, and the text features of the voiced Q&A text corresponding to the voiced Q&A instructions are spliced to obtain a fusion vector, which is then input into the Q&A system model. This embodiment will not be described in detail here.

In this embodiment, the vehicle-mounted terminal extracts features from the external environment information, and then obtains the question-and-answer results through feature fusion. In the scene, the function of asking questions based on the perception of the vehicle environment and obtaining accurate question and answer results is more intelligent.

In addition, in practical applications, the user who triggers the voice question and answer command is not limited to the driver, but may be a user sitting in another seat. At this time, the view angle of the observer is different from the shooting angle of the external environment image, so the embodiment of the present application provides Another way to get the characteristics of the external environment.

Fig. 12 is a flowchart of a process of extracting features of external environment information to obtain external environment features provided by an exemplary embodiment of the present application. The method may include the steps of:

Step 1201: Determine the viewing angle, which is the viewing angle of the observer who triggers the voice question-and-answer instruction.

The observer's perspective refers to the perspective of the observer who issues the voice question and answer command when observing the external environment.

Optionally, the viewing angle has a certain relationship with the observer's height, age, and sitting position. Therefore, when the vehicle-mounted terminal determines the observer's viewing angle, it uses the sound source localization technology to determine the location of the user who issued the voice question and answer instruction. The spatial position includes but is not limited to the sitting position and the height of the sound, and then the observer's perspective can be reasonably inferred.

In a possible implementation, the vehicle-mounted terminal is equipped with a sound source localization device, and the vehicle-mounted terminal locates the location of the user who triggers the voice question and answer command based on the sound source localization device, and then processes the external environment image according to the observer's perspective, Generate more accurate question answering results.

Step 1202 , based on the viewing angle and the shooting angle of the external environment image, image affine transformation is performed on the external environment image to obtain a transformed external environment image.

The shooting angle of view and the observation angle of view cannot be consistent to a large extent. FIG. 13 is a schematic diagram of the difference between the observer's angle of view and the shooting angle of view at a certain moment in an application scenario.

In FIG. 13 , 1301 is a driving vehicle, and 1302 is a building passing by the vehicle during driving, which can be captured by the vehicle external camera 1303 and the user 1304 in the vehicle at the same time. It can be seen from the figure that at this moment, the shooting angle of the vehicle's external camera is different from the observer's angle of view. For the same thing, since the image observed by the observer is different from the image captured by the shooting device, when the user asks a question about the building, the question and answer result obtained may not correspond to the voice question and answer command.

Affine transformation refers to the process of performing a linear transformation and a translation in a vector space to transform into another vector space. Affine transformation changes include scaling, translation, rotation, reflection, and miscutting. The straight line of the original image is still a straight line after affine transformation, and the parallel lines of the original image are still parallel lines after affine transformation. This is affine.

The on-vehicle terminal performs affine transformation on the external environment image, the purpose is to transform the image captured by the external shooting device into an image that is more in line with the viewer's perspective through image affine transformation, and can make the image features and the features described in the voice question and answer instructions Correspond to each other, and then get more accurate question and answer results.

Step 1203, perform feature extraction on the transformed external environment image to obtain external environment features.

It should be noted that this embodiment provides a manner of extracting features of external environment information to obtain features of the external environment. It can also be used in the embodiment shown in FIG. 10 . As step 1001 in the embodiment shown in FIG. 10 , it has a better implementation effect when the angle of view of the observer is different from the shooting angle of the external environment image.

To sum up, in a real driving environment, the scenery outside the car window may show different states according to the observer's perspective, which makes the description in the user's question and answer instruction deviate from the characteristics of the image captured by the camera. In this embodiment, according to the perspectives of observers at different positions in the vehicle, an affine transformation is performed on the external environment image, so that the transformed image features are more matched with the features in the user's question and answer instruction, and more accurate answers can be obtained.

Fig. 14 is a schematic diagram of a voice question answering application scenario provided by an exemplary embodiment of the present application.

In Figure 14, both the vehicle-mounted terminal and the vehicle’s external image acquisition device are turned on, and the user starts the voice question-and-answer program to ask questions about the scene in the current environment where the vehicle is located. The voice question and answer command is analyzed to generate the corresponding question and answer result, and the voice broadcast and image display are performed. In this embodiment, a question-and-answer scene related to scenery is adopted, which does not constitute a limitation to this embodiment.

In a schematic example, during the voice question-and-answer process in the vehicle driving scene, the state transition process of the vehicle terminal is shown in FIG. 15 . The direction of the arrow indicates the direction of the state transition of the vehicle terminal.

Standby state 1501 means that the vehicle-mounted terminal is in this state before the entire voice question-and-answer program starts running. In the standby state 1501, the external information collection component runs continuously to collect the external environment information of the vehicle in real time. At this time, the user does not issue a voice question and answer command.

When the vehicle-mounted terminal is in the voice receiving state 1502, the user issues a voice command, and the voice collection component starts to work, and sends the collected voice command to the vehicle-mounted terminal to judge the type of the voice command.

The information extraction state 1503 refers to the state when the vehicle-mounted terminal extracts the target external environment information from the cached external environment information.

When the vehicle-mounted terminal is in the analysis and calculation state 1504, the vehicle-mounted terminal selects the best computing power device and issues a corresponding data calculation command, and the corresponding device uses the question-and-answer analysis model to perform calculations. When the question-and-answer result corresponding to the voice question-and-answer command is generated, the The result is returned to the vehicle terminal.

The broadcast result state 1505 means that the vehicle terminal makes the image display component and the voice broadcast component perform corresponding output after further processing according to the question and answer result.

In the standby state 1501 , if the voice question-and-answer instruction is not received, the standby state 1501 is maintained, and when the voice question-and-answer instruction is received, the vehicle-mounted terminal transfers to the voice receiving state 1502 .

In the voice receiving state 1502, when the user's voice input is within the truncation waiting time, the vehicle-mounted terminal continues to maintain the voice receiving state 1502; when the voice information ASR is identified as empty, or when the voice command is determined to be a non-voice question-and-answer command, the vehicle-mounted terminal returns Enter the standby state 1501; when the voice command is determined to be a voice question and answer command, the vehicle terminal transfers to the information extraction state 1503.

In the information extraction state 1503 , when the smart vehicle fails to extract the target environment information from the external environment, it returns to the standby state 1501 ; when the vehicle terminal finishes extracting the target external environment information, the vehicle terminal transfers to the analysis and calculation state 1504 .

In the analysis and calculation state 1504, when the question and answer result text corresponding to the generated question and answer result is empty, the vehicle terminal returns to the standby state 1501; when the generated voice question and answer result is not empty, the vehicle terminal transfers to the broadcast result state 1505.

In the broadcast result state 1505, the user issues a new voice command, and the vehicle terminal directly jumps to the voice receiving state 1502, and continues to execute the next voice question-and-answer program.

In any of the above states, when the user manually interrupts the voice question-and-answer program, the vehicle-mounted terminal directly returns to the standby state 1501 .

The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

Please refer to FIG. 16 , which shows a structural block diagram of a voice question answering device in a driving scene provided by an exemplary embodiment of the present application. The device can include:

The information acquisition module 1601 is configured to acquire external environment information when a voice question-and-answer instruction is received, the external environment information is collected by the environmental information collection component during vehicle driving, and the external environment information is used to represent the The external environment in which the vehicle is located;

A result obtaining module 1602, configured to obtain a question and answer result corresponding to the voice question and answer instruction based on the external environment information and the voice question and answer instruction;

Voice broadcast module 1603, configured to perform voice broadcast based on the question and answer results.

Optionally, the result obtaining module 1602 is configured to:

Extract target external environment information from the external environment information based on the voice question and answer instruction, the correlation between the target external environment information and the voice question and answer instruction is higher than that of other external environment information and the voice question and answer instruction ; Obtaining a question and answer result corresponding to the voice question and answer instruction based on the target external environment information and the voice question and answer instruction.

Optionally, the result obtaining module 1602 is configured to:

Extract target external environment information from the external environment information based on the voice question and answer instruction, the correlation between the target external environment information and the voice question and answer instruction is higher than that of other external environment information and the voice question and answer instruction ; Based on the target external environment information and the voice question-and-answer instruction, obtain a question-and-answer result corresponding to the voice question-and-answer instruction.

Optionally, the result obtaining module 1602 is configured to:

When the problem dimension is the image dimension, extract an external environment image from the external environment information as the target external environment information; when the problem dimension is the sound dimension, extract the external environment image from the external environment information External environment audio is extracted from the external environment information as the target external environment information.

Optionally, the result obtaining module 1602 is configured to:

The target external environment information is extracted from the external environment information based on the receiving time of the voice question-answer instruction and the collection time of the external environment information.

Optionally, the result obtaining module 1602 is configured to:

When it is recognized that the voice question and answer text corresponding to the voice question and answer instruction contains a time keyword, based on the time keyword and the receiving time, determine a first collection period; set the collection time at the first The external environment information in the collection period is determined as the target external environment information; when it is recognized that the voice question and answer text corresponding to the voice question and answer instruction does not contain a time keyword, determine a second collection period based on the receiving time ; determining the external environment information whose collection time is within the second collection period as the target external environment information.

Optionally, the result obtaining module 1602 is configured to:

generating the question and answer result corresponding to the voice question and answer instruction based on the external environment information and the voice question and answer instruction;

or,

When the network status satisfies the transmission condition, report the external environment information and the voice question and answer instruction to the server, so that the server can generate the voice question and answer instruction corresponding to the external environment information and the voice question and answer instruction the question-and-answer result; receiving the question-and-answer result issued by the server;

or,

When the network state does not meet the transmission conditions, determine the target near-field device from the near-field devices based on the device computing power; send the external environment information and the voice question-and-answer instruction to the target near-field device, so that the The target near-field device generates the question-and-answer result corresponding to the voice question-and-answer instruction based on the external environment information and the voice question-and-answer instruction; and receives the question-and-answer result sent by the target near-field device.

Optionally, the result obtaining module 1602 is configured to:

Performing feature extraction on the external environment information to obtain external environment features; performing feature splicing on the external environment features and the text features of the voice question and answer text corresponding to the voice question and answer instruction to obtain fusion features; inputting the fusion features into the question and answer model to obtain the question answering result output by the question answering model.

Optionally, the result obtaining module 1602 is configured to:

Determining an observation angle of view, the observation angle of view is the angle of view of the observer who triggered the voice question and answer instruction; based on the observation angle of view and the shooting angle of the external environment image, image affine transformation is performed on the external environment image to obtain the transformed The transformed external environment image; performing feature extraction on the transformed external environment image to obtain the external environment feature.

Optionally, the information acquisition module 1601 is configured to:

In the case of receiving a voice instruction, perform instruction type recognition on the voice instruction; in the case of a question-and-answer instruction, determine that the voice question-and-answer instruction is received, and acquire the external environment information .

Optionally, the device also includes:

The image display module is used to identify the type of the voice command when the voice command is received; if the command type of the voice command is a question and answer command, determine that the voice question and answer command is received, and Obtain the external environment information.

To sum up, the voice question and answer device in the driving scene provided in this embodiment can be used to obtain external environment information by receiving voice instructions, and obtain the question and answer results corresponding to the voice question and answer instructions according to the external environment information and voice instructions, and broadcast. It solves the problem that the question answering system cannot interact according to the current driving environment during the driving process, and achieves the effect that the user can ask and answer questions visible to the user in the driving scene, and the degree of intelligence is higher.

Please refer to FIG. 17 , which shows a structural block diagram of a vehicle-mounted terminal provided by an exemplary embodiment of the present application. The terminal 1700 can be realized as the vehicle-mounted terminal in each of the foregoing embodiments. The terminal 1700 may include one or more of the following components: a processor 1710 and a memory 1720 .

Processor 1710 may include one or more processing cores. The processor 1710 uses various interfaces and lines to connect various parts of the entire terminal 1700, and executes the terminal by running or executing instructions, programs, code sets or instruction sets stored in the memory 1720, and calling data stored in the memory 1720. 1700's various functions and processing data. Optionally, the processor 1710 may use at least one of Digital Signal Processing (Digital Signal Processing, DSP), Field Programmable Gate Array (Field-Programmable Gate Array, FPGA), and Programmable Logic Array (Programmable LogicArray, PLA). implemented in the form of hardware. The processor 1710 may integrate one or more of a central processing unit (Central Processing Unit, CPU), an image processor (Graphics Processing Unit, GPU), a neural network processor (Neural-network Processing Unit, NPU) and a modem, etc. combination. Among them, the CPU mainly handles the operating system, user interface and application programs, etc.; the GPU is used to render and draw the content that needs to be displayed on the touch screen; the NPU is used to realize the artificial intelligence (AI) function; the modem is used to process Wireless communication. It can be understood that the foregoing modem may also not be integrated into the processor 1710, but implemented by a single chip.

The memory 1720 may include a random access memory (Random Access Memory, RAM), and may also include a read-only memory (Read-Only Memory, ROM). Optionally, the memory 1720 includes a non-transitory computer-readable storage medium. The memory 1720 may be used to store instructions, programs, codes, sets of codes or sets of instructions. The memory 1720 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing the operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area Data created according to use of the terminal 1700 and the like may be stored.

In addition, those skilled in the art can understand that the structure of the terminal 1700 shown in the above drawings does not constitute a limitation on the terminal, and the terminal may include more or less components than those shown in the figure, or combine certain components , or different component arrangements. For example, the terminal 1700 also includes components such as a display screen, a camera component, a microphone, a speaker, a radio frequency circuit, a sensor, an audio circuit, a WiFi module, a power supply, and a Bluetooth module, which will not be repeated here.

The embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores at least one program code, and the program code is loaded and executed by a processor to realize the driving scenario described in each of the above embodiments. question-and-answer method.

An embodiment of the present application provides a computer program product, where the computer program product includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the question answering method in the driving scene provided in various optional implementation manners of the above aspects.

It should be understood that the "plurality" mentioned herein refers to two or more than two.

In addition, the numbering of the steps described herein only exemplarily shows a possible sequence of execution among the steps. In some other embodiments, the above-mentioned steps may not be executed according to the order of the numbers, such as two different numbers The steps are executed at the same time, or two steps with different numbers are executed in the reverse order as shown in the illustration, which is not limited in this embodiment of the present application.

The above are only optional embodiments of the application, and are not intended to limit the application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the application shall be included in the protection of the application. within range.

Claims (25)

1. A method for voice question answering in a driving scene, the method comprising:

under the condition that a voice question-answer instruction is received, acquiring external environment information, wherein the external environment information is acquired by an environment information acquisition assembly in the driving process of a carrier and is used for representing the external environment of the carrier;

acquiring a question-answer result corresponding to the voice question-answer instruction based on the external environment information and the voice question-answer instruction;

and carrying out voice broadcast based on the question and answer result.

2. The method according to claim 1, wherein the obtaining, based on the external environment information and the voice question-answering instruction, a question-answering result corresponding to the voice question-answering instruction comprises:

extracting target external environment information from the external environment information based on the voice question-answer instruction, wherein the correlation between the target external environment information and the voice question-answer instruction is higher than the correlation between other external environment information and the voice question-answer instruction;

and acquiring a question-answer result corresponding to the voice question-answer instruction based on the target external environment information and the voice question-answer instruction.

3. The method according to claim 2, wherein the extracting target external environment information from the external environment information based on the voice question-answering instruction comprises:

performing question dimension identification on a voice question and answer text corresponding to the voice question and answer instruction to obtain question dimensions corresponding to the voice question and answer text, wherein the question dimensions comprise at least one of image dimensions and sound dimensions;

extracting the target external environment information from the external environment information based on the problem dimension and the type of the external environment information.

4. The method according to claim 3, wherein the extracting the target external environment information from the external environment information based on the problem dimension and a type corresponding to the external environment information comprises:

extracting an external environment image from the external environment information as the target external environment information when the problem dimension is the image dimension;

extracting external environment audio from the external environment information as the target external environment information in a case where the problem dimension is the sound dimension.

5. The method according to claim 2, wherein the extracting target external environment information from the external environment information based on the voice question-answering instruction comprises:

and extracting the target external environment information from the external environment information based on the receiving time of the voice question-answering instruction and the acquisition time of the external environment information.

6. The method according to claim 5, wherein the extracting the target external environment information from the external environment information based on the receiving time of the voice question-answering instruction and the collecting time of the external environment information comprises:

under the condition that the voice question-answer text corresponding to the voice question-answer instruction is identified to contain a time keyword, determining a first acquisition time period based on the time keyword and the receiving moment; determining the external environment information with the acquisition time in the first acquisition time period as the target external environment information;

under the condition that the voice question-answer text corresponding to the voice question-answer instruction does not contain time keywords, determining a second acquisition time period based on the receiving time; and determining the external environment information with the acquisition time in the second acquisition time period as the target external environment information.

7. The method according to claim 1, wherein the obtaining a question-answer result corresponding to the voice question-answer instruction based on the external environment information and the voice question-answer instruction comprises:

generating a question-answer result corresponding to the voice question-answer instruction based on the external environment information and the voice question-answer instruction;

or the like, or a combination thereof,

reporting the external environment information and the voice question-answer instruction to a server under the condition that the network state meets the transmission condition, so that the server generates a question-answer result corresponding to the voice question-answer instruction based on the external environment information and the voice question-answer instruction; receiving the question and answer result sent by the server;

or the like, or, alternatively,

under the condition that the network state does not meet the transmission condition, determining target near-field equipment from the near-field equipment based on equipment calculation force; sending the external environment information and the voice question and answer instruction to the target near field equipment, so that the target near field equipment generates a question and answer result corresponding to the voice question and answer instruction based on the external environment information and the voice question and answer instruction; and receiving the question and answer result sent by the target near field equipment.

8. The method according to claim 7, wherein the generating the question-answer result corresponding to the voice question-answer instruction based on the external environment information and the voice question-answer instruction comprises:

extracting the characteristics of the external environment information to obtain external environment characteristics;

performing feature splicing on the external environment features and text features of the voice question and answer text corresponding to the voice question and answer instruction to obtain fusion features;

and inputting the fusion characteristics into a question-answer model to obtain the question-answer result output by the question-answer model.

9. The method of claim 8, wherein the external environment information comprises an external environment image;

before the feature extraction is performed on the external environment information to obtain the external environment feature, the method further includes:

determining an observation visual angle, wherein the observation visual angle is the visual angle of an observer triggering the voice question-answer instruction;

performing image affine transformation on the external environment image based on the observation angle and the shooting angle of the external environment image to obtain the transformed external environment image;

the extracting the external environment information to obtain the external environment characteristics includes:

and performing feature extraction on the transformed external environment image to obtain the external environment features.

10. The method according to claim 1, wherein the obtaining external environment information in case of receiving a voice question-answer instruction comprises:

under the condition of receiving a voice instruction, performing instruction type recognition on the voice instruction;

and under the condition that the instruction type of the voice instruction is a question-answer instruction, determining that the voice question-answer instruction is received, and acquiring the external environment information.

11. The method of claim 1, further comprising:

determining an associated image frame corresponding to the question answering result in the external environment image under the condition that the external environment information comprises the external environment image;

and displaying the related image frame.

12. An apparatus for voice question answering in a driving scene, the apparatus comprising:

the information acquisition module is used for acquiring external environment information under the condition that a voice question-answer instruction is received, wherein the external environment information is acquired by an environment information acquisition assembly in the running process of the carrier and is used for representing the external environment of the carrier;

the result acquisition module is used for acquiring a question and answer result corresponding to the voice question and answer instruction based on the external environment information and the voice question and answer instruction;

and the voice broadcasting module is used for carrying out voice broadcasting based on the question and answer result.

13. The apparatus of claim 12, wherein the result obtaining module comprises:

an information extraction unit, configured to extract target external environment information from the external environment information based on the voice question-answering instruction, where a correlation between the target external environment information and the voice question-answering instruction is higher than a correlation between other external environment information and the voice question-answering instruction;

and the result acquisition unit is used for acquiring a question and answer result corresponding to the voice question and answer instruction based on the target external environment information and the voice question and answer instruction.

14. The apparatus of claim 13, wherein the information extraction unit is configured to:

performing question dimension identification on a voice question and answer text corresponding to the voice question and answer instruction to obtain question dimensions corresponding to the voice question and answer text, wherein the question dimensions comprise at least one of image dimensions and sound dimensions;

extracting the target external environment information from the external environment information based on the problem dimension and the type of the external environment information.

15. The apparatus of claim 14, wherein the information extracting unit is configured to:

extracting an external environment image from the external environment information as the target external environment information under the condition that the problem dimension is the image dimension;

extracting external environment audio from the external environment information as the target external environment information in a case where the problem dimension is the sound dimension.

16. The apparatus of claim 13, wherein the information extraction unit is configured to:

and extracting the target external environment information from the external environment information based on the receiving time of the voice question-answering instruction and the acquisition time of the external environment information.

17. The apparatus of claim 16, wherein the information extraction unit is configured to:

under the condition that the voice question-answer text corresponding to the voice question-answer instruction is identified to contain a time keyword, determining a first acquisition time period based on the time keyword and the receiving moment; determining the external environment information with the acquisition time in the first acquisition time period as the target external environment information;

under the condition that the voice question-answer text corresponding to the voice question-answer instruction is identified not to contain the time keyword, determining a second acquisition time period based on the receiving moment; and determining the external environment information with the acquisition time in the second acquisition time period as the target external environment information.

18. The apparatus of claim 12, wherein the result obtaining module comprises:

the first processing unit is used for generating a question-answer result corresponding to the voice question-answer instruction based on the external environment information and the voice question-answer instruction;

the second processing unit is used for reporting the external environment information and the voice question-answer instruction to a server under the condition that the network state meets the transmission condition, so that the server can generate the question-answer result corresponding to the voice question-answer instruction based on the external environment information and the voice question-answer instruction; receiving the question and answer result sent by the server;

the third processing unit is used for determining the target near-field equipment from the near-field equipment based on the equipment calculation force under the condition that the network state does not meet the transmission condition; sending the external environment information and the voice question and answer instruction to the target near field equipment, so that the target near field equipment generates a question and answer result corresponding to the voice question and answer instruction based on the external environment information and the voice question and answer instruction; and receiving the question and answer result sent by the target near field equipment.

19. The apparatus of claim 18, wherein the first processing unit is configured to:

extracting the characteristics of the external environment information to obtain external environment characteristics;

performing feature splicing on the external environment features and text features of the voice question-answer text corresponding to the voice question-answer instruction to obtain fusion features;

and inputting the fusion characteristics into a question-answer model to obtain the question-answer result output by the question-answer model.

20. The apparatus of claim 19, wherein the external environment information comprises an external environment image;

the device, still include:

a visual angle determining module, configured to determine an observation visual angle, where the observation visual angle is a visual angle of an observer who triggers the voice question-answering instruction;

the transformation module is used for carrying out image affine transformation on the external environment image based on the observation visual angle and the shooting visual angle of the external environment image to obtain the transformed external environment image;

the first processing unit is configured to perform feature extraction on the transformed external environment image to obtain the external environment feature.

21. The apparatus of claim 12, wherein the information obtaining module is configured to:

under the condition of receiving a voice instruction, performing instruction type recognition on the voice instruction;

and under the condition that the instruction type of the voice instruction is a question-answer instruction, determining that the voice question-answer instruction is received, and acquiring the external environment information.

22. The apparatus of claim 12, further comprising:

the image display module is used for determining an associated image frame corresponding to the question answering result in the external environment image under the condition that the external environment information comprises the external environment image; and displaying the related image frame.

23. A vehicle-mounted terminal is characterized by comprising a processor and a memory; the memory stores at least one instruction for execution by the processor to implement a method for spoken question answering in a driving scenario according to any of claims 1 to 11.

24. A computer-readable storage medium, characterized in that at least one program code is stored in the computer-readable storage medium, which is loaded and executed by a processor to implement the method for question and answer with speech in a driving scenario according to any one of claims 1 to 11.

25. A computer program product, characterized in that the computer program product comprises computer instructions, the computer instructions being stored in a computer readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executing the computer instructions to cause the computer device to perform the method for phonetic question answering in a driving scenario according to any one of claims 1 to 11.

Images