WO2024227415A1 - Question answering method and apparatus, and device and storage medium - Google Patents

Abstract

Provided are a question answering method and apparatus, and a device and a storage medium. The question answering method comprises: in response to a question answering initiation operation being detected, using a device of a user to capture image data and a question for the image data (210); extracting text information from the image data (220); acquiring expanded information associated with the text information (230); and determining a target answer for the question on the basis of the image data and the expanded information (240). In this way, in a question answering scenario of multi-modal data, a knowledge base can be introduced to expand the capability of accurately answering a question. Thus, an instant and accurate question answering service can still be provided for a user even if image data is incomplete and insufficient.

Description

Method, device, equipment and storage medium for question answering

This application claims priority to the Chinese invention patent application entitled “Method, device, equipment and storage medium for question and answering” filed on May 4, 2023, with application number: 202310492380.4. The entire contents of this application are incorporated by reference into this application.

Technical Field

Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for question answering.

Background Art

With the rapid development of information technology, more and more applications provide question-answering functions, which brings a lot of convenience to users. Applications with question-answering functions can output corresponding answers based on the voice or text input by the user. Applications with multimodal visual question answering (VQA) functions can also output audio answers to the image input by the user based on voice questions.

Summary of the invention

In a first aspect of the present disclosure, a question-and-answer method is provided. The method includes: in response to detecting a question-and-answer initiation operation, using a user's device to capture image data and a question for the image data; extracting text information from the image data; acquiring extended information associated with the text information; and determining a target answer for the question based on the image data and the extended information.

In a second aspect of the present disclosure, a device for question-answering is provided. The device comprises: a data capture module configured to capture image data and questions about the image data using a user's device in response to detecting a question-answering initiation operation; a text information extraction module configured to extract text information from the image data; an extended information acquisition module configured to acquire extended information associated with the text information; and a target answer determination module configured to determine the target answer based on the image data. Like data and extended information to determine the target answer to the question.

In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the electronic device executes the method of the first aspect.

In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

In a fifth aspect of the present disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and comprises computer executable instructions, which, when executed by a device, cause the device to perform the method of the first aspect.

It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

FIG1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

FIG2 shows a flow chart of a question-and-answer process according to some embodiments of the present disclosure;

FIG3 shows a schematic diagram of a question-and-answer interface according to some embodiments of the present disclosure;

FIG4 is a schematic diagram showing a question-answering process according to some embodiments of the present disclosure;

FIG5 shows a schematic structural block diagram of an apparatus for applying question and answer according to some embodiments of the present disclosure; and

FIG6 shows a block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented.

DETAILED DESCRIPTION

The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Some embodiments of the present disclosure are described, however, it should be understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below. As used herein, the term "model" can represent the association relationship between various data. For example, the above-mentioned association relationship can be obtained based on a variety of technical solutions currently known and/or to be developed in the future.

Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

As an optional but non-limiting embodiment, in response to receiving the user's active request, the prompt information is sent to the user in the form of a pop-up window, and the pop-up window can present the prompt information in the form of text. In addition, the pop-up window can also carry a message for the user to choose "agree" or "disagree" to provide personal information to the electronic device. Click Explore Controls.

It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation to the embodiments of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the embodiments of the present disclosure.

As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes input from the previous layer.

Generally, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also called the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association from input to output (also called the mapping of input to output) from the training data. The parameter values of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. In the application stage, the model can be used to process the actual input based on the parameter values obtained from the training to determine the corresponding output.

As briefly mentioned above, applications with multimodal visual-linguistic question-answering capabilities can According to the image input by the user, an audio answer to the image is played based on the voice question input by the user.

Visual language question answering is a multimodal understanding task that requires answering questions in the form of language after understanding the visual content. Traditionally, applications with multimodal visual language question answering functions can use trained question answering models to implement multimodal visual language question answering.

However, in some cases, the image data collected by the user may have problems such as poor quality and incomplete content. For example, the image data taken by the user is blurry, the question the user wants to ask in the image data is not fully captured, or some text in the image data is small in font. Based on such image data, it may be difficult to generate a correct answer from it. In particular, for some groups of people, such as the visually impaired, there is a greater demand for visual language question and answer, and these people often find it more difficult to detect whether the quality of the captured image data meets the question and answer needs due to their limited vision. Therefore, it is expected that in a question and answer scenario based on visual data, accurate answers can still be obtained even based on poor quality image data.

The embodiments of the present disclosure provide an improved solution for question-answering. According to the solution, image data is collected and questions about the image data are indicated. Extended information associated with text information in the image data is obtained. Answers to the questions are determined based on the image data and the extended information. In this way, a knowledge base can be introduced in a question-answering scenario of multimodal data to expand the ability to accurately answer questions. As a result, instant and accurate question-answering services can be provided to users even when image data is incomplete or insufficient.

Moreover, the question-answering scheme proposed in the present disclosure can effectively assist users, especially those with persistent or temporary visual impairment or obstruction, to accurately implement multimodal visual language question-answering. It should be understood that the scheme provided by the embodiments of the present disclosure can provide convenience for specific groups of people, but this does not imply any discrimination against specific groups of people.

1 shows a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. In the example environment 100, an application 120 is installed in a terminal device 110. A user 140 can interact with the application 120 via the terminal device 110 and/or its attached devices. The application 120 is an application having at least a question-and-answer function.

In some embodiments, the terminal device 110 communicates with the server 130 to provide services for the application 120. The terminal device 110 may be any type of mobile terminal, fixed terminal, or A fixed terminal or portable terminal includes a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio/video player, a digital camera/camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the terminal device 110 can also support any type of interface for the user (such as a "wearable" circuit, etc.).

The terminal device 110 may, for example, include a sensor of an appropriate type for detecting user gestures. For example, the terminal device 110 may include a touch screen for detecting various types of gestures made by the user on the touch screen. Alternatively or additionally, the terminal device 110 may also include other appropriate types of sensing devices such as proximity sensors to detect various types of gestures made by the user within a predetermined distance above the screen. The terminal device 110 may also include, for example, a sound collection device (such as a microphone) for collecting user audio, a sound playback device (such as a speaker) for playing audio, an image collection device (such as a camera, a webcam, etc.) for collecting images, and a display device (such as a display screen, which may be a touch screen) for interface display, etc.

The server 130 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. The server 130 may include, for example, a computing system/server, such as a mainframe, an edge computing node, a computing device in a cloud environment, etc. The server 130 may provide background services for the application 120 in the terminal device 110.

In some embodiments discussed below, the question-answering function can be implemented by multiple models with various functions. One or more of these models can be remotely deployed in the server 130, and the terminal device 110 can use these multiple models to implement the corresponding functions through communication with the server 130. In this way, the resources and power of the terminal device 110 can be saved, and the powerful resources of the server can be used to improve the computing efficiency. In some embodiments, one or more of these models can also be deployed locally on the terminal device 110. This You can choose according to the actual situation.

In some embodiments, in the environment 100 of FIG. 1 , if the application 120 is active, the terminal device 110 may present an interface 150 of the application 120. Through the interface 150, the application 120 may provide the user 140 with one or more services related to the question-and-answer function, including collecting voice, collecting images, playing voice, displaying text, and the like.

It should be understood that the structure and functionality of environment 100 are described for exemplary purposes only and does not imply any limitation on the scope of the present disclosure.

Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

FIG2 shows a flow chart of a question-answering process 200 according to some embodiments of the present disclosure. The process 200 may be implemented at the terminal device 110. For ease of discussion, the process 200 will be described with reference to the environment 100 of FIG1.

In block 210 , the terminal device 110 captures image data and a question regarding the image data in response to detecting a question-and-answer initiation operation.

In some embodiments, the terminal device 110 may capture image data and indicate questions about the image data in response to detecting a question-and-answer initiation operation. Specifically, the terminal device 110 directly detects a question-and-answer initiation operation initiated by a user. For example, the terminal device 110 may determine that a question-and-answer initiation operation is detected in response to detecting a question-and-answer initiation voice (e.g., "turn on the question-and-answer function"). For another example, the terminal device 110 may determine that a question-and-answer initiation operation is detected in response to detecting a preset operation on a hardware button (e.g., a pressing operation, a long pressing operation, etc.). In some embodiments, after the terminal device 110 detects the question-and-answer initiation operation, it runs an application 120 with a question-and-answer function, captures image data, and indicates questions about the image data.

The terminal device 110 may capture image data through an image acquisition device. The image data may be in any form (static image, video clip, etc.), any resolution, and any format (e.g., PNG, JPG, etc.). Alternatively or additionally, the image data may also be data pre-stored in the terminal device 110.

Questions may include multiple forms, that is, the terminal device 110 may capture questions in multiple forms. Questions may include questions captured in text form, for example, and the terminal device 110 may directly obtain the text sequence of the question input by the user (for example, obtaining the text sequence of "How many cups are there in the image?"). The questions may be captured in the form of voice, and the terminal device 110 may capture the voice data of the user's questions through a sound collection device. The voice data may be in any language (e.g., Chinese, English, Japanese, etc.), any time length (e.g., 3s, 5s, etc.), and any timbre. It is understood that the questions may also be captured in any other appropriate form.

In the case where the question is a question captured in the form of voice, in some embodiments, after the terminal device 110 starts the application 120, a question-and-answer interface including at least a recording control may be presented through a display device. The terminal device 110 may determine that a question-and-answer initiation operation is detected in response to detecting a predetermined operation on the recording control, and then capture image data and voice data. The predetermined operation on the recording control may include, for example, a click operation, a slide operation, a long press operation, etc., which are not limited here. In some embodiments, the predetermined operation on the recording control may also be initiated by voice or other instructions.

In some embodiments, during the process of capturing image data and voice data, the terminal device 110 may also stop capturing image data and voice data in response to receiving a capture end operation. Specifically, the terminal device 110 may determine that a capture end operation has been detected in response to detecting a voice such as "stop collecting data". The terminal device 110 may also determine that a capture end operation has been detected in response to detecting a preset operation on a hardware button (such as a press operation, a long press operation, etc.). The terminal device 110 may also determine that a capture end operation has been detected in response to detecting another predetermined operation on a recording control in a recording interface (such as a click operation, a release press operation, etc.).

Referring to FIG. 3 , FIG. 3 shows a schematic diagram of a question-and-answer interface 300 according to some embodiments of the present disclosure. The recording interface 300 may include a control display area 330, and the control display area 330 at least presents a recording control 332. The terminal device 110 may determine that a question-and-answer initiation operation is detected in response to detecting a predetermined operation on the recording control 332. The terminal device 110 may further display text such as "Recording" in the text prompt area 310 to prompt the user that the terminal device 110 is currently in a state of capturing data.

In some embodiments, in response to receiving a predetermined operation, the terminal device 110 changes the presentation effect of the recording control 332 (e.g., changes the color, size, etc. of the recording control 332) to indicate that the terminal device 110 is capturing voice data and image data. Accordingly, the terminal device 110 may stop capturing voice data and image data in response to receiving a capture end operation. In response to receiving the capture end operation, the terminal device 110 may switch the presentation effect of the recording control 332 back to the state before the capture.

In some embodiments, the terminal device 110 may convert the captured voice data into text and display it in the text prompt area 310. As shown in FIG3 , after the terminal device 110 captures the voice data, the text "the dosage of this medicine" corresponding to the voice data is displayed in the text prompt area 310. At the same time, the terminal device 110 may present the image data currently captured by the terminal device 110 in the image display area 320. The conversion of voice to text may be achieved through voice-to-text technology, and the conversion may be performed locally on the terminal device 110 or on a remote server.

In some embodiments, to ensure the accuracy of the question indicated by the determined voice data, the terminal device 110 may pre-process the captured voice data to eliminate noise (such as ambient sound) in the voice data that is not related to the question.

Referring back to FIG. 2 , at block 220 , the terminal device 110 extracts text information from the image data.

For example, the terminal device 110 can use optical character recognition (OCR) technology to extract text information from image data. Optical character recognition refers to the process of analyzing and identifying image data to obtain text information. OCR technology can extract text information from image data by subjecting image data to image processing processes such as image preprocessing, text detection, and text recognition. Specifically, in OCR technology, image preprocessing usually corrects imaging problems of image data. Common preprocessing processes include: geometric transformation (perspective, distortion, rotation, etc.), distortion correction, blur removal, image enhancement, and light correction. Text detection is to detect the location and range of the text and its layout. It usually also includes layout analysis and text line detection. Text recognition is to identify the text content on the basis of text detection and extract text information from image data.

In some embodiments, the terminal device 110 can extract text information from the image data using a trained image processing model based on OCR technology. Specifically, the terminal device 110 can input the image data into the trained image processing model and obtain the text information output by the image processing model. The image processing model can include, for example, a convolutional neural network (CNN), a feedforward neural network (FNN), a fully connected neural network (FCN), a generative adversarial network (GAN), One or more of any suitable models such as a recurrent neural network (RNN).

In some embodiments, the image processing model may be deployed in the server 130, for example, wherein the server 130 may be a remote server (e.g., a cloud server). The terminal device 110 may utilize the trained image processing model to obtain text information through communication with the server 130. Specifically, the terminal device 110 may send the captured image data to the server 130, and the server 130 may output text information based on the image data using the trained image processing model. The terminal device 110 may obtain text information from the server 130. In some embodiments, the trained image processing model may also be deployed locally on the terminal device 110, for example, and the terminal device 110 may directly utilize the trained image processing model deployed locally to extract text information based on the captured image data.

In block 230 , the terminal device 110 obtains extended information associated with the text information.

In the embodiments of the present disclosure, in the scenario where the user wants to ask questions about the image data, it is not only necessary to rely on the image data itself, but also to expand more information to assist in answering the questions.

In some embodiments, the terminal device 110 may always obtain extended information associated with text information in the image data by default. In some embodiments, the terminal device 110 may determine whether the target answer to the question can be determined from the image data. Specifically, the terminal device 110 may perform intent recognition on the question to determine the intent related to the question. Specifically, the terminal device 110 may recognize the intent based on the rule method of the dictionary and the template. Different intents will have different domain dictionaries, such as book titles, song titles, product names, object names, etc. The terminal device 110 may make a judgment based on the degree of match or overlap between the user's intent and the dictionary. The terminal device 110 may also discriminate the user's intent based on a machine learning model. The terminal device 110 may train and learn the already annotated domain corpus through machine learning and deep learning methods to obtain a model for intent recognition (e.g., a model based on fastText). The terminal device 110 then recognizes the intent indicated by the input question based on the model. It can be understood that the terminal device 110 may determine the intent indicated by the question locally, or send the question to the server 130 so that the server 130 determines the intent indicated by the question.

After the terminal device 110 obtains the intention of the question indication, it can determine whether the target of the question can be determined from the image data based on the intention and the image data and/or text information. answer.

If it is determined based on the intent and the image data that the target answer to the question can be determined from the image data, the terminal device 110 can directly determine the corresponding target answer based on the image data and the question without obtaining extended information. For example, if the question captured by the terminal device 110 is voice data corresponding to "how many cups are there", the terminal device 110 can recognize the voice data and determine that the intent corresponding to the question is to determine the number of cups in the image data. The terminal device 110 then determines based on the intent and the captured image data that the target answer to the question can be determined from the image data. The terminal device 110 can directly recognize the image data and determine that the number of cups included in the image data is 3, that is, the answer corresponding to the intent is "3".

For another example, if the question captured by the terminal device 110 is voice data corresponding to "drug dosage", the terminal device 110 can recognize the voice data and determine that the intention corresponding to the question is to determine the dosage corresponding to the drug in the image data. In the case where the text on the drug packaging in the image data is too small, the drug packaging does not contain the dosage, or the text on the drug packaging is blurred, the terminal device 110 can determine that the target answer to the question cannot be determined from the image data based on the intention and the text information extracted from the captured image data. In the case of determining that the target answer to the question cannot be determined from the image data, the terminal device 110 can obtain extended information associated with the text information.

In some embodiments, the terminal device 110 can access the knowledge base and obtain extended information from the knowledge base based on the text information. The knowledge base may include a knowledge graph. Specifically, the terminal device 110 can determine at least one keyword from the text information and obtain extended information associated with at least one keyword from the knowledge base. The terminal device 110 can match the keywords to entities, relationships, attributes, etc. in the knowledge graph to obtain corresponding information as extended information. The knowledge graph is an important data representation form in computer science. In the knowledge graph, nodes represent entities, and the edges between nodes represent the relationships between entities. Both nodes and edges can have their own attributes, which are the attributes of the entity or relationship. It can be understood that the knowledge base can also be any other suitable knowledge base and/or database, etc., which is not limited here.

In order to obtain more accurate extended information, the terminal device 110 pre- Multiple candidate knowledge bases corresponding to multiple fields may be included. The multiple candidate knowledge bases include, but are not limited to, a medical knowledge base corresponding to the medical field, an agronomic knowledge base corresponding to agriculture, a pet knowledge base corresponding to pets, a food knowledge base corresponding to food, and the like. The terminal device 110 may determine the target field corresponding to the image data or text information, and then determine the target knowledge base corresponding to the target field from the multiple candidate knowledge bases. Exemplarily, if the text information of the image data contains the name of a drug, the terminal device 110 may determine that the field corresponding to the text information is the medical field, and then determine the medical knowledge base corresponding to the medical field as the target knowledge base from the multiple candidate knowledge bases.

Furthermore, the terminal device 110 may obtain extended information associated with at least one keyword from the target knowledge base. Similarly, if the target knowledge base is a knowledge graph, the terminal device 110 may match at least one keyword with entities, relationships, attributes, etc. in the knowledge graph to obtain corresponding information as extended information.

In some embodiments, the terminal device 110 may also obtain extended information based on the image data. Specifically, the terminal device 110 may perform image recognition on the image data, and then obtain extended information from a target knowledge base in a field corresponding to the image data based on the result of the image recognition. Exemplarily, if the recognition result of performing image recognition on the image data indicates that the image data contains a cat, the terminal device 110 may determine a pet knowledge base corresponding to the pet as a target knowledge base from multiple candidate knowledge bases, and then obtain extended information associated with the cat from the target knowledge base.

In block 240 , the terminal device 110 determines a target answer to the question based on the image data and the extended information.

In some embodiments, the terminal device 110 can locally use the image data and the extended information to determine the answer to the question. Specifically, the terminal device 110 can, for example, use the trained question-answering model to determine the answer to the image data based on the image data, the extended information and the question. In some embodiments, when the question is a question captured in the form of voice, the terminal device 110 can first convert the voice data corresponding to the question into a text sequence. The terminal device 110 can convert the voice data into a text sequence based on voice technology (e.g., automatic speech recognition (ASR) technology), and then provide the image data, extended information and text sequence to the question-answering model.

In some embodiments, the terminal device 110 may input the text sequence, image data, and extended information corresponding to the question into a trained question-answering model, so that the question-answering model outputs an answer corresponding to the question. In this case, the question-answering model is a multimodal question-answering model, and its input includes data in image modality and text modality. Alternatively or additionally, in some embodiments, the terminal device 110 may also input the text sequence, text information extracted from the image data, and extended information into a trained question-answering model, so that the question-answering model outputs an answer corresponding to the question. In this case, the question-answering model may not be a multimodal question-answering model, and its input is data in text modality.

It is understood that the trained question-answering model can be deployed on the server 130 or locally. If the question-answering model is deployed on the server, the terminal device 110 can use the trained question-answering model to implement the question-answering function through communication with the server 130. If the question-answering model is deployed locally, the terminal device 110 can directly use the trained question-answering model deployed locally to implement the question-answering function.

The question-answering model may be, for example, a language model (LM), which can have question-answering capabilities by learning from a large amount of corpus. The language model may include a statistical language model and a neural network language model, wherein the neural network language model has a more powerful generalization and prediction capability than the statistical language model. In some embodiments, in order to better implement the determination of the question-answering function, the trained question-answering model used is a neural network language model.

Furthermore, since the question-answering capability of the neural network language model can be improved as the amount of data used for training and the amount of model parameters increase, in some embodiments, in order to determine a more accurate answer, the terminal device 110 uses a neural network language model with large-scale parameters, data volume, and computing volume to meet the quality requirements of question-answering in specific applications. When the scale of the language model reaches a certain level (for example, trained by more data), it will have cognitive, common sense, and logical reasoning capabilities that meet the expectations of the application. In some embodiments, in order to ensure the effectiveness of the model, the parameter weights of the model can be determined by pre-training.

In some embodiments, the terminal device 110 may also determine candidate answers to the question from the image data, and perform correction on the candidate answers based on the extended information to obtain the target answer. Correction may include at least one of error correction and completion. Specifically, the terminal device 110 may use the trained question-answering model to determine the target answer based on the image data and the question. Candidate answers to the image data. Similarly, in the case where the question is captured in the form of voice, the terminal device 110 needs to first convert the voice data corresponding to the question into a text sequence, and then input the text sequence and the image data into the trained question-answering model, so that the question-answering model outputs the candidate answer corresponding to the question. Correction (e.g., error correction and/or completion) of the candidate answer based on the image data can be automatically implemented by the question-answering model.

In the case of poor quality of image data, the candidate answers determined directly by the terminal device 110 based on the image data may be wrong. Therefore, the terminal device 110 can further correct the candidate answers based on the extended information to obtain an accurate target answer to the question. The terminal device 110 can correct the candidate answers based on the information corresponding to the question in the extended information to obtain the correct target answer.

For example, when the question is "name and function of the drug", if the image data has an improper exposure problem, the text area on the drug packaging in the image data may be unclear. The candidate answer obtained by the terminal device 110 based on such image data may be incomplete. For example, the candidate answer may be "Huanglian Shangqing Tablets can dispel wind and clear heat." After the terminal device 110 uses the extended information to complete the candidate answer, the target answer can be "Huanglian Shangqing Tablets can dispel wind and clear heat, purge fire and relieve pain." Therefore, the extended information can be used to complete the candidate answer, which helps to improve the completeness of the answer.

For example, when the question is "What is the dosage of this medicine?", if the text on the medicine package in the image data does not contain the corresponding content or the corresponding text is too small, the candidate answer obtained by the terminal device 110 based on such image data may be wrong. For example, the candidate answer may be "24 tablets, 2 plates". After the terminal device 110 uses the extended information to correct the candidate answer, the target answer may be "6 tablets at a time, twice a day". Therefore, the extended information can be used to correct the candidate answer, which helps to improve the correctness of the answer.

Referring to FIG. 4 , FIG. 4 shows a schematic diagram of a process 400 of answering according to some embodiments of the present disclosure. The process 400 may be implemented at the terminal device 110. For ease of discussion, the process 400 will be described with reference to the environment 100 of FIG. 1 . Note that the image data 401, question 402, model input 403, and answer 404 shown in FIG. 4 are merely examples for explanation and do not indicate any limitation.

In some embodiments, after acquiring the image data 401, the terminal device 110 performs text recognition 410 on the image data 401 to extract text information therefrom, for example, extracting text information "XX brand", "Huanglian Shangqing Tablets", "24 tablets·2 plates", "disperse wind and clear heat, purge fire and relieve pain" and "OTC" from the image data 401. The terminal device 110 obtains extended information associated with the text information from the knowledge base 420 (for example, a medical-related knowledge graph). For example, the extended information "Effects and effects: This product has the effects of dispersing wind and clearing heat, purging fire and relieving pain. It can treat acute conjunctivitis, acute suppurative otitis media, toothache, laryngeal paralysis, as well as mouth ulcers and recurrent mouth ulcers" and "Usage and dosage: oral medication, 6 tablets at a time, twice a day. Patients are advised to take the medicine under the guidance of a physician", etc. can be obtained from the knowledge 402.

In some embodiments, as shown in FIG4 , the terminal device 110 may determine the text information, the extended information, and the question 402 together as the model input 403 of the question-answering model. In other examples, the terminal device 110 may also determine the model input 403 of the question-answering model based on the image data 401, the extended information, and the question 402 (for example, which may be in the form of voice). When determining the model input 403, the terminal device 110 may recognize the captured voice data as text and determine the intention of the question.

After determining the model input 403, the terminal device 110 provides the model input 403 to the question-answering model 430. The question-answering model 430 may determine an answer 404 corresponding to the question based on the model input 403. The terminal device 110 may obtain the answer 404 output by the question-answering model. For example, the output answer 404 may be "6 tablets at a time, twice a day".

In some embodiments, after the terminal device 110 determines the answer, the answer can be played in voice form through a sound playing device. As shown in Figure 3, the terminal device 110 can play the answer audio through a speaker. In some embodiments, the answer can be in text form. The terminal device 110 can convert the text into voice through speech synthesis (TTS) for output. In this way, it is convenient for users, especially users with visual impairments, to quickly know the answer.

In some embodiments, alternatively, the terminal device 110 may also present the answer in text form through a display screen. In some embodiments, the terminal device 110 may also output the answer in an additional vibration form and a visual form. The visual form may include, for example, an enlarged image, a highlighted image, etc. Exemplarily, when the voice data input by the user indicates the name of an object in the query image data, the terminal device 110 plays the answer audio containing the name of the object. When the image data is magnified on the display screen, the object can be highlighted.

According to the embodiments of the present disclosure, when the captured image quality is poor and the question and answer cannot be completed, the solution proposed by the present disclosure can significantly improve the efficiency and accuracy of the question and answer, compared with requiring the user to re-collect images that meet the requirements. In this way, in the question and answer scenario of multimodal data, a knowledge base can be introduced to expand the ability to accurately answer questions. As a result, it is possible to provide users with instant and accurate question and answer services even when the image data is incomplete or insufficient.

Fig. 5 shows a schematic structural block diagram of a question-answering apparatus 500 according to some embodiments of the present disclosure. The apparatus 500 may be implemented in or included in the terminal device 110. Each module/component in the apparatus 500 may be implemented by hardware, software, firmware or any combination thereof.

As shown in the figure, the device 500 includes a data capture module 510, which is configured to capture image data and questions for the image data using the user's device in response to detecting a question-and-answer initiation operation. The device 500 also includes a text information extraction module 520, which is configured to extract text information from the image data. The device 500 also includes an extended information acquisition module 530, which is configured to acquire extended information associated with the text information. The device 500 also includes a target answer determination module 540, which is configured to determine a target answer for the question based on the image data and the extended information.

In some embodiments, the extended information acquisition module 530 includes: a keyword determination module configured to determine at least one keyword from text information; and an extended information acquisition module configured to acquire extended information associated with the at least one keyword from a knowledge base.

In some embodiments, the knowledge base includes multiple candidate knowledge bases corresponding to multiple fields, and the extended information acquisition module includes: a target field determination module, configured to determine the target field corresponding to the image data or text information; a target knowledge base determination module, configured to determine the target knowledge base corresponding to the target field from multiple candidate knowledge bases; and an information acquisition module, configured to acquire extended information associated with at least one keyword from the target knowledge base.

In some embodiments, the knowledge base includes a knowledge graph.

In some embodiments, the extended information acquisition module 530 includes: a determination module configured to determine whether the target answer to the question can be determined from the image data; and an acquisition module configured to acquire the target answer to the question if the target answer to the question cannot be determined from the image data. Extended information.

In some embodiments, the target answer determination module 540 includes: a candidate answer determination module, configured to determine candidate answers to the question from image data; and a target answer acquisition module, configured to perform correction on the candidate answers based on extended information to obtain the target answer, the correction including at least one of error correction and completion.

In some embodiments, the target answer is determined using a trained question-answering model, the model input of the question-answering model including at least one of image data and text information, extended information, and a question.

In some embodiments, the question comprises a question captured in speech form.

The units included in the device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and/or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units in the device 500 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

FIG6 shows a block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 600 shown in FIG6 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The electronic device 600 shown in FIG6 may be used to implement the electronic device 110 of FIG1 .

As shown in FIG6 , the electronic device 600 is in the form of a general electronic device. The components of the electronic device 600 may include, but are not limited to, one or more processors or processing units 610, a memory 620, a storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. The processing unit 610 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 620. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 600.

Electronic device 600 typically includes a plurality of computer storage media. Such media may be Any accessible media that can be accessed by the electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 630 can be a removable or non-removable medium and can include a machine-readable medium such as a flash drive, a disk, or any other medium that can be used to store information and/or data (e.g., training data for training) and can be accessed within the electronic device 600.

The electronic device 600 may further include additional removable/non-removable, volatile/non-volatile storage media. Although not shown in FIG. 6 , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a “floppy disk”) and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 620 may include a computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

The communication unit 640 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 600 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

The input device 650 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 660 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 600 may also communicate with one or more external devices (not shown) through the communication unit 640 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 600, or communicate with any device that allows the electronic device 600 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input/output (I/O) interface (not shown).

According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

Various aspects of the present disclosure are described herein with reference to the flowcharts and/or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and/or block diagram and the combination of each box in the flowchart and/or block diagram can be implemented by computer-readable program instructions.

These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions/actions specified in one or more boxes in the flowchart and/or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and/or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions/actions specified in one or more boxes in the flowchart and/or block diagram.

Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions/actions specified in one or more boxes in the flowchart and/or block diagram.

The flowcharts and block diagrams in the accompanying drawings illustrate possible architectures, functions, and operations of various implementations of the systems, methods, and computer program products of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a portion of an instruction that contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions labeled in the box may be The blocks in the flowchart and/or the flowchart may also occur in a different order than that indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each block in the block diagram and/or flowchart, and combinations of blocks in the block diagram and/or flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims (19)

A question-and-answer method comprising: In response to detecting the question-and-answer initiation operation, capturing image data and a question for the image data using a device of the user; extracting text information from the image data; Acquiring extended information associated with the text information; and A target answer to the question is determined based on the image data and the extended information. The method according to claim 1, wherein obtaining the extended information comprises: determining at least one keyword from the text information; and The extended information associated with the at least one keyword is obtained from a knowledge base. The method according to claim 2, wherein the knowledge base includes a plurality of candidate knowledge bases corresponding to a plurality of fields, and obtaining the extended information from the knowledge base includes: Determining a target area corresponding to the image data or the text information; Determining a target knowledge base corresponding to the target domain from the plurality of candidate knowledge bases; and The extended information associated with the at least one keyword is obtained from the target knowledge base. The method according to claim 2, wherein the knowledge base comprises a knowledge graph. The method according to claim 1, wherein obtaining the extended information comprises: determining whether a target answer to the question can be determined from the image data; and If the target answer to the question cannot be determined from the image data, the extended information is acquired. The method of claim 1, wherein determining a target answer to the question comprises: determining candidate responses to the question from the image data; and Based on the extended information, correction is performed on the candidate answer to obtain the target The correction includes at least one of error correction and completion. The method according to claim 1, wherein the target answer is determined using a trained question-answering model, the model input of the question-answering model including at least one of the image data and the text information, the extended information, and the question. The method of claim 1, wherein the question comprises a question captured in speech form. A device for question answering, comprising: A data capture module, configured to capture image data and questions regarding the image data using a user's device in response to detecting a question-and-answer initiation operation; A text information extraction module, configured to extract text information from the image data; an extended information acquisition module, configured to acquire extended information associated with the text information; and The target answer determination module is configured to determine a target answer to the question based on the image data and the extended information. The device according to claim 9, wherein the extended information acquisition module comprises: a keyword determination module, configured to determine at least one keyword from the text information; and The extended information acquisition module is configured to acquire the extended information associated with the at least one keyword from a knowledge base. The apparatus according to claim 10, wherein the knowledge base includes a plurality of candidate knowledge bases corresponding to a plurality of fields, and the extended information acquisition module includes: a target domain determination module, configured to determine a target domain corresponding to the image data or the text information; a target knowledge base determining module, configured to determine a target knowledge base corresponding to the target domain from the plurality of candidate knowledge bases; and The information acquisition module is configured to acquire the extended information associated with the at least one keyword from the target knowledge base. The apparatus according to claim 10, wherein the knowledge base comprises a knowledge graph Score. The device according to claim 9, wherein the extended information acquisition module comprises: a determination module configured to determine whether a target answer to the question can be determined from the image data; and The acquisition module is configured to acquire the extended information if the target answer to the question cannot be determined from the image data. The apparatus according to claim 9, wherein the target answer determination module comprises: a candidate answer determination module configured to determine candidate answers to the question from the image data; and The target answer obtaining module is configured to perform correction on the candidate answer based on the extended information to obtain the target answer, wherein the correction includes at least one of error correction and completion. The apparatus according to claim 9, wherein the target answer is determined using a trained question-answering model, a model input of the question-answering model comprising at least one of the image data and the text information, the extended information, and the question. The apparatus of claim 9, wherein the question comprises a question captured in speech form. An electronic device, comprising: at least one processing unit; and At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 8 when executed by the at least one processing unit. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 8. A computer program product tangibly stored in The computer storage medium contains computer executable instructions, which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 8.