CN118155630A - Voice interaction method and device based on large language model and intelligent voice equipment - Google Patents

Abstract

The application relates to the technical field of voice processing, and discloses a voice interaction method based on a large language model, which comprises the following steps: under the condition that voice interaction data input by a user are received, converting the voice interaction data into a question text; under the condition that the problem text indicates that the user intends to call the voice instruction book, acquiring voice reply information of the intelligent voice equipment according to the problem text, the built knowledge vector library and the large language model; and controlling the intelligent voice equipment to broadcast voice reply information. With the scheme, the problems of the user can be analyzed by using the problem text, the built knowledge vector library and the large language model; based on understanding the user problem, generating corresponding voice reply information; so that the user can timely acquire the professional knowledge of the intelligent voice equipment by controlling the mode of broadcasting the voice reply information by the intelligent voice equipment. The application also discloses a voice interaction device and intelligent voice equipment based on the large language model.

Description

Voice interaction method, device and intelligent voice equipment based on large language model

Technical Field

The present application relates to the field of speech processing technology, for example, to a speech interaction method, apparatus and intelligent speech device based on a large language model.

Background technique

The development of intelligent voice devices is an important branch of the field of artificial intelligence in recent years. It uses technologies such as voice recognition and natural language processing to convert human voice into text, and responds accordingly by analyzing and understanding the text. At present, intelligent voice devices are equipped with detailed paper or electronic manuals, through which users can learn about key resources of product features and operating guides. In order to facilitate users to check at any time, manufacturers usually provide the storage location or link entrance of the manual on the product packaging or official website. Users can rely on indexing or search functions to quickly find relevant instructions for specific functions to solve problems encountered in actual use. But no matter which way, it is not convenient for users to obtain professional knowledge of the device.

At present, in the relevant technologies, users can convey their intention to obtain device expertise to smart voice devices by inputting voice commands; smart voice devices can understand the user's specific intentions through the natural language processing technology they are equipped with, and provide corresponding feedback.

In the process of implementing the embodiments of the present disclosure, it is found that there are at least the following problems in the related art:

When language processing is performed in this way, the ability to understand and answer user questions of different types and in different expressions is limited, and users are unable to obtain the professional knowledge of the intelligent voice device in a timely manner.

It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field.

Summary of the invention

In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key/critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

The embodiments of the present disclosure provide a voice interaction method, apparatus, and intelligent voice device based on a large language model, which can enable users to obtain the professional knowledge of the intelligent voice device in a timely manner.

In some embodiments, the voice interaction method based on the large language model includes: when receiving voice interaction data input by the user, converting the voice interaction data into question text; when the question text indicates that the user intends to call the voice manual, obtaining voice reply information of the intelligent voice device based on the question text, the constructed knowledge vector library and the large language model; controlling the intelligent voice device to broadcast the voice reply information so that the user can promptly obtain the professional knowledge of the intelligent voice device.

In some embodiments, the voice interaction method based on a large language model includes: calling an instruction manual that matches the identification information of an intelligent voice device as a knowledge document; adding the knowledge document to a knowledge base document and performing document processing; and using word embedding technology to convert the processed document into a vector matrix to obtain a knowledge vector library.

In some embodiments, the large language model-based voice interaction method includes: determining the target English content in the document, and converting the target English content into Chinese content; the target English content includes English content of non-intelligent voice device proper nouns and brand names in the document; and/or, determining a document in a format to be converted in the document, and converting the document in the format to be converted into a document in a target format; the document to be converted is a document in a non-target format in the document.

In some embodiments, the voice interaction method based on a large language model includes: adjusting the document name so that the number of words in the document name meets a first word range; and/or adjusting the first title and the second title in the document so that the number of words in the first title and the second title meets a second word range; wherein the minimum value of the first word range is greater than the maximum value of the second word range.

In some embodiments, the voice interaction method based on a large language model includes: when interface parameters are received, the document is segmented into blocks according to the interface parameters; wherein the interface parameters include block size and/or block method, and the interface parameters are configured by the developer according to the characteristics of the knowledge base document.

In some embodiments, the voice interaction method based on the large language model includes: vectorizing the question text to obtain a user question vector; comparing the user question vector with the established knowledge vector library to retrieve the text segment with the highest vector similarity; using the question text and the text segment as input information of the large language model, and outputting corresponding voice reply information.

In some embodiments, the voice interaction method based on the large language model includes: performing a security review on the voice reply information; when the voice reply information passes the security review, controlling the intelligent voice device to broadcast the voice reply information so that the user can promptly learn the professional knowledge of the intelligent voice device.

In some embodiments, the large language model-based voice interaction method includes: controlling the intelligent voice device to broadcast sensitive question reply information when the voice reply information fails to pass the security review.

In some embodiments, the large language model-based voice interaction device includes: a processor and a memory storing program instructions, and the processor is configured to execute the aforementioned large language model-based voice interaction method when running the program instructions.

In some embodiments, the intelligent voice device includes: an intelligent voice device body; and the aforementioned large language model-based voice interaction device installed in the intelligent voice device body.

The speech interaction method, apparatus and intelligent speech device based on the large language model provided by the embodiments of the present disclosure can achieve the following technical effects:

When the question text indicates the user's intention to call the voice manual, it can use the question text, the established knowledge vector library and the large language model to deeply analyze and understand the user's question; and based on the deep understanding of the user's question, generate corresponding voice reply information; so that by controlling the intelligent voice device to broadcast the voice reply information, the user can timely learn the professional knowledge of the intelligent voice device, thereby improving the user's experience of using the intelligent voice device.

The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

FIG1-1 is a schematic diagram of an application provided by an embodiment of the present disclosure;

FIG1-2 is a schematic diagram of a voice interaction method based on a large language model provided by an embodiment of the present disclosure;

FIG2 is a schematic diagram of a method for building a knowledge vector library provided by an embodiment of the present disclosure;

FIG3 is a schematic diagram of a method for obtaining voice reply information provided by an embodiment of the present disclosure;

FIG4 is a schematic diagram of another voice interaction method based on a large language model provided by an embodiment of the present disclosure;

FIG5 is a schematic diagram of another voice interaction method based on a large language model provided by an embodiment of the present disclosure;

FIG6 is a schematic diagram of a speech interaction device based on a large language model provided in an embodiment of the present disclosure.

Detailed ways

In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

Unless otherwise stated, the term "plurality" means two or more.

In the embodiment of the present disclosure, the character "/" indicates that the preceding and following objects are in an "or" relationship. For example, A/B indicates: A or B.

The term "and/or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and/or B means: A or B, or, A and B.

The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

In the embodiments of the present disclosure, smart home appliances refer to home appliances that are formed by introducing microprocessors, sensor technology, and network communication technology into home appliances. They have the characteristics of intelligent control, intelligent perception, and intelligent application. The operation process of smart home appliances often relies on the application and processing of modern technologies such as the Internet of Things, the Internet, and electronic chips. For example, smart home appliances can be connected to electronic devices to enable users to remotely control and manage the smart home appliances.

In the embodiments of the present disclosure, the terminal device refers to an electronic device with a wireless connection function. The terminal device can communicate with the above-mentioned smart home appliances by connecting to the Internet, or can directly communicate with the above-mentioned smart home appliances through Bluetooth, Wi-Fi, etc. In some embodiments, the terminal device is, for example, a mobile device, a computer, or a vehicle-mounted device built into a hover car, or any combination thereof. Mobile devices may include, for example, mobile phones, smart home devices, wearable devices, smart mobile devices, virtual reality devices, etc., or any combination thereof, wherein wearable devices include, for example: smart watches, smart bracelets, pedometers, etc.

Figure 1-1 is an application diagram provided by an embodiment of the present disclosure; in combination with Figure 1-1, in actual applications, the intelligent voice device can be an air conditioner equipped with a voice module. Specifically, after receiving the voice interaction data input by the user, the entry gateway of the air conditioner can use voice recognition technology to convert the voice interaction data into question text. Furthermore, after the air conditioner obtains the question text, the intention domain judgment technology can be used to accurately identify the user's intention. In the case where the user intends to call the voice manual, the air conditioner can retrieve the question text. The specific retrieval process includes, after vectorizing the question text, comparing the similarity between the question text and the text in the knowledge vector library that has been built, so as to retrieve the text segment with the highest similarity, so as to use the text segment with the highest similarity and the question text as the input of the large language model to obtain the voice reply information generated by the large language model. Furthermore, the air conditioner can conduct a security review of the voice reply information; if the voice reply information passes the security review, the text is converted into voice using the text-to-speech technology, and the air conditioner is controlled to broadcast the voice reply information to answer the user's questions; if the voice reply information fails the security review, the air conditioner is controlled to generate sensitive question reply information, and the text is converted into voice using the text-to-speech technology, and the air conditioner is controlled to broadcast the sensitive question reply information. Among them, the knowledge vector library can be constructed by obtaining the manual of the air conditioner as the knowledge base document, and then by setting the importance of different documents, document processing, document segmentation, word embedding and other means.

FIG1-2 is a schematic diagram of a voice interaction method based on a large language model provided by an embodiment of the present disclosure. In conjunction with FIG1-2, an embodiment of the present disclosure provides a voice interaction method based on a large language model, including:

S11, when receiving the voice interaction data input by the user, the intelligent voice device converts the voice interaction data into question text.

S12, when the question text indicates that the user intends to call the voice manual, the intelligent voice device obtains the voice reply information of the intelligent voice device according to the question text, the constructed knowledge vector library and the large language model.

S13, the intelligent voice device controls its broadcast voice reply information so that the user can learn the professional knowledge of the intelligent voice device in time.

In this solution, an intelligent voice device refers to an intelligent home appliance device with a voice module. As an example, the intelligent voice device can be an intelligent air conditioner. Specifically, after the entry gateway of the intelligent voice device receives the voice interaction data input by the user, the intelligent voice device can use ASR technology (Automatic Speech Recognition) to convert the voice interaction data into question text. In this process, ASR will recognize syllables, words, and phrases in the voice and convert them into corresponding text forms. In this solution, automatic speech recognition technology is used to convert voice interaction data to quickly obtain more accurate question text.

Optionally, after the intelligent voice device obtains the question text, it can determine the user's intention represented by the question text. In one example, the intention domain judgment technology can be used to accurately identify the user's intention; once the intention domain judgment technology is used to determine that the user's intention is to call the voice manual, the intelligent voice device will start the voice manual function.

In this embodiment, a large language model of 13B can be used as the core base model to provide powerful and efficient natural language processing capabilities by selecting a high-performance large language model; further, the base model and the corresponding training, storage and application platform can be deployed on the intelligent voice device. Among them, the application platform can support account authentication operation management for various generated applications to ensure the security and privacy of user data; at the same time, the application platform also supports dynamic entry of the knowledge base, so that new knowledge and information can be updated to the intelligent voice device in real time. This design enables users to flexibly customize and adjust the functions of the application according to actual needs to achieve personalized services.

As a method, the above solution can be used to construct the voice manual function, and the method of using the voice manual function is consistent with the method of using other functions of the intelligent voice device. In this way, users can obtain detailed instructions for using the intelligent voice device through voice, which improves the convenience of use and user experience.

In an optimized solution, in order to ensure that the voice instructions function can be used first, the priority of intent understanding can be configured separately on the voice skill platform. In this way, when the user issues a voice instruction related to the voice instructions, the intelligent voice device will recognize and trigger this function first, ensuring that the user can quickly obtain the required information.

Specifically, the intelligent voice device obtains the voice reply information of the intelligent voice device according to the question text, the built knowledge vector library and the large language model, including: the intelligent voice device vectorizes the question text to obtain the user question vector. The intelligent voice device compares the user question vector with the built knowledge vector library to retrieve the text segment with the highest vector similarity. The intelligent voice device uses the question text and the text segment as input information of the large language model and outputs the corresponding voice reply information. With this solution, after determining to start the voice manual function, the voice reply information can be accurately obtained.

Furthermore, after receiving the voice reply information, the intelligent voice device can be controlled to broadcast the voice reply information so that the user can learn the professional knowledge of the intelligent voice device in a timely manner.

By adopting the voice interaction method based on the large language model provided in the embodiment of the present disclosure, when the question text indicates the user's intention to call the voice manual, the question text, the constructed knowledge vector library and the large language model can be used to deeply analyze and understand the user's question; and based on the deep understanding of the user's question, corresponding voice reply information is generated; so that by controlling the intelligent voice device to broadcast the voice reply information, the user can timely learn the professional knowledge of the intelligent voice device, thereby improving the user's experience of using the intelligent voice device.

FIG2 is a schematic diagram of a method for building a knowledge vector library provided by an embodiment of the present disclosure; in conjunction with FIG2 , optionally, the knowledge vector library is built in the following manner:

S21, the intelligent voice device calls the instruction manual that matches the identification information of the intelligent voice device as a knowledge document.

S22, the intelligent voice device adds the knowledge document to the knowledge base document and performs document processing.

S23, the intelligent voice device uses word embedding technology to convert the processed document into a vector matrix to obtain a knowledge vector library.

In this solution, the identification information of the intelligent voice device includes the model information of the intelligent voice device. Specifically, the intelligent voice device can call the instruction manual that matches the model information of the intelligent voice device from the server. Here, the server is a cloud server, and the matching relationship between the model information and the instruction manual of different intelligent voice devices can be stored in the server. With this solution, the instruction manual that matches the identification information of the intelligent voice device can be called more accurately. In this way, after the intelligent voice device calls the instruction manual that matches the identification information of the intelligent voice device, it can be used as a knowledge document.

Furthermore, the intelligent voice device adds the knowledge document to the knowledge base document. Here, the knowledge base document also includes various technical principle documents, brand promotion documents and other contents. It should be noted that if there is content in the knowledge base document that cannot be directly processed by natural language processing technology, the organizer will be reminded to manually organize it to convert it into text form. Among them, the content that cannot be directly processed by natural language processing technology includes but is not limited to picture content or table content.

Furthermore, after the intelligent voice device adds the knowledge document to the knowledge base document, the document needs to be processed. Here, the document processing process includes but is not limited to one or more of document language processing, document format processing, and document naming processing. In addition, after the aforementioned processing of the document, the document needs to be cut and divided into blocks. With this solution, subsequent word embedding operations are facilitated.

Furthermore, after the intelligent voice device processes the document, the word embedding technology can be used to convert the processed document into a vector matrix to obtain a knowledge vector library. Specifically, in the word embedding stage, documents such as instructions can be converted into a vector matrix form, that is, converted into a format that can be understood by the computer. As an example, after testing and research, the M3E (Moka Massive Mixed Embedding) model specifically for Chinese can be selected to ensure the accuracy and efficiency of vectorization. With this solution, the knowledge base document will be successfully converted into a knowledge vector library, providing strong data support for subsequent voice instruction manual applications and other related functions.

Optionally, before the intelligent voice device processes the document, the intelligent voice device sets different importance levels for different documents. This solution effectively avoids the situation where multiple different responses to the same question appear, ensuring that the intelligent voice device can select the most appropriate response text according to the importance of the document.

Optionally, document processing includes:

The intelligent voice device determines target English content in the document and converts the target English content into Chinese content; the target English content includes English content of non-intelligent voice device proper nouns and brand names in the document.

and / or,

The intelligent voice device determines a document in a format to be converted in the document, and converts the document in the format to be converted into a document in a target format; the document to be converted is a document in a non-target format in the document.

In this solution, it can be understood that in different languages, the representation methods for mapping words or other text units into continuous vector spaces are not the same. In order to avoid the situation where language mixing may cause garbled characters or affect subsequent semantic analysis, document language processing can be performed. Specifically, the intelligent voice device can determine the target English content in the document. Among them, the target English content includes non-intelligent voice device proper nouns and non-brand name English content in the document. With this solution, accurate determination of the target English content can be achieved. Furthermore, the intelligent voice device can convert the target English content into Chinese content to ensure the language uniformity in the document.

Understandably, it has been tested that documents in docx format perform better than documents in pdf format and md format in terms of processing effect and compatibility. Therefore, documents in docx format can be used as target format documents. In this way, after the intelligent voice device determines the document in the format to be converted in the document, the document in the format to be converted can be converted into the target format document. This solution helps to ensure that subsequent natural language processing tasks can process these documents more efficiently.

Optionally, document processing includes:

The intelligent voice device adjusts the document name so that the number of characters in the document name meets the first character number range.

and / or,

The intelligent voice device adjusts the first title and the second title in the document so that the number of words in the first title and the second title meets the second word number range.

The minimum value of the first word number range is greater than the maximum value of the second word number range.

In this solution, in order to effectively improve the readability and management efficiency of the document, the number of words in the document name and the title name in the document can be adjusted. Specifically, the intelligent voice device can adjust the document name so that the number of words in the document name meets the first word number range. As an example, the value of the first word number range is 9 to 11. With this solution, concise and clear words or phrases can be used to name the document to ensure the simplicity of the document name.

Optionally, avoid using meaningless numbers, symbols or abbreviations when naming the document to ensure the regularity of the document naming.

Optionally, the intelligent voice device tags and weights the paragraphs in the document so as to distinguish the importance and relevance of different paragraphs.

In this solution, in order to effectively improve the readability and management efficiency of the document, the first title and the second title in the document can also be adjusted. Specifically, the intelligent voice device can adjust the first title and the second title so that the number of words in the first title and the second title meets the second word range. As an example, the value of the second word range is 4 to 6. With this solution, the simplicity of the title name can be ensured.

Optionally, upon receiving the interface parameters, the intelligent voice device segments the document according to the interface parameters;

The interface parameters include the block size and/or block mode, and the interface parameters are configured by the developer according to the characteristics of the knowledge base document.

In this solution, it can be understood that the segmentation of the context data of the document is crucial to improving the efficiency of information retrieval and processing. Therefore, developers can flexibly configure the interface parameters according to the characteristics and requirements of the knowledge base documents. That is, developers can design the segmentation size and segmentation method as interface parameters. Thus, when the intelligent voice device receives the interface parameters, the intelligent voice device can cut and segment the document according to the interface parameters. With this solution, through reasonable document segmentation, long documents can be divided into multiple paragraphs or chapters, which facilitates more refined analysis and processing of subsequent natural language processing tasks.

FIG3 is a schematic diagram of a method for obtaining voice reply information provided by an embodiment of the present disclosure; in conjunction with FIG3 , optionally, in S12, the intelligent voice device obtains voice reply information of the intelligent voice device according to the question text, the constructed knowledge vector library and the large language model, including:

S31, the intelligent voice device vectorizes the question text to obtain a user question vector.

S32, the intelligent voice device compares the user question vector with the established knowledge vector library to retrieve the text segment with the highest vector similarity.

S33, the intelligent voice device uses the question text and text segment as input information of the large language model and outputs corresponding voice reply information.

In this solution, the intelligent voice device can vectorize the question text to obtain the user question vector. Specifically, the user's question text can be converted into a vector form through word embedding technology. Among them, word embedding technology is a technology that maps words or phrases from a vocabulary to a vector space. Through this technology, the intelligent voice device can convert text information into a numerical form that can be understood and processed by a computer. This vectorized representation not only retains the semantic information in the text, but also facilitates subsequent vector operations and similarity comparisons.

Furthermore, the intelligent voice device can compare the user question vector with the built knowledge vector library to retrieve the text segment with the highest vector similarity. Specifically, the user question vector can be compared with each text segment vector in the knowledge vector library to retrieve the text segment with the highest similarity. Here, the text segment with the highest vector similarity can be one or more.

Optionally, the comparison may be performed by a variety of means, including but not limited to metadata filtering, graph relationship retrieval, keyword retrieval, etc.

Optionally, metadata filtering includes adding metadata to the data block, such as launch date, R&D department, model, brand, etc. In this way, the text segment related to the user's question can be located more quickly, thereby improving the retrieval efficiency.

Optionally, graph relationship retrieval includes converting entities in knowledge base documents into nodes, establishing relevant connections between two entities, and constructing a graph data structure. In this way, the relationship between knowledge can be understood more accurately, especially for multi-hop problems, graph data indexing can significantly improve the relevance of retrieval.

Optionally, keyword search, as a traditional search method, helps to quickly find text blocks related to user questions and increase search efficiency.

As a preferred solution, the retrieval effect can be further improved by re-ranking, reformulation, hypothetical conversation embedding (HyDE) and sub-queries.

Specifically, the re-ranking technology is to re-rank the search results according to factors such as combined relevance and matching degree, so as to make the results more suitable for the voice manual business scenario.

The reformulation technology allows the large language model to rephrase the question and resubmit it when the retrieval enhancement generation system cannot find relevant context, so as to improve the retrieval success rate.

Hypothetical conversation embedding technology can generate hypothetical responses and perform embedding vector lookup together with user questions, thereby improving the performance of retrieval-augmented generation systems.

Subquery technology is used to decompose complex questions to improve the response effect of large language models.

Furthermore, the intelligent voice device can use the question text and text segment as input information of the large language model and output the corresponding voice reply information. Specifically, the intelligent voice device can input the user question text and the retrieved text segment to the large language model at the same time. The large language model can use its natural language processing capabilities to combine the user question and the information of the relevant text segment to generate a reply to the corresponding question. With this solution, not only can more accurate and comprehensive answers be provided, but also personalized replies can be made according to the actual needs of the user.

FIG4 is a schematic diagram of another voice interaction method based on a large language model provided by an embodiment of the present disclosure; in conjunction with FIG4 , another voice interaction method based on a large language model provided by an embodiment of the present disclosure includes:

S41, when receiving the voice interaction data input by the user, the intelligent voice device converts the voice interaction data into question text.

S42, when the question text indicates that the user intends to call the voice manual, the intelligent voice device obtains the voice reply information of the intelligent voice device according to the question text, the constructed knowledge vector library and the large language model.

S43, the intelligent voice device performs a security review on the voice reply information.

S44, when the voice reply information passes the security review, the intelligent voice device controls it to broadcast the voice reply information so that the user can learn the professional knowledge of the intelligent voice device in time.

In this solution, after receiving the voice reply information from the intelligent voice device, the intelligent voice device needs to conduct a security review of the voice reply information to ensure that the generated reply content is safe, accurate, legal, and meets the requirements of the business scenario.

Specifically, intelligent voice devices conduct security reviews of voice response information, including: whether there are misleading outputs, privacy leaks, copyright issues, publicity and bias, as well as policy and regulatory issues in the voice response information.

Understandably, when generating responses, large language models may produce misleading outputs due to biased understanding of context or data contamination. Although such outputs may seem reasonable, they may actually contain erroneous or misleading information, causing users to make wrong decisions or misunderstandings. Therefore, the responses generated by large language models need to be rigorously reviewed and verified to ensure their accuracy and reliability.

Understandably, during the training of a large language model, if a dataset containing sensitive information is used, the model may learn sensitive information and inadvertently leak it in subsequent interactions. Therefore, when training a large language model, it is necessary to ensure the privacy and security of the dataset and avoid using data containing sensitive information.

Understandably, when a large language model generates answers, it may copy some of the content in the training data, which may cause copyright issues. Therefore, to avoid this, it is necessary to ensure the legality of the copyright of the training data and comply with relevant copyright laws when using large language models.

Understandably, due to the limitations and bias of training data, large language models may output some content that is socially biased and unfair, which may cause users to misunderstand or discriminate against certain groups or viewpoints. Therefore, in order to reduce the impact of bias, it is necessary to optimize the training algorithm and data set of large language models to improve the fairness and objectivity of the model.

Understandably, the output of the large language model may involve sensitive content of laws and regulations, such as political sensitivity, illegal speech, etc. Therefore, in order to avoid violating relevant laws and regulations, the answers generated by the large language model need to be strictly filtered and reviewed to ensure that their content complies with the requirements of laws and regulations.

Furthermore, if the voice reply information passes the security review, the intelligent voice device can control it to broadcast the voice reply information so that the user can learn the professional knowledge of the intelligent voice device in time. Specifically, if the voice reply information passes the security review, the text can be converted into speech using text-to-speech technology (TTS), and the user can be replied to in voice form. This solution makes it easier for users to learn professional knowledge in a timely manner, which helps to improve the user experience of the intelligent voice device.

FIG5 is a schematic diagram of another voice interaction method based on a large language model provided by an embodiment of the present disclosure; in conjunction with FIG5 , another voice interaction method based on a large language model provided by an embodiment of the present disclosure includes:

S51, when receiving the voice interaction data input by the user, the intelligent voice device converts the voice interaction data into question text.

S52, when the question text indicates that the user intends to call the voice manual, the intelligent voice device obtains the voice reply information of the intelligent voice device according to the question text, the constructed knowledge vector library and the large language model.

S53, the intelligent voice device performs a security review on the voice reply information.

S54, when the voice reply information fails to pass the security review, the intelligent voice device controls it to broadcast the sensitive question reply information.

In this solution, after the intelligent voice device conducts a security review of the voice reply information, if the voice reply information fails the security review, the intelligent voice device controls it to broadcast the sensitive question reply information. Here, the sensitive question reply refers to the preset standard answers for sensitive, controversial or high-risk questions that may arise. This solution effectively protects user rights and information security.

Optionally, an embodiment of the present disclosure provides a voice interaction method based on a large language model, comprising:

When receiving the voice interaction data input by the user, the intelligent voice device converts the voice interaction data into question text.

The intelligent voice device calls the exclusive vector library based on the user's identification information.

When the question text indicates that the user intends to call the voice manual, the intelligent voice device obtains the voice reply information of the intelligent voice device based on the question text, the exclusive vector library and the large language model.

The intelligent voice device controls the intelligent voice device to broadcast voice reply information and provide personalized voice services for users.

The voice interaction method based on the large language model provided by the embodiment of the present disclosure can accurately convert voice interaction data into question text, so as to call the exclusive vector library suitable for the user in combination with the user's identification information; thus, when it is recognized that the user intends to call the voice instructions, the large language model is used in combination with the question text and the exclusive vector library to generate an intelligent voice reply. This reply not only accurately matches the user's question, but also fully considers the user's personalized needs, and the reply content is more in line with the user's expectations. At the same time, by controlling the intelligent voice device to broadcast the generated voice reply, the user can obtain the required voice instruction information without performing additional operations, which improves the convenience and efficiency of the interaction and the user's experience.

Optionally, build a dedicated vector library by:

The intelligent voice device obtains the user privacy document, user history document and instruction document of the intelligent voice device required for building a dedicated vector library.

The intelligent voice device adds the user privacy document, the user history document and the instruction document of the intelligent voice device to the knowledge base document and performs document processing.

Intelligent voice devices use word embedding technology to convert processed documents into vector matrices to obtain a proprietary vector library.

With this solution, the knowledge base documents will be successfully converted into a dedicated vector library, which only contains the private information of the corresponding user, providing personalized services without the risk of information leakage. At the same time, it provides strong data support for subsequent voice manual applications and other related functions.

Optionally, the intelligent voice device obtains user history documents, including:

When the user completes the interaction through the voice instruction function of the intelligent voice device, the intelligent voice device determines whether the intelligent voice device receives positive feedback from the user.

When the intelligent voice device receives positive feedback from the user, the intelligent voice device compares the interactive dialogue with the pre-stored documents for similarity.

If the similarity between the interactive dialogue and the pre-stored document is lower than a preset threshold, the intelligent voice device will use the interactive dialogue as a user history document.

In this solution, users can complete interactive operations through the voice instructions function. Specifically, after the user completes the interaction through the voice instructions function of the intelligent voice device, the semantic assistant can consult the user about the user experience through voice or application. In this way, it is possible to judge whether it has received positive feedback from the user based on the user experience of the user's reply. For example, if the user replies with a positive evaluation, it is determined to be positive feedback. With this solution, it is possible to accurately judge the user feedback information in combination with the user's interactive evaluation.

Furthermore, when the intelligent voice device receives positive feedback from the user, the intelligent voice device can compare the similarity between the interactive dialogue and the pre-stored document. Specifically, when the intelligent voice device receives positive feedback from the user, the intelligent voice device can compare the similarity between the interactive dialogue and the pre-stored document, including:

When the intelligent voice device receives positive feedback from the user, the intelligent voice device extracts the question-answer pair of the interaction; the intelligent voice device vectorizes the extracted question-answer pair to obtain the text vector of the question-answer pair; the intelligent voice device compares the text vector of the question-answer pair with the document vector pre-stored in the current exclusive vector library for similarity; wherein the question-answer pair is a round of dialogue or multiple rounds of dialogue. With this solution, it is possible to accurately compare whether the dialogue of the interaction is similar to the document pre-stored in the exclusive vector library.

Furthermore, the intelligent voice device can store a preset threshold value in combination with the comparison requirements. In this way, if the similarity between the interactive dialogue and the pre-stored document is lower than the preset threshold value, the interactive dialogue is used as the user's historical document. With this solution, the user's historical documents can be accurately obtained, so as to improve the personalization of the voice reply information output by the large language model.

Optionally, the intelligent voice device obtains the user's privacy document, including:

The intelligent voice device uses the user portrait document as the user privacy document; or, the intelligent voice device uses the user portrait document and the privacy document uploaded by the user as the user privacy document; wherein, the user portrait document is obtained by matching the user's identification information, the user portrait document includes one or more of the user's income, assets, marital status, living conditions, family composition, age, and region, and the privacy document uploaded by the user includes one or more of the user's sleep-related information and the user's house-related information.

In this solution, the intelligent voice device can call the user portrait document and the privacy document uploaded by the user. Specifically, the user portrait document can be matched in the intelligent voice device through the user's identification information. Here, the user portrait document refers to the content obtained through the user portrait analysis. The user portrait document includes one or more of the user's income, assets, marital status, living conditions, family composition, age, and region, and the user portrait document can also be information related to the aforementioned content. In this way, when the intelligent voice device does not receive the privacy document uploaded by the user, the user portrait document is used as the user privacy document; when the intelligent voice device receives the privacy document uploaded by the user, the user portrait document and the privacy document uploaded by the user are used as the user privacy document. Here, the privacy document uploaded by the user includes one or more of the user's sleep-related information and the user's house-related information. The user's sleep-related information includes the user's sleep time and the user's sleep state. The user's house-related information includes the user's house area and house orientation, etc. With this solution, the user's privacy document can be accurately obtained so as to improve the personalization of the voice reply information output by the large language model.

Optionally, the intelligent voice device obtains a manual document of the intelligent voice device, including:

According to the preset association relationship, the intelligent voice device uses the instruction document associated with the identification information of the intelligent voice device as the instruction document of the intelligent voice device.

In this solution, the intelligent voice device can also obtain the identification information of the intelligent voice device obtained by its entry gateway. Here, the identification information of the intelligent voice device is the model of the intelligent voice device. In this way, the intelligent voice device can use the instruction document associated with the model of the intelligent voice device as the instruction document of the intelligent voice device according to the preset association relationship. With this solution, the accurate acquisition of the instruction document of the intelligent voice device can be achieved.

Optionally, the intelligent voice device obtains voice reply information of the intelligent voice device according to the question text, the exclusive vector library and the large language model, including:

The intelligent voice device vectorizes the question text to obtain the user question vector.

The intelligent voice device compares the user question vector with the exclusive vector library to retrieve the text segment with the highest vector similarity.

The intelligent voice device uses the question text and text segments as input information of the large language model and outputs the corresponding voice reply information.

This solution can not only provide more accurate and comprehensive answers, but also provide personalized responses based on the user's actual needs.

FIG6 is a schematic diagram of a speech interaction device based on a large language model provided in an embodiment of the present disclosure; in combination with FIG6, an embodiment of the present disclosure provides a speech interaction device 200 based on a large language model, including a processor (processor) 201 and a memory (memory) 202. Optionally, the device 200 may also include a communication interface (Communication Interface) 203 and a bus 204. Among them, the processor 201, the communication interface 203, and the memory 202 can communicate with each other through the bus 204. The communication interface 203 can be used for information transmission. The processor 201 can call the logic instructions in the memory 202 to execute the speech interaction method based on the large language model of the above embodiment.

In addition, the logic instructions in the memory 202 described above may be implemented in the form of software functional units and when sold or used as independent products, may be stored in a computer-readable storage medium.

The memory 202 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions/modules corresponding to the method in the embodiment of the present disclosure. The processor 201 executes the function application and data processing by running the program instructions/modules stored in the memory 202, that is, the speech interaction method based on the large language model in the above embodiment is implemented.

The memory 202 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 202 may include a high-speed random access memory and may also include a non-volatile memory.

The embodiments of the present disclosure provide an intelligent voice device, including: an intelligent voice device body, and the above-mentioned voice interaction device 200 based on a large language model. The voice interaction device 200 based on a large language model is installed on the intelligent voice device body. The installation relationship described here is not limited to placement inside the intelligent voice device body, but also includes installation connections with other components of the intelligent voice device, including but not limited to physical connections, electrical connections, or signal transmission connections. It can be understood by those skilled in the art that the voice interaction device 200 based on a large language model can be adapted to a feasible intelligent voice device body, thereby realizing other feasible embodiments.

An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned voice interaction method based on a large language model.

The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, such as: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, individual components and functions are optional, and the order of operations may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and/or" as used in this application refers to any and all possible combinations of one or more associated listings. In addition, when used in this application, the term "comprise" and its variants "comprises" and/or including (comprising) refer to the existence of stated features, wholes, steps, operations, elements, and/or components, but do not exclude the existence or addition of one or more other features, wholes, steps, operations, elements, components and/or groups of these. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same or similar parts between the embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the system, method and computer program product according to the embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in an order different from that disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each box in the block diagram and/or flowchart, and the combination of boxes in the block diagram and/or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

Claims (10)

1. A method of voice interaction based on a large language model, comprising:

Under the condition that voice interaction data input by a user are received, converting the voice interaction data into a question text;

under the condition that the problem text indicates that the user intends to call a voice instruction, obtaining voice reply information of the intelligent voice equipment according to the problem text, the built knowledge vector library and the large language model;

And controlling the intelligent voice equipment to broadcast the voice reply information so that a user can timely acquire the professional knowledge of the intelligent voice equipment.

2. A method according to claim 1, characterized in that the knowledge vector base is built by:

Invoking a specification matched with the identification information of the intelligent voice equipment to serve as a knowledge document;

adding the knowledge document into a knowledge base document, and performing document processing;

And converting the processed document into a vector matrix by using a word embedding technology to obtain a knowledge vector library.

3. The method of claim 2, wherein document processing comprises:

determining target English content in a document, and converting the target English content into Chinese content; the target English content comprises English content of proper nouns and brand names of non-intelligent voice equipment in the document;

And/or the number of the groups of groups,

Determining a to-be-converted format document in the document, and converting the to-be-converted format document into a target format document; the document to be converted is a document in a non-target format in the document.

4. The method of claim 2, wherein document processing comprises:

adjusting the document name so that the word number of the document name meets a first word number range;

And/or the number of the groups of groups,

Adjusting the first title and the second title in the document so that the number of words of the first title and the second title meets a second word number range;

wherein the minimum value of the first word number range is larger than the maximum value of the second word number range.

5. The method according to claim 3 or 4, further comprising:

Under the condition that the interface parameters are received, cutting and blocking the document according to the interface parameters;

the interface parameters comprise block sizes and/or block modes, and the interface parameters are configured by a developer according to the characteristics of the knowledge base document.

6. The method according to any one of claims 1 to 5, wherein obtaining the voice reply information of the intelligent voice device according to the question text, the built knowledge vector library and the large language model comprises:

vectorizing the problem text to obtain a user problem vector;

Comparing the user problem vector with the built knowledge vector library to search out a text segment with highest vector similarity;

And taking the problem text and the text segment as input information of a large language model, and outputting corresponding voice reply information.

7. The method according to any one of claims 1 to 5, wherein controlling the intelligent voice device to report the voice reply message so that the user knows the expertise of the intelligent voice device in time includes:

performing security examination on the voice reply information;

And under the condition that the voice reply information passes the security examination, controlling the intelligent voice equipment to broadcast the voice reply information so as to enable a user to know the professional knowledge of the intelligent voice equipment in time.

8. The method as recited in claim 7, further comprising:

And controlling the intelligent voice equipment to broadcast the sensitive problem reply information under the condition that the voice reply information does not pass the security examination.

9. A large language model based speech interaction device comprising a processor and a memory storing program instructions, wherein the processor is configured to perform the large language model based speech interaction method of any one of claims 1 to 8 when the program instructions are run.

10. An intelligent speech device, comprising:

an intelligent voice device body;

The large language model based voice interaction device according to claim 9, installed on the intelligent voice equipment body.