KR102441456B1 - Method and system for mimicking tone and style of real person - Google Patents

Abstract

Disclosed are a dialogue method and system that mimics the tone and style of a real person. The conversation method includes the steps of: generating a dialogue model composed of a tone and style of the target person based on data related to the target person; and providing an answer corresponding to the request by using the conversation model according to the request of the client.

Description

A communication method and system that mimics the tone and style of a real person

The description below relates to conversational technology using artificial intelligence (AI).

Even with the same message, the words of celebrities have a greater ripple effect.

Meanwhile, an increasing number of people have nostalgia for celebrities who have already died.

With the recent development of artificial intelligence, robots have reached a level where they can write similarly to humans.

In addition, artificial intelligence conversation systems for the purpose of preventing loneliness or psychological treatment are being studied.

For example, in Korea Patent Publication No. 10-2016-0074954 (published on June 29, 2016), state variables such as the user's mood or situation are collected and counseling is provided through inference through an expert system technique according to the collected information. An artificial intelligence conversation system that provides is disclosed.

A method and system capable of providing a conversation with a target person by generating a dialogue model having the tone and style of the person on the basis of the contents including the voice of the target person are provided.

An artificial intelligence conversation method executed in a computer system, the computer system comprising at least one processor configured to execute computer readable instructions contained in a memory, the artificial intelligence conversation method comprising: , generating a dialogue model comprising the tone and style of the target person based on data related to the target person; and providing, by the at least one processor, an answer corresponding to the request by using the conversation model according to the request of the client.

According to one aspect, the generating may include: collecting data for learning the dialogue model using content including the voice and style of the target person; and learning by matching text information and voice information through a neural network based on the collected data.

According to another aspect, the generating may include generating the dialogue model by learning a file including the voice or style of the target person uploaded from the client.

According to another aspect, in the generating step, a file containing the voice or style of the target person uploaded from the client with respect to the mood selected by the client is learned to generate a dialogue model of a version corresponding to the mood may include steps.

According to another aspect, the generating may include exposing the conversation model to a tradable platform through purchase according to whether or not disclosure is set in the client.

According to another aspect, the artificial intelligence conversation method further includes, by the at least one processor, receiving evaluation information for the answer from the client and re-learning the conversation model based on the evaluation information can do.

According to another aspect, the providing may include providing a conversation using a voice call or a conversation using a chat room as a conversation function with the target person.

According to another aspect, the providing may include providing an interface for feeding back a user evaluation of the conversation with the target person through the client when the voice call is terminated when the conversation using the voice call is provided. may include steps.

According to another aspect, in the providing step, when a message provided as an answer of the target person is selected when a conversation using the chat room is provided, an interface for feeding back a user evaluation of the message through the client is provided. may include steps.

According to another aspect, the interface may include at least one of a recommended answer including an answer different from the message, and an interface for directly writing an answer different from the message.

A computer system comprising: at least one processor implemented to execute computer readable instructions contained in a memory, wherein the at least one processor is configured with a tone and style of the target person based on data related to the target person the process of creating a dialogue model; and a computer system that processes a process of providing an answer corresponding to the request by using the conversation model according to the request of the client.

According to embodiments of the present invention, a conversation with the target person may be provided by generating a dialogue model having the tone and style of the target person by using the content including the target person's voice.

1 is a block diagram for explaining an example of an internal configuration of a computer system according to an embodiment of the present invention.
2 to 9 are exemplary views for explaining a process of generating a dialogue model having the tone and style of the target person according to an embodiment of the present invention.
10 is a diagram illustrating an example of a client-server environment supporting an artificial intelligence conversation according to an embodiment of the present invention.
11 to 16 are diagrams illustrating an example of an application screen for a conversation with a target person according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Embodiments of the present invention relate to conversation technology using artificial intelligence (AI).

Embodiments including those specifically disclosed in this specification may provide an environment in which conversations are made with a voice and style of a real person by utilizing artificial intelligence technology.

1 is a block diagram for explaining an example of an internal configuration of a computer system according to an embodiment of the present invention. For example, an artificial intelligence dialog system according to embodiments of the present invention may be implemented through the computer system 100 of FIG. 1 . As shown in FIG. 1 , the computer system 100 is a component for executing an artificial intelligence conversation method, including a processor 110 , a memory 120 , a persistent storage device 130 , a bus 140 , and an input/output interface ( 150 ), and a network interface 160 .

Processor 110 may include or be part of any device capable of processing a sequence of instructions as a component for an artificial intelligence conversation. Processor 110 may include, for example, a computer processor, a processor in a mobile device, or other electronic device and/or a digital processor. The processor 110 may be included in, for example, a server computing device, a server computer, a series of server computers, a server farm, a cloud computer, a content platform, and the like. The processor 110 may be connected to the memory 120 through the bus 140 .

Memory 120 may include volatile memory, persistent, virtual, or other memory for storing information used by or output by computer system 100 . The memory 120 may include, for example, random access memory (RAM) and/or dynamic RAM (DRAM). Memory 120 may be used to store any information, such as state information of computer system 100 . Memory 120 may also be used to store instructions of computer system 100 including instructions for, for example, artificial intelligence conversations. Computer system 100 may include one or more processors 110 as needed or appropriate.

Bus 140 may include a communications infrastructure that enables interaction between various components of computer system 100 . Bus 140 may carry data between, for example, components of computer system 100 , such as between processor 110 and memory 120 . Bus 140 may include wireless and/or wired communication media between components of computer system 100 and may include parallel, serial, or other topological arrangements.

Persistent storage 130 is a component, such as memory or other persistent storage, as used by computer system 100 to store data for an extended period of time (eg, compared to memory 120 ). may include Persistent storage 130 may include non-volatile main memory as used by processor 110 in computer system 100 . Persistent storage 130 may include, for example, flash memory, a hard disk, an optical disk, or other computer-readable medium.

The input/output interface 150 may include interfaces to a keyboard, mouse, voice command input, display, or other input or output device. Configuration commands and/or input for an artificial intelligence conversation may be received via input/output interface 150 .

Network interface 160 may include one or more interfaces to networks such as a local area network or the Internet. Network interface 160 may include interfaces for wired or wireless connections. Configuration commands and/or input for an artificial intelligence conversation may be received via network interface 160 .

Also, in other embodiments, computer system 100 may include more components than those of FIG. 1 . However, there is no need to clearly show most of the prior art components. For example, the computer system 100 is implemented to include at least some of the input/output devices connected to the above-described input/output interface 150, or a transceiver, a global positioning system (GPS) module, a camera, various sensors, It may further include other components such as a database and the like.

Hereinafter, a specific embodiment of a conversation method and system that mimics the tone and style of a real person will be described.

Referring to FIG. 2 , in the embodiments of the present invention, as the collection and storage of data becomes very easy, a large amount of data can be accumulated through contents (books, Internet posts, videos, etc.) containing the voice and style of the target person. As computing technology for processing artificial intelligence is greatly improved, it is possible to imitate the tone and style of the target person based on the learned data through machine learning of accumulated data and learning using artificial intelligence. Conversational model can be implemented.

Referring to FIG. 3 , the artificial intelligence dialog system according to the present invention can learn and build a voice model and a dialog model of a famous person such as a great person, and learn and build a voice model and a dialog model of each user for each user who uses the service can do. For example, as shown in FIG. 4 , the processor 110 may analyze the conversation intention and object from the input voice through natural language processing such as morphological analysis for the voice input from the user. Subsequently, the processor 110 may analyze the story based on the dialogue intention and the analysis result of the object, and then generate and output a dialogue corresponding to the corresponding story.

For example, referring to FIG. 5 , in the case of the target person 'Sun-Shin Lee', the processor 110 selects the diary of Nanjung as the related content, and performs data pre-processing, morpheme analysis, word dictionary generation, etc. You can create a model. The data preprocessing process may include deletion of special characters or symbols, conversion to a target language, etc., and the morpheme analysis process may include removing unnecessary words such as prepositions and conjunctions. In the word dictionary generation process, words remaining after the morpheme analysis process may be generated as a word dictionary related to the target person (word2vec), and may be vectorized through word embedding. In the model generation process, a character generation model and a question answer model can be generated by learning the whole sentence related to the target person and the question and answer pair. In addition, the processor 110 may generate a sentence-based tone generation model by learning the noun and the entire sentence after going through a morpheme analysis process as a tone generation model of the target person 'Sun-Shin Yi'.

The processor 110 may use Bi-directional Encoder Representations form Transformers (BERT), which is a natural language model for natural language processing. The BERT model is a model that trains the model in advance with unlabeled data, such as wiki or book data, and then transfers data to data with a specific task (labeled data). , a methodology to increase performance by designing a general-purpose solution, implementing it in a scalable form, and training it with many machine resources.

The processor 110 may utilize a waveNet and a pixel convolution neural network (CNN) to generate a speech model. Therefore, it is possible to generate a voice model having an intrinsic naturalness, and it can be used for a dialogue model that produces a variety of voices with one model. In the case of a speech model using a pixel CNN, the characteristics of a speaker can be identified among many speakers, and the performance is superior to other speech synthesis techniques and natural speech synthesis is possible.

The process of processing the original text of 'Nanjung Diary' as a content related to the target person 'Lee Soon-shin' is shown in FIG. 6 . After the processor 110 undergoes a pre-processing process of removing unnecessary words such as Chinese characters and parentheses, the title (indicated by an underscore) is configured as a template that can be answered when there is a question related to the title, and the The entire sentence corresponding to the content (indicated by the dotted line box) can be learned by using a neural network (eg, RNN, BERT, etc.).

The process of realizing the target character 'Sun-Shin Lee' is shown in FIG. 7 . The processor 110 collects data based on a source (eg, drama Yi Sun-sin) that implements the voice of the target person 'Sun-Shin Yi', and converts the speech corresponding to the voice of 'Sun-Shin Yi' to a text having the style of 'Sun-Sin Lee'. It can be learned by matching with In other words, after constructing data related to the target person based on the voice data, the text information and the voice information can be matched using a neural network.

Information for implementing a dialogue function that mimics the tone and style of the target person may be configured as shown in FIG. 8, and a service environment may be configured as shown in FIG. 9 so that the dialogue model can be structurally and flexibly taken. The processor 110 can advance the deep learning model by continuously conducting real-time learning of the artificial intelligence model, and based on the advanced artificial intelligence model, the user can provide a dialog function that imitates the tone and style of the target person. can

As shown in FIG. 10, the artificial intelligence dialog system according to the present invention can provide an artificial intelligence dialog function through an API call method in a platform through interworking between a client and a server in a client-server environment.

Hereinafter, a specific embodiment will be described through the client-side service screen.

Referring to FIG. 11 , the client is a dedicated application installed in an electronic device such as a smart phone, a mobile phone, or a tablet PC used by the user. have. In this case, the person list is a list of conversation partners, and may include virtual people that can be communicated with by learning in advance such as a great person or a famous person on the service platform. The contact list may include a subject learned from the user's own voice (another me) and a subject learned from another person's voice (my own friend) in a user login environment, rather than a person provided by the service platform.

Referring to FIG. 12 , the client provides a 'create new' interface screen for creating a user who can directly communicate with each other. The 'New' interface screen may include an interface for inputting information about a user to be newly added, and an interface for selecting whether to disclose, uploading a file, and the like. It is decided whether or not to disclose the added user's conversation model to the outside (other people) according to the disclosure set in the client. If it is treated as private, it will be exposed only to the user's own contacts, and if it is treated as public, it may be exposed to the store platform that can be traded through purchase. When additional users are disclosed, revenue is distributed among users. In addition, through the 'New' interface screen, files containing the voice or style of the person to be added, such as voice files, video files, text files, etc., can be uploaded as data required for learning, and the platform By learning the file, you can create an interactive conversational model.

Referring to FIG. 13 , the client provides a 'make another me' interface screen for creating a user who can communicate with each mood. The 'Make Another Me' interface screen includes an interface for selecting a user by mood (eg, happy me, depressed me, etc.) and uploading a file containing a voice or style of the selected mood. In other words, when a user selects a specific mood and uploads a conversation or article with an atmosphere that can feel that mood, the platform can learn based on the mood selected by the user and the uploaded file to create a corresponding version of the conversation model.

Referring to FIG. 14 , the client may provide an interface for selecting a conversation method with a conversation partner in providing a list of learned conversation partners with which the user can talk. The learned conversation method with the conversation partner may include a method of communicating in the form of sending and receiving text-based messages through a chat room, a method of communicating through a voice call, and the like. Whether the interface for selecting a conversation method is activated or not is determined according to data that can be supported by a conversation model such as voice or handwriting, and an impossible option can be displayed as disabled.

Referring to FIG. 15 , the client may provide a voice call interface when talking with the interlocutor through a voice call, and during a call, a call screen similar to a general call screen may be displayed to convey a feeling similar to an actual phone call. . In this case, since the conversation partner is a learned virtual person, not a real person, an image or icon representing the conversation partner may be displayed on the voice call interface to distinguish them. The client may additionally provide an interface capable of feeding back a user evaluation of the conversation with the interlocutor when the voice call is terminated. The platform can receive user evaluation and use it as data for retraining the conversational model of the interlocutor.

Referring to FIG. 16 , the client may provide a chatting conversation interface capable of communicating in the form of sending and receiving messages in a chat room, and may provide a chatting environment similar to that of an existing messenger application. The platform can receive a user's message, analyze the received message, and provide a response that does not interrupt the conversation based on the learned conversation partner's data. The client may provide an interface for evaluating the content of the conversation with the interlocutor. It is an interface that the user can directly evaluate for each chat message. For example, if you long press the contact's reply message, the 'Like' button and Report button are activated, and you can send feedback on the answer by pressing a specific button. Based on user feedback, the platform can retrain and enhance the conversational model of a contact. If the user clicks the 'Like' button on a contact's answer, it is considered positive data, and if the user clicks the report button, it is considered negative data. or provide additional other options.

When the user presses the report button on the answer of the interlocutor, the client may provide a detailed screen in which a specific evaluation is possible, for example, the user may provide an interface through which the user can directly write and deliver a correct recommended answer. As another example, when the user presses the report button on the answer of the conversation partner, the client may provide a suggested answer composed of an answer different from the corresponding answer. In this case, when it is determined that there is no appropriate answer in the recommended answer, an interface through which the user can directly write and deliver a correct answer may be provided. On the platform, the conversation model of the interlocutor can be re-learned and strengthened based on the answer selected from the suggested answers by the user or the answer written directly.

The client may provide a conversation history interface that summarizes the history of conversations/calls with the user and the conversation history interface. have. In this case, the conversation history interface may expose the last conversation history or call history with the user for each conversation partner.

The client may provide a store interface through which the user may purchase chat contacts. The store interface may include a pre-trained and ready-to-use conversation list, and may include an interface through which a user can select and pay.

According to embodiments of the present invention, it is possible to provide communication as if talking with a real person through a virtual conversation partner, and unlike a real person, it is possible to communicate anytime and anywhere, and social costs can also be reduced. The artificial intelligence conversation system according to the present invention can be a solution to prevent loneliness and overcome the sense of loss through conversation with a deceased celebrity.

According to embodiments of the present invention, even if the same message is delivered by a celebrity, it has more ripple effect. It can enhance the learning effect.

By applying the embodiments of the present invention, it is possible to dub through the extraction of the voice of a celebrity. Although it takes a lot of time and effort to actually dub or record a voice, if there is a recording file containing the voice of a person prepared in advance through the present invention, the voice can be extracted without the participation of the real person, and furthermore, the dialogue is of a specific person can be changed according to In addition, it is possible to insert a voice based on the tone of a specific person in the past without a separate ad-lip, and the voice extracted from the recording file can be used in movies, games, and dramas.

By applying the embodiments of the present invention, it is possible to produce educational content having the tone and style of a famous person. Even with the same educational content, there is a difference in the learning effect depending on who teaches. Public interest campaigns are more effective when celebrities participate, but there are limitations in terms of time and cost when celebrities actually participate. Therefore, a message or learning content to be delivered can be produced with the voice of a virtual person learned by artificial intelligence.

Embodiments of the present invention may be linked with an artificial intelligence speaker. As the number of speakers of artificial intelligence speakers is becoming more diverse, the use of artificial intelligence speakers can be expanded by adding the voice of a person desired by the user to the artificial intelligence speaker.

In addition, embodiments of the present invention can be applied to corporate work. For example, it is possible to support a model in which the user performs the role of a personal assistant through conversation with a conversation partner having his or her own voice, a model in which business support is performed through conversation with various business experts, and the like.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, the devices and components described in the embodiments may include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable logic unit (PLU). It may be implemented using one or more general purpose or special purpose computers, such as a logic unit, microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be embodied in any type of machine, component, physical device, computer storage medium or device for interpretation by or providing instructions or data to the processing device. have. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. In this case, the medium may be to continuously store the program executable by the computer, or to temporarily store the program for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributed on a network. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media may include recording media or storage media managed by an app store that distributes applications, sites that supply or distribute other various software, and servers.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (15)

In an artificial intelligence conversation method executed in a computer system,
the computer system comprising at least one processor configured to execute computer readable instructions contained in a memory;
The artificial intelligence conversation method is
generating, by the at least one processor, a dialogue model comprising the tone and style of the target person based on data related to the target person; and
providing, by the at least one processor, an answer corresponding to the request by using the conversation model according to the request of the client;
including,
The step of providing an answer corresponding to the request comprises:
Providing a conversation using a chat room as a conversation function with the target person
including,
The step of providing a conversation using the chat room includes:
providing an interface for feeding back a user evaluation of the message through the client when a message provided as an answer of the target person is selected
including,
The interface includes an interface for inputting a user evaluation regarded as positive data for the message, a recommended answer composed of an answer different from the message, and an interface for directly writing an answer different from the message,
The artificial intelligence conversation method is
re-learning, by the at least one processor, the dialog model based on the answer selected from the recommended answer or the directly written answer
Artificial intelligence conversation method further comprising. According to claim 1,
The generating step is
collecting data for learning the dialogue model by using the content including the voice and style of the target person; and
Learning by matching text information and voice information through a neural network based on the collected data
An artificial intelligence conversation method that includes. According to claim 1,
The generating step is
generating the dialogue model by learning a file including the voice or style of the target person uploaded from the client
An artificial intelligence conversation method that includes. According to claim 1,
The generating step is
generating a dialogue model of a version corresponding to the mood by learning a file including the voice or style of the target person uploaded from the client with respect to the mood selected by the client
An artificial intelligence conversation method that includes. 4. The method of claim 3,
The generating step is
Exposing the conversation model to a tradable platform through purchase according to the disclosure set in the client
An artificial intelligence conversation method that includes. delete According to claim 1,
The step of providing an answer corresponding to the request comprises:
Providing a conversation using a voice call as a conversation function with the target person
Artificial intelligence conversation method further comprising. 8. The method of claim 7,
The step of providing a conversation using the voice call comprises:
When the voice call is terminated, providing an interface for feeding back a user evaluation of the conversation with the target person through the client
An artificial intelligence conversation method that includes. delete delete In a computer system,
at least one processor implemented to execute computer readable instructions contained in a memory
including,
the at least one processor,
generating a dialogue model consisting of a tone and style of the target person based on data related to the target person; and
A process of providing an answer corresponding to the request using the conversation model according to the request of the client
process the
The process of providing an answer corresponding to the request is,
Provides a conversation using a chat room as a conversation function with the target person,
In the case of providing a conversation using the chat room, when a message provided as an answer of the target person is selected, an interface for feeding back a user evaluation of the message through the client is provided,
The interface includes an interface for inputting a user evaluation regarded as positive data for the message, a recommended answer composed of an answer different from the message, and an interface for directly writing an answer different from the message,
the at least one processor,
Re-learning the dialog model based on the answer selected from the recommended answer or the directly written answer
A computer system characterized by a. 12. The method of claim 11,
The generating process is
Learning by matching text information and voice information through a neural network based on the collected data after collecting data for learning the dialogue model using the content including the voice and style of the target person
A computer system characterized by a. 12. The method of claim 11,
The generating process is
Generating the dialogue model by learning a file including the voice or style of the target person uploaded from the client
A computer system characterized by a. delete 12. The method of claim 11,
The process of providing an answer corresponding to the request is,
Provides a conversation using a voice call as a conversation function with the target person,
When providing a conversation using the voice call, when the voice call is ended, providing an interface for feeding back a user evaluation of the conversation with the target person through the client
A computer system characterized by a.

Images