JP7572388B2 - Data processing device, data processing method and program - Google Patents

Description

The present invention relates to a data processing device, a data processing method, and a program.

Technology is known that synthesizes text data and phoneme data to read text in a desired voice (see, for example, Patent Document 1).

JP 2002-328694 A

However, this type of speech synthesis technology combines prepared phonemes to read text aloud, but does not fully meet the needs of users, such as busy parents who want to deepen family bonds by listening to someone read aloud in their own voice. Furthermore, conventional speech synthesis devices output a voice that sounds like a mechanical reading of text data, which can be unnatural to the listener.

The present invention has been made in consideration of these points, and aims to enable reading of text reproduced in the desired voice, with the desired intonation or between other readers.

The data processing device of the first aspect of the present invention includes a recitation information storage unit that stores recitation information, which is time-series data indicating features of the voice of a first speaker reciting content consisting of character strings, and which associates a time in the playback time of the content with a character spoken at that time, a volume, and a pitch; a vocalization information storage unit that stores vocalization information generated by sampling a voice spoken by a second speaker, which associates multiple characters with the timbre of the voice spoken when pronouncing each of the multiple characters; a generation unit that generates recitation data for outputting the characters at each of the times in the recitation information as a sound consisting of the timbre corresponding in the vocalization information and the volume and pitch at that time; and an output control unit that controls the output of the recitation data.

The reading information storage unit may store the reading information further associated with first reference pitch data indicating a reference pitch in the content, the vocalization information storage unit may further store second reference pitch data indicating a reference pitch for the second speaker, and the generation unit may generate the reading data for outputting the tone corresponding in the vocalization information to the characters at each of the times in the reading information at a pitch determined based on the pitch at that time and the ratio between the first reference pitch data and the second reference pitch data.

The device may further include a dialect information storage unit that stores dialect information associating a plurality of words, a dialect to which each of the plurality of words corresponds, one or more characters constituting the dialect, and a pitch for pronouncing the characters constituting the dialect, and the reading information storage unit stores the reading information further associating the words constituting the reading information with one or more of the characters constituting the words, and the generation unit may further generate replacement reading information in which the one or more characters and the pitch corresponding to the one or more characters included in the reading information are replaced with characters included in the dialect corresponding in the dialect information to the words constituting each of the one or more characters, and a pitch for pronouncing the characters included in the dialect, and generate the reading data that causes the replaced characters at each of the times of the generated replacement reading information to be output as a sound consisting of the tone corresponding in the utterance information, the volume at that time, and the replaced pitch.

The reading information storage unit may store the reading information in which the time is associated with a flag indicating the timing of inserting a phrase, and the generation unit may generate the reading data that outputs a phrase selected from a plurality of predetermined phrases at the timing indicated by the flag.

The speech information storage unit may further store image data corresponding to the second speaker, and the output control unit may control the display unit to display an image corresponding to the second speaker when controlling the reading data to be output.

The device further includes an imaging data acquisition unit that acquires imaging data of a user, and a determination unit that determines the state of the user by performing image analysis on the imaging data acquired from the imaging data acquisition unit, and the output control unit may stop outputting the reading data or change the output mode of the reading data when the determination unit determines that the state of the user is that the user is asleep.

The device may further include an imaging data acquisition unit that acquires imaging data of a user, and a determination unit that performs image analysis of the imaging data acquired from the imaging data acquisition unit to determine the state of the user and notify the information terminal of the determined state of the user.

The reading information storage unit may store a plurality of pieces of reading information in which the content is read by a plurality of different first speakers, in association with a situation for which each piece of reading information is suitable, the determination unit may determine the user's situation based on at least one of the user's attributes and the user's state, and the generation unit may select the reading information associated with the user's state determined by the determination unit, and generate the reading data that outputs the characters at each time of the selected reading information as a sound consisting of the tone corresponding to the voice information in the speech information and the volume and pitch at that time.

The recitation information storage unit further stores dictionary information that associates words included in the content to be recited in the recitation information with the meanings of the words in association with the recitation information, and the data processing device has a voice information acquisition unit that acquires voice information spoken by a user viewing the content, and a voice recognition unit that performs voice recognition on the voice information acquired by the voice information acquisition unit and acquires the content of the user's utterance, and when the content of the user's utterance acquired by the voice recognition unit is a question about the content, the generation unit refers to the dictionary information and generates answer information indicating an answer to the question, and the output control unit may control to output the answer information.

The system may further include a reaction information storage unit, and the voice recognition unit may store reaction information in the reaction information storage unit that associates the user's utterance content with the time corresponding to the timing at which the user uttered the utterance content.

The data processing method of the second aspect of the present invention may include a step of acquiring recitation information, which is time-series data indicating features of the voice of a first speaker who recites content composed of character strings stored in a recitation information storage unit and executed by a computer, and which associates a time in the playback time of the content with a character spoken at that time, a volume, and a pitch; a step of acquiring utterance information, which is generated by sampling the voice spoken by a second speaker stored in the utterance information storage unit and which associates and stores a plurality of characters with the timbre of the voice spoken when pronouncing each of the plurality of characters; a step of generating recitation data for outputting the characters at each of the times in the recitation information as sounds consisting of the timbre corresponding in the utterance information and the volume and the pitch at the times; and a step of controlling the output of the recitation data.

In the program of the third aspect of the present invention, a computer is caused to execute the steps of: acquiring recitation information, which is time-series data showing the characteristics of the voice of a first speaker who recites content composed of character strings stored in a recitation information storage unit, and which associates a time in the playback time of the content with the character spoken at that time, the volume, and the pitch; acquiring utterance information, which is generated by sampling the voice spoken by a second speaker stored in the utterance information storage unit, and which associates and stores a plurality of characters with the timbre of the voice spoken when pronouncing each of the plurality of characters; generating recitation data for outputting the characters at each of the times in the recitation information as sounds consisting of the timbre corresponding in the utterance information and the volume and the pitch at the time; and controlling the output of the recitation data.

The present invention has the effect of enabling text to be read aloud in a voice that reproduces the desired intonation and pauses between other readers.

FIG. 1 is a diagram illustrating an overview of a data processing system S according to an embodiment. 1 is a block diagram showing a configuration of a data processing device 1. FIG. 4 is a diagram showing an example of a data structure of reading information stored in a reading information storage unit 121. FIG. FIG. 13 is a diagram showing an example of a data structure of metadata associated with reading information. 4 is a diagram showing an example of a data structure of utterance information stored in the utterance information storage unit 122. FIG. A figure showing an example of the data structure of dialect information stored in the dialect information storage unit 123. 4 is a diagram showing an example of a data structure of dictionary information stored in a reading information storage unit 121. FIG. 3 is a flowchart showing a process flow in the data processing device 1.

[Overview of data processing device 1]
1 is a diagram illustrating an overview of a data processing system S according to an embodiment. The data processing system S is a system for allowing a user to listen to a reading of content. The data processing system S includes a data processing device 1 and an information terminal 2.

The data processing device 1 is a device that generates reading data for outputting a voice that reproduces a reading of distinctive content by a first speaker in the voice quality of a second speaker that is different from the speaker who read the content. The data processing device 1 is, for example, a server or a personal computer.

In a suitable example of use, the first speaker is a professional reader, narrator, actor, voice actor, etc. The second speaker is, for example, the parent, sibling, grandparent, friend, etc. of the user listening to the content, or the creator of the content, such as a writer. By generating reading data using such voice by the data processing device 1, the user can enjoy a distinctive reading by a professional narrator, etc., in the voice of someone familiar to the user. Reading using the data processing device 1 is effective, for example, in cases where you want a baby to remember the voice of its parent, or to feel reassured by having the baby hear the voice of its parent.

The information terminal 2 is a device for outputting audio corresponding to the reading data input from the data processing device 1. The information terminal 2 is, for example, a personal computer, a smart speaker, a smartphone, or a tablet. The data processing device 1 and the information terminal 2 may be configured as one unit.

The data processing device 1 stores recitation information indicating characteristics of the voice of the first speaker S1 reciting the content. The content is, for example, a picture book, a novel, a manga, or a book such as a textbook, a script for a play, or a paper theater. The recitation information is time-series data indicating characteristics of the voice of the first speaker S1 reciting the content. The recitation information associates a time in the playback time of the content with the characters spoken at that time, the volume, and the pitch. As an example, the data processing device 1 acquires audio data of the content reciting the content by the first speaker S1, and generates the recitation information by analyzing the voice volume, pitch, and spoken characters of the acquired audio data. The recitation information can also be said to be data indicating the volume and pitch at which the characters constituting the content are spoken and at what time after the start of playback of the content.

The data processing device 1 may store multiple pieces of reading information generated by different first speakers reading the same content. Storing reading information in which different first speakers read the same content makes it possible to select a reading that is appropriate for the situation from among unique readings. Also, multiple pieces of reading information generated by the same speaker reading different content may be stored.

The data processing device 1 stores voice information indicating the voice quality of a second speaker S2, who is different from the first speaker S1. The voice information associates a plurality of characters with the timbre of the voice uttered by the second speaker S2 when pronouncing each of the plurality of characters. As an example, the data processing device 1 acquires voice data obtained by sampling the voice uttered by the second speaker S2, and generates the voice information by analyzing the pitch, timbre, and spoken characters of the acquired voice data.

The user U operates the information terminal 2 to select the content to be listened to and the second speaker. The information terminal 2 transmits operation information including the content to be listened to and the second speaker to the data processing device 1. The data processing device 1 acquires recitation information corresponding to the content to be listened to and speech information corresponding to the second speaker, which are included in the operation information.

The data processing device 1 generates reading data by synthesizing the acquired reading information and vocalization information. Specifically, the data processing device 1 generates reading data that indicates, in a time series, a sound consisting of a tone color in the vocalization information that corresponds to the character at each time included in the reading information, and the volume and pitch at the time indicated by the reading information.

Then, the data processing device 1 outputs the reading data to the information terminal 2. By configuring the data processing device 1 in this way, it is possible to read the text aloud in a desired voice, reproducing the pauses and intonation of another reader's reading.

[Configuration of data processing device 1]
2 is a block diagram showing the configuration of the data processing device 1. The data processing device 1 has a communication unit 11, a storage unit 12, and a control unit 13. The storage unit 12 has a recitation information storage unit 121, an utterance information storage unit 122, a dialect information storage unit 123, and a reaction information storage unit 124. The control unit 13 has an operation acceptance unit 131, a generation unit 132, an output control unit 133, an imaging data acquisition unit 134, a determination unit 135, a voice information acquisition unit 136, and a voice recognition unit 137.

The communication unit 11 is a communication interface for transmitting and receiving data to and from other devices. The storage unit 12 is a storage medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), an SSD (Solid State Drive), or an HDD (Hard Disk Drive). The storage unit 12 stores various programs executed by the control unit 13.

The reading information storage unit 121 stores reading information, which is time-series data indicating the characteristics of the voice of a first speaker who reads content consisting of character strings, and which associates a time in the playback time of the content with the characters spoken at that time, the volume, and the pitch. Figure 3 is a diagram showing an example of the data structure of reading information stored in the reading information storage unit 121. In the reading information shown in Figure 3, "time", "characters", "volume", and "pitch" are associated with each other.

The "time" in the reading information indicates the time at which the data is located during the playback time of the content. As an example, the "time" is expressed in milliseconds or microseconds. The "time" may also be expressed as an index indicating the timing. In this case, the time can be obtained by multiplying the sampling period by the index.

"Characters" in the reading information indicate the characters that are spoken at that time. In FIG. 3, as an example, "characters" are shown as kana characters that correspond to the sounds that are spoken. When shown as kana characters, a specified flag ("!" in FIG. 3) is inserted in the "characters" at the timing when the consonants are pronounced. This distinguishes whether the sound that is spoken is a sound that includes a consonant or a vowel. "Characters" may be shown by spoken vowels and consonants instead of kana characters. A "-" in "characters" indicates that no characters are spoken at that time. In other words, that time indicates that the first speaker has taken a pause in the reading.

The "volume" in the reading information indicates the loudness of the sound spoken at the associated time. As an example, the "volume" may be expressed as a ratio to a reference volume, or as the sound pressure (dB). The "pitch" indicates the pitch of the characters spoken at the associated time. The "pitch" may be indicated by the pulse period or by the frequency.

The reading information storage unit 121 stores reading information that further associates words that make up the reading information with one or more characters that make up the word. In the reading information shown in FIG. 3, a "word" is further associated. The "word" indicates which word and which character the "character" uttered at the relevant time indicates. As an example, information is stored indicating that "mu" at "time" 1 to 5 is the first character of the word "mukashi" corresponding to word ID M01, and that "ka" at times 6 to 11 is the second character of the word "mukashi". The word ID is information that identifies a word.

The reading information storage unit 121 stores reading information that associates a time with a flag indicating the timing for inserting a phrase. In the reading information shown in FIG. 3, "phrase insertion" is further associated. "Phrase insertion" is a flag indicating the timing for inserting a specific phrase at each time, and "1" indicates that it is the timing for inserting a phrase. In addition, the reading information may be associated with a flag indicating the type of phrase to be inserted depending on the scene of the content.

The reading information storage unit 121 may store a plurality of pieces of reading information in which the content is read by a plurality of different first speakers in association with a situation in which each piece of reading information is suitable.
The reading information storage unit 121 stores metadata of the reading information in association with the reading information. Fig. 4 is a diagram showing an example of the data structure of the metadata associated with the reading information. The metadata of the reading information is associated as metadata with a "reading information ID" that identifies the reading information, "content" that identifies the content to be read aloud, "speaker ID" information that identifies the second speaker who read the content, and "characteristics" that indicate the situation in which the reading information is suitable. The "characteristics" include information that indicates the situation in which each reading is suitable, such as "cheerful", "sleepy", "calming", and "exciting".

The vocalization information storage unit 122 stores vocalization information generated by sampling the voice uttered by the second speaker, in which multiple characters are associated with the tone of the voice uttered when pronouncing each of the multiple characters. FIG. 5 is a diagram showing an example of the data structure of vocalization information stored in the vocalization information storage unit 122. The vocalization information associates information identifying the speaker, characters, and sound waveforms corresponding to the tone of the characters. As an example, the sound waveforms in FIG. 5 store information that records the waveforms corresponding to each character in chronological order.

The dialect information storage unit 123 stores dialect information that associates a plurality of words, a dialect to which each of the plurality of words corresponds, one or more characters that constitute the dialect, and a pitch for pronouncing the characters that constitute the dialect. FIG. 6 is a diagram showing an example of the data structure of dialect information stored in the dialect information storage unit 123. In the dialect information, a "dialect type", a "word ID", a "word", a "dialect", a "character", and a "pitch" are associated. The "dialect type" indicates, for example, Kansai dialect, Okinawa dialect, etc. The "dialect" indicates the dialect that corresponds to the "word" in the dialect indicated by the "dialect type". As an example, it is stored that "mukashi" corresponds to "nkashi". The "character" indicates the character that constitutes the "dialect". It is stored that "nkashi" is composed of "n", "ka", and "shi". The "pitch" indicates the pitch that should be given to each character when pronouncing the dialect.

The control unit 13 is, for example, a CPU (Central Processing Unit). The control unit 13 executes a control program stored in the storage unit 12, thereby functioning as an operation reception unit 131, a generation unit 132, an output control unit 133, an imaging data acquisition unit 134, a determination unit 135, a voice information acquisition unit 136, and a voice recognition unit 137.

[Speech synthesis processing]
The operation acceptance unit 131 accepts operation information corresponding to a user's operation from the information terminal 2. As an example, the operation acceptance unit 131 accepts a playback instruction including a content selected by the user.

The generation unit 132 generates reading data that outputs the characters at each time in the reading information as a sound consisting of a tone corresponding to the voice information and the volume and pitch at that time. The generation unit 132 acquires reading information and voice information corresponding to the content selected by the user. First, the generation unit 132 acquires the sound waveform of the character in the voice information that corresponds to the "character" at each time in the acquired reading information. That is, if the character in the reading information at a certain time is "mu", the generation unit 132 acquires the waveform of "mu" in the voice information. Then, the generation unit 132 increases or decreases the pitch and wave strength of the acquired sound waveform based on the pitch and volume at that time in the reading information. By repeating this for the characters at each time included in the reading information, the generation unit 132 generates reading data.

The output control unit 133 controls the output of the reading data. As an example, the output control unit 133 transmits the reading data and an instruction to output the reading data as audio to the information terminal 2.

However, for example, if the pitch of the voices of the first and second speakers differs significantly, reproducing the timbre of the second speaker using the pitch of the first speaker will result in an unnatural reading. Therefore, the data processing device 1 can be configured to adjust the output pitch based on the difference in pitch between the first and second speakers, thereby providing a natural reading.

In this case, the reading information storage unit 121 stores the reading information further associated with first reference pitch data indicating the reference pitch in the content. The metadata of the reading information shown in FIG. 4 includes "reference pitch" indicating the first reference pitch data in association with the reading information. The first reference pitch data is, as an example, the average or median pitch of the voice of the first speaker reading the content.

The speech information storage unit 122 may further store second reference pitch data indicating the reference pitch of the second speaker. As an example, the second reference pitch data is the average and median of the pitches recorded when recording the voice of the second speaker.

The generation unit 132 generates reading data that outputs a tone in the vocalization information that corresponds to a character at each time in the reading information, at a pitch determined based on the pitch at that time and the ratio between the first standard pitch data and the second standard pitch data. The generation unit 132 calculates the ratio between the first standard pitch data and the second standard pitch data. The generation unit 132 then multiplies the pitch at each time of the reading data by the calculated ratio to determine the pitch to be spoken at that time of the reading data. By configuring the data processing device 1 in this way, reading can be performed with a natural intonation that takes into account the difference in pitch between the first and second speakers.

By visually showing the second speaker to the user listening to the reading, the user can feel as if the second speaker is actually reading. In this case, the utterance information storage unit 122 further stores image data corresponding to the second speaker. The utterance information shown in FIG. 5 is associated with a "speaker image," which is image data including, as a subject, the second speaker who provided the voice included in the utterance information. The image data is, for example, a still image, a video, a three-dimensional image, an AR (Augmented Reality) image, or a VR (Virtual Reality) image.

When controlling to output the reading data, the output control unit 133 may control to display an image corresponding to the second speaker on the display unit. The generation unit 132 acquires image data associated with the speech information corresponding to the selected second speaker. The output control unit 133 displays the acquired image data on the display unit of the information terminal 2. As an example, the output control unit 133 may output an image of the mouth of a person contained in the image data so that the image changes in conjunction with the output of the reading data. This configuration allows the user to get the feeling that the second speaker is actually speaking.

[Replacement of reading information]
Although an example of changing the tone of the reading based on the vocalization information has been described, the user can be more entertained by varying the content of the reading. Therefore, the data processing device 1 may read a part of the reading information by replacing it with a dialect, or may be configured to insert a phrase in the middle of the reading.

The following describes reading in a dialect. In this case, the operation reception unit 131 acquires operation information including, as an example, a flag indicating whether or not to perform reading in a dialect and the dialect selected by the user.

The generating unit 132 then generates replacement reading information in which one or more characters and a pitch corresponding to the one or more characters included in the reading information are replaced with characters included in a dialect in the dialect information corresponding to a word constituted by each of the one or more characters and a pitch for pronouncing the characters included in the dialect, and generates reading data for outputting the replaced characters at each time of the generated replacement reading information as a sound consisting of a tone corresponding to the vocalization information, a volume at that time, and a replaced pitch. If the user's operation content acquired by the operation receiving unit 131 indicates that a reading in a dialect has been selected, the generating unit 132 further acquires dialect information. The generating unit 132 then compares the words in the acquired reading information with the words included in the dialect information, and replaces the matching words with the dialect included in the dialect information. As an example, the word "mukashi" included in the reading information is replaced with "nkashi", which is the corresponding dialect in the reading information.

Then, reading data is generated to output the pitch of the replaced portion at the pitch of the dialect information. If the number of characters of a word included in the reading information does not match that of the corresponding dialect in the dialect information, as an example, the time to read each character may be shortened or extended depending on the ratio of the number of characters that make up the word and the dialect so that the time to read the word matches.

By configuring the data processing device 1 to generate reading data in which some of the words included in the reading information are replaced with a dialect, for example, when the second speaker speaks in a dialect, it becomes possible to perform reading that is close to the second speaker's original speaking style.

Next, an example of inserting a phrase in the middle of a reading will be described. The generating unit 132 generates reading data that outputs a phrase selected from a plurality of predetermined phrases at the timing indicated by the flag. When the user's operation content accepted by the operation accepting unit 131 indicates that a predetermined phrase is to be inserted and read aloud, the generating unit 132 generates reading data that outputs a sound corresponding to the predetermined phrase in the tone of the second speaker at the timing at which a flag indicating the timing at which the phrase is to be inserted in the reading information is attached. The storage unit 12 stores, as the predetermined phrase, phrases that convey impressions such as "That's amazing" or "That's interesting" and phrases that predict developments such as "Is everything okay?" or "I wonder what will happen next?". The generating unit 132 may generate reading data in which a phrase randomly selected from the predetermined phrases is inserted at the timing at which a flag is attached.

The phrase may be, for example, a catchphrase such as "I wonder if..." or "It's...". In this case, the phrase indicating the catchphrase may be inserted at the timing when a flag is added in the reading information. Also, the characters corresponding to the timing when a flag is added in the reading information may be replaced with the phrase indicating the catchphrase.

The data processing device 1 may be configured to insert a phrase appropriate to the scene of the content being read aloud. That is, the generation unit 132 inserts a phrase corresponding to the phrase to be inserted according to the scene of the content attached to the reading information at the timing when the flag is attached. In this case, each phrase is associated with an emotion corresponding to the phrase. As an example, the generation unit 132 may insert a phrase expressing an upbeat emotion such as "It's fun" when the scene of the content is a bright scene, and insert a phrase expressing an anxious emotion such as "Is everything okay?" when the scene of the content is a crisis situation.

By configuring the data processing device 1 in this way, it becomes possible to output a varied reading, which can provide greater enjoyment to the user.
[Control according to user's situation]

The output of the reading may be controlled based on the situation of the user listening to the reading of the content. By configuring it in this way, for example, in a usage scenario in which the data processing system S is used to read to a child to help them fall asleep, the reading can be stopped when the child falls asleep, or the volume can be gradually reduced before stopping.

The imaging data acquisition unit 134 acquires imaging data of the user. The imaging data acquisition unit 134 acquires imaging data captured by the imaging means of the information terminal 2. The determination unit 135 determines the state of the user by performing image analysis of the imaging data acquired from the imaging data acquisition unit 134 using a known image analysis technique. As an example, the determination unit 135 may determine the user's emotions, whether the user is concentrating on the recitation, the drowsiness the user is feeling, or whether the user is asleep, etc., by performing image recognition on the acquired imaging data.

The output control unit 133 controls the output mode based on the user's state determined by the determination unit 135. As an example, when the determination unit 135 determines that the user's state is asleep, the output of the reading data is stopped or the output mode of the reading data is changed. When the determination unit 135 determines that the user is asleep or feeling drowsy, the output control unit 133 may control the volume at which the reading data is output to be lowered or may control the output of the reading data to be stopped.

The output control unit 133 may switch the content to be read to other content based on the user's status determined by the determination unit 135. As an example, when the determination unit 135 determines that the user's status indicates that the user is not interested in reading aloud, the output control unit 133 may control the output of other content such as music. In addition, the output control unit 133 may control the generation unit 132 to generate reading data from the reading information of the other content and the selected vocalization information, and output the reading data.

The data processing device 1 may be configured to notify related parties of a user who is listening to the content that a specific situation has occurred when the specific situation occurs. For example, the data processing device 1 may notify a guardian of the user U who is listening to the reading or a caregiver who is caring for the user U.

The determination unit 135 may notify the information terminal 2 of the determined state of the user. For example, when the determination unit 135 determines that the user is angry or crying as a result of image analysis of the acquired imaging data, the determination unit 135 transmits a message notifying a predetermined notification destination of the user's state.

Incidentally, the reading information includes various unique reading information given by the first speaker, and the user can enjoy the reading even more by selecting appropriate reading information based on the user's situation determined by the determination unit 135.

The determination unit 135 determines the user's situation based on at least one of the user's attributes and the user's state. For example, the user's situation may be determined based on attributes such as the age and gender of the user U that have been registered in advance. For example, if the user is a young child, it may be determined that it is appropriate to select "exciting" reading information. Furthermore, as already described, the determination unit 135 may determine the user's situation by performing image recognition on the acquired imaging data.

The generation unit 132 selects the reading information associated with the user's situation determined by the determination unit 135, and generates reading data that outputs the characters at each time of the selected reading information as a sound consisting of a tone corresponding to the voice information and a volume and pitch at that time. The generation unit 132 acquires reading information associated with a situation that matches the user's situation determined by the determination unit 135, and generates reading data based on the acquired reading information and voice information.

[User interaction]
Although an example in which the output is controlled based on the user's reaction has been described, the data processing device 1 may be configured to control the output so as to give a response corresponding to the content of the user's utterance.

In this case, the reading information storage unit 121 further stores, in association with the reading information, dictionary information that associates words included in the content to be read in the reading information with the meanings that the words indicate. FIG. 7 is a diagram showing an example of the data structure of dictionary information stored in the reading information storage unit 121. In the dictionary information, "content", "word", and "meaning" are associated. "Content" indicates whether the meaning corresponding to the word is a general meaning or a meaning in a specific content. For example, when "content" is "general", it indicates a general meaning, and when information that identifies the content (for example, "Momotaro") is stored, it indicates a meaning specific to that content.

The voice information acquisition unit 136 acquires voice information uttered by a user viewing the content. The voice information acquisition unit 136 acquires voice information indicating the voice uttered by the user detected by a microphone mounted on the information terminal 2.

The voice recognition unit 137 recognizes the voice information acquired by the voice information acquisition unit 136 and acquires the user's speech content. The voice recognition unit 137 performs morphological analysis, syntactic analysis, and semantic analysis on the acquired speech content using known natural language processing technology, and classifies the user's speech content. As an example, the voice recognition unit 137 classifies the speech content into "question", "expression of emotion", etc. If the classified speech content is a question, the voice recognition unit 137 analyzes the acquired speech content to identify the content being asked. For example, if the user's utterance content is "What is millet dumpling?", it identifies that the question is about the meaning of millet dumpling.

When the user's utterance content acquired by the voice recognition unit 137 is a question about the content, the generation unit 132 refers to the dictionary information and generates answer information indicating the answer to the question, and the output control unit 133 controls to output the answer information. When the user's utterance content is a question about the content, the generation unit 132 searches the dictionary information and acquires the meaning corresponding to the question content identified by the voice recognition unit 137. The generation unit 132 searches between the general meaning recorded in the dictionary information and the meaning specific to the content for which the reading data is being output.

The generation unit 132 generates a response sentence by applying the acquired meaning to a predetermined format. The generation unit 132 then generates response information that reads the generated response sentence aloud using vocalization information corresponding to the second speaker, and outputs the generated response information to the information terminal 2.

The data processing device 1 may further be configured to record how the user responded to the reading. The voice recognition unit 137 stores in the reaction information storage unit 124 reaction information that associates the content of the user's utterance with the time corresponding to when the user spoke the content. The voice recognition unit 137 may store in the reaction information storage unit 124 reaction information that is further associated with information that identifies the content that was being read when the user spoke. By configuring the data processing device 1 in this way, it is possible to record the user's reaction to the reading and preserve the user's memories.

[Processing flow in data processing device 1]
Fig. 8 is a flowchart showing the flow of processing in the data processing device 1. The flowchart shown in Fig. 8 starts from the point when the device is ready to accept the selection of the reading information. The operation accepting unit 131 accepts the selection of the second speaker and the content to be read from the information terminal 2 (S101). The generating unit 132 acquires the reading information corresponding to the selected content from the reading information storage unit 121 (S102). The generating unit 132 acquires the utterance information corresponding to the selected second speaker from the utterance information storage unit 122 (S103).

The generation unit 132 determines whether a dialect selection has been accepted (S104). If a dialect selection has been accepted (YES in S104), the generation unit 132 replaces the words included in the reading information with the corresponding dialect (S105). If a dialect selection has not been accepted (NO in S104), the generation unit 132 skips the replacement process.

The generation unit 132 generates reading data based on the reading information and the vocalization information (S106). Then, the output control unit 133 controls the generated reading data to be output to the information terminal 2 (S107). Then, the data processing device 1 ends the process.

By configuring the data processing device 1 as described above, it is possible to output a reading of the text that reproduces the pauses between other readers and the reading with the desired intonation in the desired voice.

Furthermore, this invention will make it possible to contribute to Goal 9 of the United Nations' Sustainable Development Goals (SDGs), which is "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and infrastructure."

Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the invention. For example, all or part of the device can be configured by distributing or integrating functionally or physically in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment resulting from the combination also has the effect of the original embodiment.

Reference Signs List 1 Data processing device 2 Information terminal 11 Communication unit 12 Storage unit 13 Control unit 121 Reading information storage unit 122 Vocalization information storage unit 123 Dialect information storage unit 124 Response information storage unit 131 Operation acceptance unit 132 Generation unit 133 Output control unit 134 Image capture data acquisition unit 135 Determination unit 136 Voice information acquisition unit 137 Voice recognition unit

Claims (12)

a recitation information storage unit that stores recitation information which is time-series data indicating features of the voice of a first speaker reciting content consisting of character strings, the recitation information associating a time in the playback time of the content with a character uttered at the time, a volume, and a pitch;
a speech information storage unit that stores speech information generated by sampling a voice uttered by a second speaker, the speech information being stored in association with a plurality of characters and a tone of a voice uttered when pronouncing each of the plurality of characters;
a generating unit that generates reading data for outputting the characters at each of the times of the reading information as sounds having the tone color corresponding to the voice information in the utterance information and the volume and pitch at the times;
an output control unit that controls the reading data to be output;
A data processing device comprising: the reading information storage unit stores the reading information further associated with first reference pitch data indicating a reference pitch in the content;
the utterance information storage unit further stores second reference pitch data indicating a reference pitch of the second speaker;
the generating unit generates the reading data for outputting the tone color in the utterance information corresponding to the character at each of the times of the reading information at a pitch determined based on a pitch at the time and a ratio between the first standard pitch data and the second standard pitch data.
2. A data processing apparatus according to claim 1. a dialect information storage unit that stores dialect information that associates a plurality of words, a dialect corresponding to each of the plurality of words, one or more characters that constitute the dialect, and a pitch for pronouncing the character that constitutes the dialect,
The reading information storage unit stores the reading information further associated with a word constituting the reading information and one or more characters constituting the word,
The generating unit further generates replacement reading information in which the one or more characters and the pitch corresponding to the one or more characters included in the reading information are replaced with a character included in the dialect corresponding to the word constituted by each of the one or more characters in the dialect information and a pitch for pronouncing the character included in the dialect, and generates the reading data for outputting the replaced character at each of the times of the generated replacement reading information as a sound consisting of the tone corresponding to the speech information in the utterance information, the volume at the time, and the replaced pitch.
3. A data processing device according to claim 1 or 2. the recitation information storage unit stores the recitation information in which the time is associated with a flag indicating a timing for inserting a phrase;
the generating unit generates the reading data for outputting a phrase selected from a plurality of predetermined phrases at a timing indicated by the flag;
A data processing device according to any one of claims 1 to 3. the speech information storage unit further stores image data corresponding to the second speaker;
the output control unit controls a display unit to display an image corresponding to the second speaker when the output control unit controls the reading data to be output;
A data processing device according to any one of claims 1 to 4. an imaging data acquisition unit that acquires imaging data of an image of a user;
A determination unit that determines a state of the user by performing image analysis on the imaging data acquired from the imaging data acquisition unit,
The output control unit stops output of the reading data or changes an output mode of the reading data when the determination unit determines that the state of the user is asleep.
A data processing device according to any one of claims 1 to 5. an imaging data acquisition unit that acquires imaging data of an image of a user;
a determination unit that determines a state of the user by performing image analysis on the imaging data acquired from the imaging data acquisition unit, and notifies an information terminal of the determined state of the user;
A data processing apparatus according to any one of claims 1 to 5, further comprising: The reading information storage unit stores a plurality of pieces of reading information in which the content is read by a plurality of different first speakers, in association with a situation in which each piece of reading information is suitable;
The determination unit determines a situation of the user based on at least one of an attribute of the user and a state of the user;
The generation unit selects the reading information associated with the user's situation determined by the determination unit, and generates the reading data for outputting the characters at each of the times of the selected reading information as a sound having the tone corresponding to the voice information in the utterance information and the volume and pitch at the time.
8. A data processing device according to claim 6 or 7. The reading information storage unit further stores dictionary information that associates words included in the content to be read aloud in the reading information with meanings of the words in the reading information, in association with the reading information;
The data processing device includes: a voice information acquisition unit that acquires voice information uttered by a user who is viewing content;
a voice recognition unit that recognizes the voice information acquired by the voice information acquisition unit and acquires the contents of the user's speech,
the generation unit, when the speech content of the user acquired by the voice recognition unit is a question about the content, refers to the dictionary information and generates answer information indicating an answer to the question;
The output control unit controls to output the answer information.
A data processing device according to any one of claims 1 to 5. Further comprising a reaction information storage unit,
The data processing device according to claim 9 , wherein the voice recognition unit stores in the reaction information storage unit reaction information that associates a content of an utterance by the user with the time corresponding to a timing at which the content of the utterance was uttered by the user. A step of acquiring recitation information executed by a computer, the recitation information being time-series data indicating features of the voice of a first speaker reciting content composed of character strings stored in a recitation information storage unit, the recitation information associating a time in the playback time of the content with a character spoken at that time, a volume, and a pitch;
acquiring speech information generated by sampling a voice uttered by a second speaker and stored in a speech information storage unit, the speech information being stored in association with a plurality of characters and a tone of a voice uttered when pronouncing each of the plurality of characters;
generating reading data for outputting the characters at each of the times in the reading information as sounds having the tone color corresponding to the voice information in the utterance information and the volume and pitch at the times;
A step of controlling to output the reading data;
A data processing method comprising the steps of: acquiring, in a computer, recitation information which is time-series data indicating features of the voice of a first speaker who recites content composed of character strings stored in a recitation information storage unit, the recitation information associating a time in the playback time of the content with a character uttered at that time, a volume, and a pitch;
acquiring speech information generated by sampling a voice uttered by a second speaker and stored in a speech information storage unit, the speech information being stored in association with a plurality of characters and a tone of a voice uttered when pronouncing each of the plurality of characters;
generating reading data for outputting the characters at each of the times in the reading information as sounds having the tone color corresponding to the voice information in the utterance information and the volume and pitch at the times;
A step of controlling to output the reading data;
A program that executes the following.

Images