JP2007164732A - Computer executable program and information processing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology, which allows addition of a voice recognition function to a program comprising a user interface without changing the program. <P>SOLUTION: An information processing device is equipped with: a means for retrieving screen display definition information which includes a user interface component defined by component information for the user interface and defines a screen display form of a first part formed on a computer screen; a means for displaying a second screen part including the user interface component; a means for storing a table of utterance components, which associates the character string information specified by an utterance with the component information; a means for acquiring the character string information generated by receiving the utterance; a means for specifying a user interface component corresponding to the character strings being generated; and a means for executing a process corresponding to the user interface component. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a computer-executable program for incorporating a voice recognition function into a user interface and an information processing apparatus equipped with the program.

  Conventionally, as a technique for incorporating a voice recognition function into a user interface, for example, an application program that can be used on the web, for example, the following Patent Document 1 is known. In this technology, a web server and a voice server are provided on a network, and the web server and the voice server notify each other of the status and synchronize with each other, thereby seamlessly performing voice access and web access from a terminal on the network. We are trying to cooperate.

  However, in such a configuration, a complicated system is required for the web server and the voice server to notify each other and synchronize with each other in the network. Therefore, in order to enable simple voice access in an existing user interface, a voice recognition function may be provided in the terminal itself. Then, the conversion from the voice to the character string and the input of the converted character string to the user interface may be executed on the terminal.

  However, in order to add a speech recognition function to a user interface through a normal screen, a speech recognition engine (see, for example, Non-Patent Document 1) is installed in a terminal, and a computer program (hereinafter, referred to as a user interface) that configures the user interface. It is necessary to provide an interface portion for acquiring information recognized from the speech recognition engine in the program).

In order to provide such an interface part in a program, it is usually necessary to modify and recompile the source program. In other words, it is necessary to develop a new program incorporating an interface with a speech recognition engine. Therefore, adding a voice recognition function to a program that has already been distributed to end users without upgrading is not normally assumed in the current technology.
JP 2004-246865 A "Speech recognition engine", [online], IBM Japan, Ltd., [Search on December 12, 2005], Internet (URL: http://www-06.ibm.com/jp/voiceland/ technology / p03.html)

  The present invention has been made to solve such problems. An object of the present invention is to provide a technique capable of adding a voice recognition function to a program constituting a user interface without modifying the program.

  The present invention employs the following means in order to solve the above problems. That is, the present invention includes means for retrieving component information that defines the user interface component from screen display definition information that defines a display mode of the first screen portion configured on the computer screen, including user interface components; Information processing comprising: means for accepting input of character string information corresponding to the user interface component to be identified by utterance; and means for storing the character string information in the utterance component table in association with the component information Device.

  The present invention also includes means for retrieving screen display definition information that includes a user interface component defined by component information and defines a display mode of a first screen portion configured on a computer screen, and the user interface component Means for displaying a second screen portion including the text, means for storing a speech component table that associates the character string information specified by speech and the component information, and obtaining character string information generated by receiving the speech An information processing apparatus comprising: means; means for specifying a user interface component corresponding to the generated character string information; and processing means for executing processing according to the user interface component.

  According to the present invention, by associating the character string information specified by the utterance with the component information defining the user interface component of the first screen portion, the user interface component through the second user interface component by the utterance. It is possible to execute processing according to the above.

  The present invention may be a method in which a computer executes any one of the processes described above. Further, the present invention may be a computer-executable program that causes a computer to function as any of the above-described means. Further, the present invention may be a computer-readable recording medium recorded with such a computer-executable program.

  According to the present invention, a voice recognition function can be added to a program constituting a user interface without modifying the program.

  Hereinafter, an information system according to the best mode for carrying out the present invention (hereinafter referred to as an embodiment) will be described with reference to the drawings. The configuration of the following embodiment is an exemplification, and the present invention is not limited to the configuration of the embodiment.

<Outline of information system>
FIG. 1 shows a configuration diagram of an information system according to the present embodiment. This information system includes a server 1 that provides a service by a web application program on a web page (also referred to as a website) to a user computer (hereinafter referred to as a client 2) on a network, and a client 2 that receives the service from the server 1. Contains.

Here, the network may be a public network such as the Internet, a LAN (Local Area Network), a dedicated line, or a VPN (Virtual Private Network).
) Or the like.

The server 1 executes a web server program and provides the client 2 with a service by an application program that can be used through a web page. The server 1 transmits information described in, for example, HTML (HyperText Markup Language) or XML (eXtensible Markup Language) to the client 2. In addition, the server 1 executes a linked program from information described based on JSP (Java (registered trademark) Server Pages) or IIS (Internet Information Service). Then, the server 1 dynamically generates a web page and transmits it to the client 2.

For example, the client 2 executes a browser program as an application program. The client 2 requests the server 1 to provide information by HTML, XML, etc., and displays the provided information on the screen of the display device. Accordingly, the web page of the server 1 is displayed on the display device of the client 2, and the user interface of the application program executed on the server 1 or another computer linked with the server 1 can be used on the client 2.

  Each of the server 1 and the client 2 includes a CPU, a memory, an input / output interface, a display device, a hard disk, a communication interface with a network, a microphone that collects user's utterance contents, a speaker that outputs sound, and a removable portable medium. It has a driving device and the like. The server 1 and the client 2 realize functions as the server 1 and the client 2 by executing respective computer programs. In any case, since the components and operations of the server 1 and the client 2 are widely known, the description thereof will be omitted.

  A feature of this information system is that when a user accesses a web page displayed on the client 2 (and a user interface of an application program executed on the server 1), the user can access the web page.

  That is, the client 2 recognizes the user's utterance content and generates a character string corresponding to the utterance content. Then, the client 2 sets the character string in the user interface displayed as a web page. For example, the client 2 displays a web page corresponding to the character string, or focuses a window displaying the web page corresponding to the character string (in a state selected by a pointer such as a mouse).

  Further, the client 2 sets a character string in a component on the screen, for example, a text input field, labeled with the character string. Further, the client 2 selects an option corresponding to the character string from the pull-down menu list. In addition, the client 2 presses a button with a label corresponding to the character string. Thus, in this information system, the user can operate the graphical user interface component on the web through voice.

  The server 1 supports user interface construction based on such utterances. The server 1 has a program (hereinafter referred to as an utterance definition tool) for relating utterance contents on the client 2 to user interface parts on a web page.

  The utterance definition tool analyzes a web page specified by a user who manages the server 1 and picks up user interface components arranged on the web page. Here, the user who manages the server 1 is a user who distributes a web page to the client 2 and causes the client 2 to use an application program, for example, an application service provider.

  The utterance definition tool analyzes a definition file describing a specified web page, for example, a file such as HTML, XML, JSP, IIS, etc., and extracts the configuration of user interface components included in the web page. Then, the utterance definition tool receives audio information for selecting each extracted user interface component.

  For example, the speech tool causes the user to select a specific user interface component, a pull-down menu labeled “product selection” (accept selection). In this state, the user speaks a word that is spoken when the user wants to focus on the pull-down menu.

Then, the spoken voice information is detected by, for example, a microphone, and is converted into a character string (for example, “Shohinsen Taku”) through a voice character conversion tool (so-called voice recognition program). “Shohinsen Taku” is delivered to the utterance definition tool.

  The utterance definition tool includes information for identifying the user interface (for example, a URL indicating the location of an HTML file, tag information for displaying the pull-down menu in the HTML file), and a character string on the user interface (for example, “Product selection”) and a character string converted from the utterance content (for example, “Shohinsen Taku”) are related and stored in the database.

  The database stores utterance definition information relating utterance contents and user interface components, and various management information (referred to as various masters) for constructing utterance definition information.

  The client 2 is provided with the database from the server 1 in advance. In addition to the phonetic character conversion tool, the client 2 is installed with a program called an engine.

  The engine searches the database based on the character string delivered from the speech to character conversion tool, identifies the user interface component related to the character string, and executes processing corresponding to the user interface component. For example, the engine displays a browser window including a user interface component corresponding to the character string.

  The engine also changes the hierarchical relationship on the display between the window and other windows. For example, the engine displays a browser window including the user interface component at the top. Further, the engine selects the user interface component (positions the pointer) or sets a character string in the user interface component.

  Thus, in this information system, the utterance definition information is constructed by the utterance definition tool on the server 1 and provided from the server 1 to the client 2. In the client 2, the speech character conversion tool converts the utterance content into a character string. In addition, the user interface component to be associated is specified from the converted character string by the engine that cooperates with the phonetic character conversion tool, and processing corresponding to the user interface component is executed.

  Thus, in this information system, a user interface based on utterance can be added to a user interface provided as a web page of the server 1 without modifying the program of the server 1.

  FIG. 2 shows a flowchart showing processing on the server 1. This flowchart shows a program that is divided into three vertically long areas consisting of right and left and center, and each area is executed by the server 1. Therefore, the linkage relationship between programs is also shown in the flowchart of FIG. In FIG. 2, the left area shows the application forming the user interface. The central area shows the utterance definition tool. Moreover, the area | region of the right side shows a phonetic character conversion part. Note that the speech character conversion unit includes, for example, a commercial speech recognition program and utterance definition information obtained as a result of speech recognition.

  In the present embodiment, the voice recognition program is not limited, and any program may be used as long as it has an interface with a general application program.

  In this process, first, an utterance definition tool is activated on the server 1 (S1). Then, the operation screen of the utterance definition tool is displayed on the display device of the server 1. Further, it is assumed that the user interface (for example, the screen of the web application user interface on the browser screen) of the application program to which the voice recognition function is to be added has already been activated on the server 1.

The user drags and drops a screen corresponding to the user interface onto the operation screen of the utterance definition tool (S2). Then, the utterance definition tool defines definition information (corresponding to the screen display definition information of the present invention) that defines the user interface, for example, identification information (URL, Uniform Resource Locator, etc.) indicating the location of the HTML file, XML file, etc. ) To get. As is well known, this definition information includes the hard disk of the server 1,
Alternatively, it is stored in a storage device of another computer connected to the server 1 through a network.

  Then, the information defining the user interface is analyzed (corresponding to means for retrieving the component information by the CPU of the server 1 that executes this process), and the components constituting the user interface (corresponding to the user interface components of the present invention) For example, definition information (equivalent to the component information of the present invention) such as a label, an input field, a pull-down menu list, and a push button label is acquired. Then, the utterance definition tool generates a screen showing the user interface (S3).

  Next, the utterance definition tool sets utterance readings for each field on the user interface or each user interface component according to the user's operation (S4). That is, the definition tool selects (focuses) a user interface component to be set for reading in accordance with an operation through a user input device (a pointing device such as a keyboard and a mouse). In this state, the voice input through the microphone is converted into a character string by the voice character conversion tool. The utterance definition tool acquires the character string converted through the application interface of the phonetic character conversion tool (means that the CPU of the server 1 executing this process receives input of character string information corresponding to the user interface component of the present invention) Equivalent to However, instead of inputting voice input through a microphone, an utterance character string may be manually input using a keyboard or a pointing device.

  Furthermore, attributes for the user interface parts are set as necessary. Then, information identifying the user interface component (for example, an HTML file tag), a character string based on speech, attribute information, and the like are paired and stored in the database of the speech character conversion unit (S5). In FIG. 2, this database is shown as a dictionary file and a profile.

  Next, the utterance definition tool creates unique information for operating the target application and stores it in the database (S6). That is, the character string converted in the process of S4, the attribute set in S5, and the like are stored in the database in association with the selected user interface component (the database is stored in the speech component table of the present invention). Equivalent to the means).

FIG. 3 is a flowchart showing processing on the client 2 side using the user interface to which the voice recognition function is added according to the setting of FIG. This flowchart shows a program that is divided into three vertically long areas including right and left and the center, and each area is executed by the client 2. Therefore, the linkage relationship between programs is also shown in the flowchart of FIG. In FIG. 3, the left area shows the application forming the user interface. The engine in the center area controls the voice recognition function. Moreover, the area | region of the right side shows a phonetic character conversion part.

  The database information (dictionary file and profile, corresponding to the speech component table of the present invention) set in the processing of S4-S6 in FIG. 2 is downloaded to the client 2 in advance (client 2 that executes this processing). The CPU corresponds to means for receiving information on the utterance component table from another information processing apparatus, and the CPU of the server 1 that provides information on the database provides information on the utterance component table of the present invention. Equivalent to The database information may be downloaded from the server 1 every time the client 2 accesses the server 1. Further, when the client 2 accesses the server 1, it may be downloaded from the server 1 when it is detected that the database information does not exist in the client 2. Further, when the client 2 accesses the server 1, it may be downloaded from the server 1 when it is detected that the database information is updated. The hard disk of the client 2 that stores this database corresponds to means for storing the speech component table of the present invention.

  The client 2 is installed with a general browser and a voice character conversion tool. Furthermore, an engine that receives a character string that is a speech recognition result of the speech-to-speech conversion tool and searches a database (dictionary file and profile) is installed. When the server 1 detects that the engine is not installed in the client 2, information about the engine and the database may be downloaded. The CPU of the server 1 that executes such processing corresponds to means for distributing the computer program of the present invention.

  With such installation completed, the engine is first started (S11). When the user interface of the application program (corresponding to the first screen portion of the present invention) is started in the state where the engine is started (S12A), the engine defines definition information (this book) that defines the user interface of the application program. For example, identification information (URL or the like) indicating the location of an HTML file, XML file, or the like is acquired. For example, the engine detects the URL every time the browser switches the display destination URL. Then, the engine determines whether or not the URL matches the URL set in the database (dictionary file and profile).

  If the identification information is set in the database, the engine determines that the user interface defined by the identification information is the target of speech recognition. In that case, the engine reads the definition information (corresponding to the component information of the present invention) that defines the user interface component from the storage location indicated by the identification information, and superimposes it on the original user interface (the present invention). (Corresponding to the second screen portion) is generated (S13).

  Therefore, in this state, the pseudo screen is displayed on the display device so as to overlap the user interface of the application being executed by the user (the client 2 that controls this display displays the second screen portion of the present invention). Equivalent to the means to do). However, when viewed from the user, the user interface of the original application program appears to be displayed. In the information system of the present embodiment, the steps up to this stage are referred to as preparation work on the client 2 side.

  Thus, the user performs voice input in a state where the preparation work is completed. For example, the user speaks into the microphone (S14). Then, the voice character conversion tool converts the voice into an ASCII code (S15). Further, the voice character conversion tool generates a character string (text) from the ASCII code.

  Then, the voice-recognized character string is delivered to the engine (S17, which corresponds to a means for acquiring character string information generated by the CPU of the client 2 executing this process by receiving an utterance). The engine searches the database (dictionary file and profile) based on the character string delivered from the voice recognition tool (the CPU of the client 2 that executes this process identifies the user interface component corresponding to the generated character string information) Equivalent to the means to do).

  When the user interface component associated with the character string is defined in the database, the engine executes a process corresponding to the user interface component (S19, the CPU of the client 2 executing this process responds to the user interface component). Corresponds to a processing means for executing the processing).

  For example, when the user interface component is a window constituting a part of the screen, the engine displays the window. Further, the engine displays the window in the highest hierarchy among the hierarchy composed of a plurality of windows. When the user interface component is a text input field, the engine sets a character string in the input field. When the user interface component is a pull-down menu title, the engine displays a list (option) of the pull-down menu. When the user interface component is one of the elements (options) included in the pull-down menu list, the engine selects the option. If the user interface component is a push button label, the engine presses the push button. In this way, the engine controls to repeat the processes from S14 to S19.

<Data structure>
Hereinafter, the data structure of the database (dictionary file and profile) used by the information system of this embodiment will be described. In the present embodiment, the database is composed of a plurality of tables, and is stored in a storage device such as a hard disk, for example.

  FIG. 4 is a diagram showing a configuration of a table called a URL master. The URL master records the storage location of the definition information that defines the web page displayed on the browser. That is, the URL master stores information indicating the user interface that is the processing target of the engine.

  As shown in FIG. 4, the URL master stores a field (L_ID) of information for identifying each row of the table, a field (S_URL) for storing the URL, a field (S_TITLE) for storing the title of the web page, and the URL in the database. The registered date (D_REGISTER), the date of updating the URL information (D_UPDATE), and the like.

  FIG. 5 is a diagram showing the configuration of the field master. The field master defines user interface components on each web page. The URL master is a table generated as a result of analyzing user interface component definition information in the server 1.

  As shown in FIG. 5, the L_ID of the URL master is specified at the top of each line of the field master. Thus, each row in the field master is associated with any row in the URL master.

In addition, the field master is a unit price when the cursor movement word (field name of the user interface part, S_TITLE), data type (S_FIELD_TYPE, S_TAG_TYPE), attribute (S_FIELD_INFO), and the value to be set in the field is a price. And field format (S_UNIT, S_FORMAT), values extracted from the field (S_DEFAULTFORM, S_WRITEFORM), and the like.

Of these, S_DEFAULTFORM is a character string for notation. In the interface portion with the voice character conversion tool, if there is a single-byte / double-byte space in the character string of the user interface component, recognition may not be possible. Therefore, S_WRITETENFORM is obtained by deleting half-width and full-width spaces from the notation character string. Note that values extracted from the field (S_DEFAULTFORM and S_WRITEFORM) are input characters when the user interface component is a text input field. Blank unless the column is fixed, the value of the element if the user interface component is a pull-down menu option, and the label if the user interface component is a push button. When the user interface component is a title of a window or pull-down menu, the title character string is used. Values extracted from this field (S_DEFAULTFORM and S_WRITEFORM) are called values associated with user interface components.

  FIG. 6 is a diagram showing the configuration of the utterance master. The utterance master defines a reading (S_SPOKENFORM) corresponding to a value (S_WRITEFORM) associated with each user interface component. For example, “Coffee” is associated with “Coffee”, and “Shohinsen Taku” is associated with “Product selection”.

  A plurality of readings may be set for one of the values associated with each user interface component. For example, for the value of “Shiryo Plasma TV 50 inch YPT-50”, “Gojyu Chun Pizuma”, “Gyo Ji Chun Pizuma Telebi”, “Yonbushi Pizuma Temare Gyujo” Inch "etc. are set. For example, when the information system of this embodiment adds a voice recognition function to a user interface for Internet shopping, an end of using a service for the product name “Shiryo Plasma TV 50 inch YPT-50”. It is assumed that the user speaks various readings. The utterance master may be set with many possible readings for the value (S_WRITEFORM).

  Further, as shown in FIG. 6, each row is provided with a field (D_REGISTER) for reading setting date.

  FIG. 7 is a diagram showing a configuration of a table called a mobile word master. The movement word master is a table that defines values used as movement words among values associated with user interface components. When the character string converted from the utterance result is registered as a value (S_WRITEFORM) in the mobile word master, the pointer of the pointing device moves to the user interface component corresponding to the value. That is, the user interface component is selected (focused). As shown in FIG. 7, the mobile word master stores a value (S_WRITEFORM), a reading (S_SPOKENFORM), and a data registration date (D_REGISTER) in pairs.

  FIG. 8 is a diagram showing a reserved word master. The reserved word master is a table in which the system defines a relationship between a value (S_WRITEFORM) reserved in advance before being used in the server 1 and a reading (S_SPOKENFORM).

  In the reserved word master, for example, readings are defined for character strings that are frequently used and are almost determined how to read, such as telephone numbers, FAX numbers, and local station numbers.

  For example, when a character string “today” is input, if it exists in the reserved word master, the date of the client 2 is acquired and the date is input. For example, when the date of the client 2 is December 12, 2005 and utters “Tomorrow”, the date of today is acquired, added by one day, and “2005/12/13” is input. (Between / is based on the definition of the attribute master in FIG. 10).

FIG. 9 is a diagram illustrating a unit master. The table of FIG. 9 represents a list of units when defining a character string. Specifically, S_DISPLAY represents a unit of a character string, and S_ATTR is linked with S_ATTR in FIG. 10 and represents a format attribute of the character string.
FIG. 10 shows an attribute master. The unit of the character string is defined by the unit master of FIG. 9, and the display attribute of the character string is defined by the attribute master of FIG. The attribute master in FIG. 10 is linked to the field master in FIG. 5 by S_FORMAT. That is, the display format of the character string displayed on each user interface component is determined by searching the attribute master of FIG. 10 based on the field master S_FORMAT.

<Example>
An embodiment according to the information system will be described with reference to FIGS. In this embodiment, an example of adding a voice recognition function to an Internet shopping site will be described.

FIG. 11 is a diagram illustrating an operation of associating a voice input with a user interface component (also referred to as a field) in the server 1. FIG. 11 shows an operation screen (also referred to as a window) 10 of a definition tool (Voice Moderato Translator (trademark)) executed on the server 1.
It is shown. The operation screen 10 has a web page display unit 11 in a substantially left half area of the screen. The web page display unit 11 displays a user interface for adding a voice recognition function, for example, a web page of a web application. Server 1 user
However, for example, by dragging and dropping a web page to which a voice recognition function is to be added to the web page display unit 11, the web page is displayed.

  The substantially right half of the screen is a setting area for analysis results and speech information of the web page displayed on the web page display unit 11. That is, the operation screen 10 has respective areas of an object hierarchy display unit 12, an audio target URL display unit 13, a recognition word registration unit 14, and a reading setting unit 16.

  The object hierarchy display unit 12 analyzes a web page designated as a processing target, that is, a web page displayed on the web page display unit 11, and a user interface component on the web page (also referred to as an object in FIG. 11). Displays the relationship in a hierarchical manner. Generally, a user interface is configured by a hierarchical combination of user interface components. In addition, the user interface component is configured by a hierarchical combination of a plurality of lower components. The object hierarchy display part 12 shows the hierarchical structure of the web page to be processed.

  For example, the user interface is defined by defining a window area called a form at the top, and arranging a text box (text input field), a pull-down menu, a check button, and the like on the form. The text box generally includes a label indicating a title and a character string input field. The pull-down menu is composed of a label indicating a title, a list indicating options, and an arrangement of elements constituting the list.

  The voice-target URL display unit 13 displays a URL indicating a web page to which a voice recognition function is added together with the title of the web page. This URL is acquired by the definition tool, for example, when the user drags and drops a web page onto the web page display unit 11. The title is extracted from a definition file (HTML, XML, etc.) indicated by the URL.

  The recognition word registration unit 14 registers a character string to be recognized by utterance in association with the user interface component to be processed. For example, in FIG. 11, a pull-down menu titled “Product Selection” is operated at a location indicated by the title “Product” on the web page.

  At this time, the object hierarchy display unit 12 indicates that the pull-down menu “product selection” is being processed in color (looks black in FIG. 11), and the recognition word registration unit 14 has the title “product”. It is indicated that the data type is “selection”, that is, a user interface component including options.

  As shown in FIG. 11, the object hierarchy display unit 12 has a moving word registration button 15. When the user presses the moving word registration button, a moving word registration screen is displayed. The moving word registration button 15 is a button for setting a character string displayed in the title as a moving word.

  FIG. 12 shows a moving word registration screen. On the post-movement registration screen, a character string set for the movement word and its reading are displayed as a set. For example, if the character string “Product selection” (reading “Syohinsen taku”) is registered as a moving word, the sound “Shohinsen taku” will be displayed while the web page to be processed is displayed. When uttered, the pull-down menu with the title “product selection” is in focus.

  In FIG. 12, the reading setting unit 15 defines a character string (input character) to be set for the user interface component to be processed or used when operating the user interface component. Here, for example, an utterance voice “shohinsen taku” for the character string “product selection” which is the title of the pull-down menu “product selection” is defined. In addition, “coffee” for “coffee”, “daiz” for “soybean”, and the like, which are pull-down menu options, are set. With this setting, when “Shohinsentaku” is spoken, the pull-down menu titled “Product selection” is focused, and when “Kohi” is spoken in that state, “Coffee” This choice is selected. As described above, a plurality of readings may be set for the input character. After the setting, when the user presses the update button 17, the setting content is stored in the database together with the URL.

  The user executes the above settings for each user interface component of the web page. The user interface component set as described above is a speech recognition processing target.

  FIG. 13 shows an example of processing on the client 2 of an end user who uses Internet shopping and an application program of the client 2 related to this processing.

  In the client 2, a browser 20, an engine 21, and a voice character conversion tool 22 are already installed. Further, it is assumed that the browser 20 and the engine 21 are being executed on the client 2.

  In FIG. 13, the browser 20 displays an Internet shopping site. The database defining the voice input of the Internet shopping site is downloaded when the client 2 first accesses the Internet shopping site. For example, the latest database may be downloaded from the server 1 when the engine 21 is installed.

  When the engine 21 is activated, the engine 21 monitors a URL indicating a web page displayed by the browser 20. Then, the engine 21 determines whether the URL displayed by the browser 20 is registered in the URL master of the database. If the URL is registered in the URL master of the database, the engine 21 determines that the URL is a target for voice recognition processing. Then, the engine 21 reads the definition file (HTML, XML, etc.) indicated by the URL, generates a pseudo screen similar to the web page displayed by the browser 20, and displays the pseudo screen superimposed on the display of the browser 20. Therefore, when viewed from the end user, it looks as if the web page is displayed by the browser 20.

  In this state, when the end user speaks a voice, the voice is taken into the voice character conversion tool 22 executed by the client 2 as voice data through the microphone and the input / output interface. The phonetic character conversion tool 22 performs phoneme analysis on the phonetic data and converts the phonetic data into an ASCII code string. Further, the phonetic character conversion tool 22 searches the dictionary, decomposes the ASCII code string into words (or morphemes), and collates with the dictionary. Then, the phonetic character conversion tool 22 generates a text that is a sequence of words (or morphemes) and passes it to the engine 21 through an argument.

  The engine 21 searches a reserved word master in the database from words (or morphemes) in the text, and determines an input character corresponding to the spoken voice and a user interface component to which the input character is to be input. Alternatively, the mobile word master is searched, and the user interface component to be selected is determined based on the spoken voice. Alternatively, the utterance master and the field master are searched to determine an input character corresponding to the spoken voice and a user interface component to which the input character is to be input. Then, the input character is set in the corresponding user interface component, and is displayed on the pseudo screen of the web page on the display device (display).

  FIG. 14 shows an example of a web page set by voice input. For example, when the end user displays the web page of FIG. 14 with the browser while the engine 21 is running, the engine 21 detects that the URL is registered in the URL master of the database. The engine 21 reads the configuration of the web page based on the URL, and displays the web page pseudo screen superimposed on the browser.

  Then, for example, when the end user speaks “Let's do nothing”, the “order contents” portion in the pseudo screen is focused. Here, for example, when “Uketsubango is Ichini-sanshi” is uttered, it is converted into “Uketsu-bangogo”, “ha”, and “Ichini-sanshi” by the phonetic character conversion tool. The engine 21 searches for the utterance master by “invoke bango”, and converts “invoke bango” into “acceptance number”. Further, the engine 21 searches the field master, determines the field “reception number”, and obtains information for identifying the field (values of I_FIELD and I_VALUE in FIG. 5). Further, the engine 21 determines “1234” based on “Ichi ni sanshi” after “ha”, and sets “1234” in the field of “reception number”.

Also, for example, when the end user speaks “Kohi”, it is converted into “Kohi” by the phonetic character conversion tool. The engine 21 refers to the utterance master (FIG. 6) on the basis of “Kohi” and acquires the character string “coffee”. Next, the engine 21 refers to the field master (FIG. 5) corresponding to the web page (linked to the URL master by L_ID) based on “coffee”, and sets the user interface component (“coffee”) ( 5 and the URL of the web page on which the user interface component is displayed (the URL in the URL master line in FIG. 4 determined by the L_ID value in FIG. 5).

  FIG. 14 is a diagram illustrating an example of inputting a zip code and a telephone number. As for postal codes, for example, when an end user speaks “Yubinbangouichi-san no Yongorokuna”, the text-to-speech converter uses the “Yubinbango”, “Ichini-san”, “No”, “Yongo-kunna” Is converted to. The engine 21 searches the post-reservation master (or utterance master) by “Yubinbango”, and detects “zip code”. Further, the engine 21 searches the attribute of the field master and determines the field “zip code”. If there are a plurality of fields “zip code”, the current position of the pseudo page of the web page is acquired from a file constituting the web page such as an HTML file or an XML file, and the first field “zip code” after the position is acquired. To decide.

  Also, the later “Ichini-san”, “No”, and “Yongorokuna” are acquired, characters other than numbers are excluded, and a list of numbers is generated. The format is converted based on the attribute acquired by the field master, and “123-4567” is set in the field “zip code”.

  As for phone numbers, for example, when an end user utters “Denbanbangoichi no Sanyokuronoku no Nanachi Kyusero”, the phonetic conversion tool will be used to create “Denbanbango” “Ichini” “No”. It is converted to “San Yongoroku”, “No”, and “Nana Hachikyuzuro”. The engine 21 searches the post-reservation master (or utterance master) by “Denwa Bango”, and converts it to “phone number”. Further, the engine 21 searches the field master attribute to determine the field “telephone number”. If there are a plurality of fields “telephone numbers”, the current position of the pseudo screen of the web page is acquired, and the first field “phone number” after that position is determined. In addition, the subsequent “ichini”, “no”, “sanyongoroku”, “no”, and “nanahachikyuzero” are acquired, and characters other than numbers are excluded to generate an enumeration of numbers. The format is converted based on the attribute acquired by the field master, and “12-3456-7890” is set in the field “telephone number”.

<Effect of embodiment>
As described above, according to the information system of the present embodiment, the utterance definition tool of the server 1 analyzes the web page constituting the user interface and extracts the hierarchical structure of the user interface parts constituting the web page. Then, the utterance definition tool receives input characters of reading corresponding to each user interface component, and the URL to which each user interface component belongs and identification information for identifying each user interface component (values such as L_ID, I_FIELD, and I_VALUE in FIG. 5). ) And store it in the database.

On the other hand, the client 2 used by the end user downloads from the server 1 a database in which input characters by utterances are defined for the user interface components of each web page. Then, the engine 21 executed by the client 2 generates a pseudo screen by superimposing the web page displayed by the browser, and displays the pseudo screen. In this state, when the end user utters a voice, a corresponding user interface component is determined from text including a word (or morpheme) obtained through the phonetic character conversion tool, and processing corresponding to the user interface component is executed. . For example, when the user interface component is the title of a text input field or the title of a pull-down menu, the user interface component is in a focused state. In addition, the corresponding part of the text is set in the text input field. Also, if the corresponding part of the text is a pull-down menu option, that option is selected.

  Thus, according to this information system, a definition file (HTML, XML, JSP, IIS, etc.) that defines a web page and an application program that configures the web page can be changed to a user interface on the web. A voice recognition function can be added.

<Modification>
In the above-described embodiment, the example in which the voice recognition function is added mainly in the information system including the server 1 and the client 2 on the network is shown. However, the implementation of the present invention is not limited to such a configuration. For example, both the utterance definition tool and the engine 21 may be installed in a stand-alone computer. That is, in a stand-alone computer, a database constructed by an utterance definition tool may be used, and voice input may be performed on a web page displayed on the computer.

  Further, the utterance definition tool and the engine 21 may be distributed to end users as an integrated program. In that case, the end user may register a character string based on an utterance in a database in association with a user interface component on a web page such as a web application to be used. Then, using the database definition set by the end user himself / herself, voice input may be performed on the web page.

  Moreover, in the said embodiment, the example which adds a voice input function to the user interface displayed on a web page was shown. However, implementation of the present invention is not limited to user interface components on web pages. That is, even if the file is not an HTML file or an XML file, the structure of the user interface part on the screen, the character string indicating the title of the user interface part, and the character string to be input are defined as in the definition tool and the engine 21. When an external program (a program other than an application that is a target of voice input) can be specified, the present invention can be implemented.

  For example, in a stand-alone computer, the configuration of the user interface configured on the document creation program may be analyzed from the macro definition information of the document creation program such as a word processor, a spreadsheet program, or a presentation tool.

  For example, the utterance definition tool may read the macro definition information and construct a database as in the above embodiment. Then, the engine 21 that receives the provision of the database monitors the activation of the document creation program, and when the document creation program is activated, a pseudo process of the activated document creation program may be executed. The user interface screen generated by the pseudo process may be displayed so as to be superimposed on the original user interface of the document creation program. After such preparation, the voice input result may be set in the user interface of the pseudo process as in the above embodiment.

  Further, for example, when the application program includes a user interface program and a processing program, and the user interface program and the processing program communicate with each other through inter-process communication, in a stand-alone environment, the same as in the above-described embodiment. A pseudo screen can be generated by superimposing on the screen of the user interface program, and a voice input function can be added. The voice input result may be converted into a character string, set on the screen of the user interface program of the pseudo process, and passed to the processing program by inter-process communication.

  In addition, for example, in an application in which the configuration of parts on a window is defined as a resource file outside a binary program, a user interface part can be extracted and a voice input function can be added by analyzing the resource file. it can.

  Computer-executable programs such as the engine 21 and the speech definition tool may be installed in the client 2 or the server 1 through a network. These programs may be stored in a computer-readable recording medium (for example, DVD, CD-ROM, removable disk, etc.) and distributed. Alternatively, computer-executable programs such as the engine 21 and the utterance definition tool may be stored in a server that provides application services on the network, and only the functions of the program may be provided to the server 1 or the client 2.

It is a block diagram of the information system which concerns on one embodiment of this invention. It is a flowchart which shows the process on a server. It is a flowchart which shows the process by the side of the client using the user interface to which the speech recognition function was added. It is a figure which shows the structure of URL master. It is a figure which shows the structure of a field master. It is a figure which shows the structure of an utterance master. It is a figure which shows the structure of a movement word master. It is a figure which shows a reserved word master. It is a figure which shows a unit master. It is a figure which shows an attribute master. It is a figure which shows operation which matches audio | voice input with respect to user interface components in a server. It is a figure which shows a movement word registration screen. It is a figure which shows the example of a process on the client of the end user using Internet shopping. It is a figure which shows the example of the web page set by audio | voice input. It is a figure which shows the input example of a postal code and a telephone number.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Server 2 Client 10 Operation screen 11 Web page display part 12 Object hierarchy display part 13 Voice object URL display part 14 Recognition word registration part 15 Moving word registration button 16 Reading setting part 20 Browser 21 Engine 22 Spoken character conversion tool

Claims (14)

Computer
Means for retrieving component information defining the user interface component from screen display definition information defining a display mode of a first screen portion configured on a computer screen including a user interface component;
Means for receiving input of character string information corresponding to the user interface component, which is a target specified by utterance;
Means for storing the character string information in the utterance component table in association with the component information;
A computer-executable program that functions as a computer.   The computer-executable program according to claim 1, further causing the computer to function as means for receiving a designation for processing the first screen portion. Means for displaying a second screen portion including the user interface component;
The computer execution according to claim 1 or 2, further comprising: a computer functioning as processing means for identifying a user interface component corresponding to character string information converted by receiving an utterance and executing processing according to the user interface component. Possible program.   The processing includes displaying a screen portion including a user interface component corresponding to the character string information, changing a hierarchical relationship on the display between the screen portion and another screen portion, and changing the user interface component corresponding to the character string information. The computer-executable program according to claim 3, wherein the computer-executable program is selection or setting of the character string information in the user interface component.   The computer further functions as means for providing information of the speech component table to another computer having means for displaying the first and second screen portions based on the screen display definition information via communication means. The computer-executable program according to any one of claims 2 to 4.   The character string information converted from the utterance is sent to the user interface component to another computer having means for displaying the first and second screen parts based on the screen display definition information via communication means. 6. The computer-executable program according to claim 2, further causing the computer to function as means for distributing the computer program including means for associating.   The computer-executable program according to claim 5 or 6, further causing the computer to function as means for receiving character string information converted from an utterance and set in the user interface component from the other computer. Computer
Retrieving part information defining the user interface part from screen display definition information defining a display mode of a first screen part configured on a computer screen including a user interface part;
Receiving character string information corresponding to the user interface component, which is a target specified by utterance;
Storing the character string information in an utterance component table in association with the component information. Means for retrieving component information defining the user interface component from screen display definition information defining a display mode of a first screen portion configured on a computer screen including a user interface component;
Means for receiving input of character string information corresponding to the user interface component, which is a target specified by utterance;
An information processing apparatus comprising: means for storing the character string information in an utterance component table in association with the component information. On the computer,
Means for retrieving screen display definition information that includes a user interface component defined by the component information and defines a display mode of the first screen portion configured on the computer screen;
Means for displaying a second screen portion including the user interface component;
Means for storing an utterance component table that associates character string information specified by utterance with the component information;
Means for acquiring character string information generated by accepting an utterance;
Means for identifying a user interface component corresponding to the generated character string information;
A computer-executable program that functions as processing means for executing processing according to the user interface component.   The computer-executable program according to claim 10, wherein the second screen portion is configured to overlap with the first screen portion displayed according to the screen display definition information.   The processing includes displaying a screen portion including a user interface component corresponding to the character string information, changing a hierarchical relationship on the display between the screen portion and another screen portion, and changing the user interface component corresponding to the character string information. The computer-executable program according to claim 10 or 11, which is a selection or setting of the character string information in the user interface component.   The computer-executable program according to any one of claims 10 to 12, further causing the computer to function as means for receiving provision of information of the utterance part table from another information processing apparatus via communication means. Means for retrieving screen display definition information that includes a user interface component defined by the component information and defines a display mode of the first screen portion configured on the computer screen;
Means for displaying a second screen portion including the user interface component;
Means for storing an utterance component table that associates character string information specified by utterance with the component information;
Means for acquiring character string information generated by accepting an utterance;
Means for identifying a user interface component corresponding to the generated character string information;
An information processing apparatus comprising: processing means for executing processing according to the user interface component.

Images