JP5749229B2 - Gesture management program and information processing apparatus - Google Patents

Description

  The present invention relates to a gesture management program and an information processing apparatus, and more particularly to a technique for receiving an operation instruction from an operator.

  Today, an application is operated on an information processing device such as a server, and the server is accessed from a remote communication terminal device such as a mobile device via a network such as the Internet. A technique for displaying and operating the application by displaying the operation content of the application of the server is known.

  Many of the applications used as described above are assumed to be used by a personal computer. Therefore, even when an operator operates the application using a mobile device such as a mobile phone, the same operation as that performed on a personal computer is performed. It is inconvenient because it may require operation. For this reason, a technique for improving the operability when using the above-described application in a mobile device has been proposed. For example, as disclosed in Patent Document 1 below, a technique has been proposed in which information that can be input using a key included in a mobile device is converted into information corresponding to an application. Further, in Patent Document 2, it is possible to input the same input information as when the predetermined key operation is performed by the operation using the touch panel by associating the operation performed on the touch panel with the predetermined operation using the hard key provided in the device. A technology has been proposed.

JP 2009-80562 A JP 2010-86452 A

  However, the techniques disclosed in Patent Documents 1 and 2 are based on the premise that the application has a function of accepting an operation instruction based on an operation with a key included in the device. For this reason, these techniques only improve the operability only in the operation range where the key operation is performed. In today's mobile devices, in addition to such key operations, it is possible to accept instructions to switch the display screen based on operations such as the operator sliding a finger on the display panel. Often used to. Therefore, regardless of the key operation as described above, if the operator can operate the application by an operation that is normally performed on the mobile device, the operability of the operator for the application can be further improved. Further, if such an operation becomes possible, even if the application is installed and used in the communication terminal device, the application can be operated by an operation familiar to the operator. It should be noted that correcting the application side requires a great deal of labor in order to enable input of instructions by operations that are not originally supported.

  The present invention has been made in order to solve the above-described problem. When an application is used in an information processing apparatus such as a mobile device, the application can be performed by an operation not supported by the application without correcting the application side. For example, the operator can operate the application by an operation familiar to the information processing apparatus.

According to the first aspect of the present invention, there is provided an information processing apparatus that obtains display data related to an application from another information processing apparatus that operates the application via a network and displays the display data on the display unit. The
A correspondence storage unit that stores a gesture associated in advance for each instruction reception image included in the display data;
A gesture detection unit for detecting an operator's gesture performed on the touch panel included in the display unit;
An image reading unit for reading an instruction receiving image associated with the gesture detected by the gesture detecting unit from the associated storage unit;
A determination unit that determines whether or not an instruction reception image read by the image reading unit exists in the acquired display data;
An information processing apparatus as a transmission control section that causes the communication section to transmit an operation instruction associated with an instruction receiving image determined to be present in the display data by the determination section to the other information processing apparatus Is a gesture management program that allows

The invention according to claim 4 includes a display unit,
A communication unit that acquires display data related to the application from another information processing apparatus that operates the application via a network;
A display control unit that causes the display unit to display the display data acquired by the communication unit;
A correspondence storage unit that stores a gesture associated in advance for each instruction reception image included in the display data;
A gesture detection unit for detecting an operator's gesture performed on the touch panel included in the display unit;
An image reading unit that reads an instruction receiving image associated with the gesture detected by the gesture detecting unit from the associated storage unit;
A determination unit that determines whether or not an instruction reception image read by the image reading unit exists in the acquired display data;
Information including: a transmission control unit that causes the communication unit to transmit an operation instruction associated with an instruction receiving image determined to be present in the display data by the determination unit to the other information processing apparatus. It is a processing device.

  In these inventions, when the operator performs a gesture stored in the associated storage unit on the touch panel of the display unit, the gesture detection unit detects the gesture, and the image reading unit corresponds to the gesture. The attached instruction reception image is read out. Then, the determination unit displays the application in which the read instruction reception image (for example, an image displayed as a button for receiving an operation instruction from the operator by the touch panel function) is displayed on the display unit. If it is determined that the data is present in the data, the transmission control unit causes the communication unit to transmit an operation instruction associated with the instruction reception image to another information processing apparatus. For this reason, even if an instruction input method that is not supported by the application, such as a gesture that an operator normally performs on a mobile device, if the gesture is stored in the associated storage unit, the application can be performed using the gesture. And the operability of the operator for the application can be further improved.

  Further, for example, even when an application downloaded from the server or the like via the Internet is installed and used in a communication terminal device such as a mobile device, the operation of the operator for the application is similarly performed as described above. The property can be further improved.

  Therefore, according to the present invention, when an application is used in an information processing apparatus such as a communication terminal device, the application can be operated even if the application is an unsupported operation without correcting the application side. For example, it becomes possible for the operator to operate the application by an operation familiar with the information processing apparatus.

The invention according to claim 2 is the gesture management program according to claim 1, wherein a gesture storage unit that stores a predetermined gesture;
Among the gestures stored in the gesture storage unit, a gesture reception unit that receives a gesture specified by an operator;
An instruction receiving image receiving unit that receives an instruction receiving image specified by an operator among the instruction receiving images present in the display data;
An information processing apparatus is further provided as an association management unit that associates the gesture received by the gesture reception unit with the instruction reception image received by the instruction reception image reception unit and stores the association in the association storage unit. The gesture management program according to claim 1, wherein the gesture management program is made to function.

  According to the present invention, the operator selects a desired gesture from the gesture storage unit, inputs the designation of the gesture to the gesture reception unit, and further displays the instruction reception image present in the display data. Among these, by inputting the designation of a desired instruction receiving image into the instruction receiving image receiving unit, the correspondence management unit stores the correspondence between the operator desired gesture and the instruction receiving image in the corresponding storage unit. . Therefore, according to the present invention, the operator can input the operation instruction corresponding to the instruction reception image in the application and operate the application with the selected desired gesture.

The invention according to claim 3 is the gesture management program according to claim 1, and receives an instruction receiving image designated by an operator among the instruction receiving images existing in the display data. An image receiving unit for receiving instructions;
An information processing apparatus is further provided as an association management unit that associates the gesture detected by the gesture detection unit with the instruction reception image received by the instruction reception image reception unit and stores the association in the association storage unit. It is to function.

  In this invention, when the operator performs a desired gesture on the touch panel of the display unit, the gesture is detected by the gesture detection unit, and the desired gesture performed by the operator and the instruction reception image by the association management unit Is stored in the correspondence storage unit. Therefore, according to the present invention, the operator can input the operation instruction corresponding to the instruction reception image in the application and operate the application by a desired gesture performed by the operator. Become.

  According to the present invention, when an application is used in an information processing apparatus such as a mobile device, the application can be operated by an operation not supported by the application without modifying the application side. The application can be operated by an operation familiar to the processing apparatus.

It is a figure which shows the outline of the communication system provided with the server and the communication terminal device. It is a figure which shows schematic structure of a server. It is a block diagram which shows schematic structure of a communication terminal device. It is a flowchart which shows the process which receives the input of the operation instruction with respect to the application which the display data acquired from the server shows. It is a figure which shows an example of the display screen of the display part in a communication terminal device. It is a block diagram which shows schematic structure of the communication terminal device which performs the gesture registration process which concerns on 1st Embodiment. It is a flowchart which shows 1st Embodiment of the process which registers the gesture in a communication terminal device. It is a figure which shows an example of the display screen of the display part in a communication terminal device. It is a block diagram which shows schematic structure of the communication terminal device which performs the gesture registration process which concerns on 2nd Embodiment. It is a flowchart which shows 2nd Embodiment of the process which registers the gesture in a communication terminal device. It is a figure which shows an example of the display screen of the display part in a communication terminal device.

  Hereinafter, a gesture management program and an information processing apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram schematically illustrating a communication system including a server and a communication terminal device.

  The communication system 10 includes a server 2 and a communication terminal device 3. The server 2 stores one or more applications in a storage device such as a built-in HDD (Hard disk drive). The server (an example of another information processing device in the claims) 2 has a control unit that operates according to the application, and communicates with the communication terminal device 3 via a network such as the Internet by the communication unit. The display data indicating the operation content of the application is transmitted to the communication terminal device 3.

  The communication terminal device 3 is a mobile device such as a smartphone or a mobile phone, for example, and acquires display data of the application from the server 2 by communicating with the server 2 via the network, and the display image is displayed by a browser. Is displayed on the display. The communication terminal device 3 receives an instruction for the displayed application from the operator by a touch panel function or key operation, and transmits the received instruction to the server 2 from a browser or the like. As a result, the application running on the server 2 can be operated from the communication terminal device 3. The communication terminal device 3 is an example of an information processing device in the claims (not “another information processing device” in the claims).

  As shown in FIG. 1, the plurality of communication terminal devices 3 can similarly display and operate applications running on the server 2 by accessing the server 2. Yes.

  Next, the configuration of the server 2 will be described. FIG. 2 is a diagram showing a schematic configuration of the server 2.

  The server 2 includes a control unit 20, a communication interface 21, and a storage device 22.

  The control unit 20 includes a CPU, a ROM, a RAM, and the like, and functions as the control unit 201 and the communication unit 202 by the operation of the CPU according to an operation program (not shown) stored in the storage device 22.

  The control unit 201 governs overall operation control of the server 2. The communication unit 202 performs processing necessary for communication with the communication terminal device 3 and the communication terminal device 3 via the Internet.

  The communication interface 21 includes a communication module and transmits / receives various data to / from the communication terminal device 3 via the Internet.

  The storage device 22 includes an HDD or the like, and stores the above-described operation program and one or more applications.

  The control unit 201 operates according to the application stored in the storage device 22, and the communication unit 202 causes the communication interface 21 to transmit display data indicating the operation content of the application to the communication terminal device 3.

  The communication interface 21 transmits the display data to the communication terminal device 3 via the Internet, while receiving an instruction from the communication terminal device 3 for the application being operated, and the like. Output to the unit 201. The control unit 201 advances the operation according to the application based on the instruction.

  Next, the configuration of the communication terminal device 3 will be described. FIG. 3 is a block diagram illustrating a schematic configuration of the communication terminal device 3.

  The communication terminal device 3 includes a control unit 300, a ROM 312, a RAM 313, a memory 314, a display unit 315, a call function unit 317, a communication interface 318, and an input unit 319. These units can transmit / receive data or signals to / from each other through a signal line (CPU bus or the like).

  The control unit 300 includes a CPU and the like. The ROM 312 stores an operation program for basic operations of the communication terminal device 3. The RAM 313 is used as an operation area of the control unit 300.

  The memory 314 is a storage medium for storing various data and programs. The memory 314 stores (installs) a gesture management program according to an embodiment of the present invention. The gesture management program causes the control unit 300 to function as the association storage unit 302, the gesture detection unit 303, the image reading unit 304, the determination unit 305, and the transmission control unit 306. However, the associated storage unit 302, gesture detection unit 303, image reading unit 304, determination unit 305, and transmission control unit 306 of the control unit 300 are each a hardware circuit, regardless of the operation based on the gesture management program. It may be constituted by.

  The gesture management program is an application that operates on the server 2, and provides an operation instruction for an application for which display data indicating the operation content is displayed on the display unit 315, by the operator of the application and the communication terminal device 3. It is a program to accept with intervening. The gesture management program is, for example, an operation instruction reception image that is a predetermined operation for receiving instructions on various instruction reception images displayed on the display unit 315 by the application operating on the server 2. Instead of the touching operation, a gesture prepared in advance is accepted as an operation for accepting the instruction.

  The control unit 300 includes a control unit 301, a correspondence storage unit 302, a gesture detection unit 303, an image reading unit 304, a determination unit 305, a transmission control unit 306, and a browser 307.

  The control unit 301 manages operation control of each unit included in the communication terminal device 3.

  The associated storage unit 302 stores a gesture associated with each instruction reception image in advance. This instruction receiving image is made up of display image data (HTML format data etc., which is obtained from the server 2 by the browser 307 and indicates the display contents of the application running on the server 2. In the present embodiment, , Which is an image included in the display data) and is displayed in order to receive an instruction from the operator. When the display position of the instruction reception image on the display screen of the display unit 315 is touched by the operator, an instruction from the operator is received by a touch panel function included in the display unit 315.

  A gesture refers to a slide operation configured by a contact operation by an operator on the display screen of the display unit 315. For example, on the display screen, the operator (1) slides fingers etc. from right to left, (2) slides fingers etc. from left to right, (3) slides fingers etc. from top to bottom (4) Slide a finger or the like from bottom to top. This gesture is detected by the gesture detection unit 303 via the touch panel of the display unit 315.

  The gesture detection unit 303 detects an operator gesture performed on the touch panel included in the display unit 315. For example, the movement of the contact position by the operator based on the contact operation accompanied by the slide movement performed by the operator on the display screen of the display unit 315 is detected one after another by the touch panel function, and the contact position at every time passage Is input to the gesture detection unit 303, the gesture detection unit 303 determines whether or not the locus of each contact position information indicates a shape having the same shape as the gesture. Determine the gesture by.

  The image reading unit 304 reads the instruction receiving image associated with the gesture detected by the gesture detection unit 303 from the association storage unit 302.

  The determination unit 305 determines whether or not an instruction reception image corresponding to the instruction reception image read by the image reading unit 304 exists in the display data of the application acquired from the server 2 by the browser 307. .

  The transmission control unit 306 displays an operation instruction associated with the instruction reception image determined by the determination unit 305 as being present in the acquired display data with respect to the server 2 operating the application. In step S307, control for transmission is performed.

  A browser (communication unit) 307 is an application for browsing an HTML document published on the web and acquiring and displaying display data of an application operating on a server on the web. The browser 307 functions by the control unit 300 operating according to the browser application stored in the memory 314. For example, the browser 307 accesses a server on the web specified by the URL, downloads data such as a document or image described in HTML or XHTML, analyzes the content, analyzes the content of the web page, By reproducing (rendering) the layout, it is possible to browse web pages and display application display data.

  The display unit 315 includes an LCD (Liquid Crystal Display) or the like. On the display unit 315, operation guidance for the operator of the communication terminal device 3 is displayed under the control of the control unit 301.

  The call function unit 317 executes a function for making a telephone call using a public line. The communication terminal device 3 may be configured not to include the call function unit 317.

  The communication interface 318 functions as an interface for performing data communication with the server 2 via the Internet by connecting to a wireless LAN router, and an interface for performing communication necessary for a call function by the call function unit 317, for example.

  The input unit 319 includes a key provided on the housing of the communication terminal device 3, a touch panel provided on the display screen portion of the display unit 315, and the like, and various operation instructions are input from the user.

  Next, processing for receiving an input of an operation instruction for the application indicated by the display data acquired from the server 2 will be described. FIG. 4 is a flowchart illustrating a process of receiving an input of an operation instruction for the application indicated by the display data acquired from the server 2. FIG. 5 is a diagram illustrating an example of a display screen of the display unit 315 in the communication terminal device 3. It is assumed that the control unit 301 of the communication terminal device 3 has already performed an operation according to the gesture management program.

  In the communication terminal device 3, when an instruction to connect to the wireless LAN router is input by an operation of the input unit 319 by the operator, the control unit 301 connects to the wireless LAN router 5 via the communication interface 318. (S1). Then, the control unit 301 connects to the Internet and connects to the server 2 on the web indicated by the URL specified by the operator (S2).

  After connecting to the server 2, the control unit 301 transmits a request for starting the operation of the application designated by the operator to the server 2 (S3).

  In the server 2, when the control unit 201 receives the operation start request of the application via the communication interface 21 (S11), the server 2 starts the operation according to the designated application stored in the HDD 22 (S12). At this time, the communication unit 202 causes display data associated with the operation of the application to be transmitted from the communication interface 21 to the communication terminal device 3 via the interface (S12).

  In the communication terminal device 3, when the browser 307 receives the display data via the communication interface 318, the display data is displayed on the display unit 315, for example, on the display screen D1 as shown in FIG. (S4). The communication and display of the display data by the browser 307 are continued during the operation of the application in the server 2.

  While the display data is displayed on the display unit 315 by the browser 307, the operator moves the slide as described above on the display panel of the display unit 315 on which the display screen D1 is displayed as shown in FIG. When a contact operation with a touch (hereinafter referred to as a slide contact operation) is performed, the gesture detector 303 detects a gesture performed by the operator based on contact position information for each elapsed time that the touch panel detects for the slide contact operation. Detect (YES in S5). Note that the gesture detection unit 303 stores a plurality of gestures in advance, and detects a gesture when the locus of the contact position information acquired from the touch panel has the same identity as the gesture.

  Subsequently, the image reading unit 304 reads the instruction receiving image associated with the detected gesture from the associated storage unit 302 (S6).

  The determination unit 305 determines whether or not the display data acquired by the browser 307 from the server 2 and displayed on the display unit 315 includes an instruction reception image corresponding to the read instruction reception image. If the determination unit 305 determines that the display reception image is included in the display data (YES in S7), the transmission control unit 306 Then, a transmission instruction for causing the server 2 to transmit an instruction similar to the instruction transmitted when the operator performs a touch operation on the display position of the instruction reception image is output (S8). For example, the transmission control unit 306 uses the contact position information input from the touch panel to the browser 307 when the operator performs a contact operation on the display position of the instruction reception image as the transmission instruction. Output.

  When the gesture is not detected by the gesture detection unit 303 at S5 (NO at S5), and when the determination unit 305 determines that the display reception image is not included in the display data at S7. (NO in S7), the process ends.

  Upon receiving the transmission instruction, the browser 307 receives an instruction to be transmitted when the operator performs a contact operation on the display position of the instruction reception image, for example, the instruction reception image is in contact with the server 2. An instruction indicating that it has been designated by the operation is transmitted from the communication interface 318 (S9).

  On the other hand, in the server 2, when the communication interface 21 receives the instruction from the communication terminal device 3 (S13), the control unit 201 performs processing according to the instruction in accordance with the application operating at this time ( S14). That is, the control unit 201 executes processing corresponding to the instruction reception image designated on the display screen of the display unit 315 in the communication terminal device 3 by the operator according to the application.

  According to the series of processes described above, when an operator performs a gesture stored in the associated storage unit 302 on the touch panel of the display unit 315, the gesture detection unit 303 detects the gesture, and the image reading unit 304 The instruction reception image associated with the gesture is read out, and the determination unit 305 determines that the read instruction reception image exists in the display data of the application displayed on the display unit 315. Then, the transmission control unit 306 causes the browser 307 to transmit the instruction associated with the instruction reception image to the server 2. For this reason, even if an instruction input method that the application does not support, for example, an instruction input by a gesture that the operator normally performs in the communication terminal device 3, if the gesture is stored in the associated storage unit 302. The application can be operated by this gesture, and the operability of the operator for the application can be further improved.

  Further, for example, even when an application downloaded from the server or the like via the Internet is installed and used in the communication terminal device 3, similarly, the operability of the operator for the application is further improved as described above. Can be improved.

  Therefore, according to the above-described series of processing, when using an application in the communication terminal device 3, the application can be operated even if the application is an unsupported operation without correcting the application side. It becomes possible for the operator to operate the application by an operation familiar with the communication terminal device 3.

  Next, a first embodiment of processing for registering a gesture in the communication terminal device 3 will be described. FIG. 6 is a block diagram showing a schematic configuration of the communication terminal apparatus 3 that performs the registration. In addition, description is abbreviate | omitted about the structure of the communication terminal device 3 already demonstrated using FIG.

  As shown in FIG. 6, in addition to the configuration shown in FIG. 3, the control unit 300 of the communication terminal device 3 further includes a gesture storage unit 308, a gesture receiving unit 309, an instruction receiving image receiving unit 310, and a correspondence. A management unit 311 is provided. The memory 314 stores (installs) a gesture management program corresponding to gesture registration by the gesture storage unit 308, the gesture reception unit 309, the instruction reception image reception unit 310, and the association management unit 311. The control unit 300 operates according to the gesture management program, so that the control unit 300 further includes a gesture storage unit 308, a gesture reception unit 309, and an instruction reception image reception unit in addition to the units described above with reference to FIG. 310 and the association management unit 311. However, the gesture storage unit 308, the gesture reception unit 309, the instruction reception image reception unit 310, and the association management unit 311 of the control unit 300 are each performed by a hardware circuit without depending on the operation based on the gesture management program. It may be configured.

  The gesture storage unit 308 stores a predetermined gesture that is a candidate for receiving designation from the operator by the gesture receiving unit 309.

  The gesture reception unit 309 receives a gesture specified by the operator via the touch panel function of the display unit 315 among the gestures stored in the gesture storage unit 308.

  The instruction receiving image receiving unit 310 receives an instruction receiving image designated by the operator among the instruction receiving images existing in the display data acquired by the browser 307 from the server 2.

  The association management unit 311 stores the gesture received by the gesture reception unit 309 in the association storage unit 302 in association with the instruction reception image received by the instruction reception image reception unit 310.

  FIG. 7 is a flowchart showing a first embodiment of a process for registering a gesture in the communication terminal device 3. FIG. 8 is a diagram illustrating an example of a display screen of the display unit 315 in the communication terminal device 3.

  When a gesture registration instruction is input by an operation of the input unit 319 by the operator (YES in S21), the gesture reception unit 309 displays a gesture reception screen D2 as illustrated in FIG. 8 on the display screen of the display unit 315. Display. The gesture reception unit 309 displays gesture reception images d21 to d24 indicating gestures as reception candidates on the gesture reception screen D2 (S22).

  For example, the gesture acceptance image d21 is an image showing an arrow from left to right, and shows an operation in which the operator slides a contact operation from left to right on the display screen. Similarly, the gesture reception image d22 shows an operation in which the operator slides the contact operation from the bottom to the top of the display screen. The gesture reception image d23 shows the gesture reception image d23 in which the operator slides the contact operation from the right to the left. The gesture acceptance image d24 indicates an operation in which the operator slides the contact operation from the top to the bottom of the display screen.

  Furthermore, the gesture reception unit 309 displays a registration instruction reception image d25 for receiving a registration instruction associated with the selected gesture and the instruction reception image on the gesture reception screen D2.

  In addition, the gesture reception unit 309 displays the display data acquired by the browser 307 or the like from the server 2 in the partial area D20 on the gesture reception screen D2, and displays the operation content of the application. Thereby, the operator can visually recognize the operation content of the application, and when the touch operation is performed on the partial area D20, the display unit 315 includes an instruction for the instruction receiving image displayed at the touch operation position. Input is possible using a touch panel.

  When the operator touches the display position of a desired gesture reception image among the gesture reception images d21 to d24 when the gesture reception screen D2 is displayed, the gesture displayed at the contact position via the touch panel. The gesture accepting unit 309 accepts the designation of the gesture corresponding to the accepting image (YES in S23).

  Subsequently, when the operator touches the display position of the registration instruction reception image d25 on the gesture reception screen D2, the instruction reception image that will be specified thereafter is used as an instruction reception image that is associated with the gesture for which the above specification has been received. The instruction receiving image receiving unit 310 is in a state of receiving.

  When the operator touches an instruction receiving image as an instruction receiving image in the partial area on the display screen D1, for example, a “return” button d26 shown in FIG. 7, “return” as an instruction receiving image is displayed. The instruction receiving image receiving unit 310 receives an instruction for associating the button d26 with the gesture for which the designation is received via the touch panel (YES in S24).

  The association management unit 311 causes the association storage unit 302 to store the association between the “return” button d26 as an instruction reception image and the gesture whose designation is received in S23 (S25).

  If no gesture is designated in S23 (NO in S23) and if designation of an instruction acceptance image is not accepted in S24 (NO in S24), the process ends.

  Thereby, the operator selects a desired gesture from the gestures stored in the gesture storage unit 308, inputs the designation of the gesture into the gesture reception unit 309, and further displays the above-mentioned information displayed on the display unit 315. By inputting the designation of a desired instruction reception image among the instruction reception images existing in the display data to the instruction reception image reception unit 310, the correspondence management unit 311 allows the operator's desired gesture and instruction reception. The correspondence of the image for use is stored in the correspondence storage unit 302. For this reason, it becomes possible to input an operation instruction corresponding to an instruction reception image in an application running on the server 2 by an operator-desired gesture to operate the application.

  Next, a second embodiment of processing for registering a gesture in the communication terminal device 3 will be described. FIG. 9 is a block diagram showing a schematic configuration of the communication terminal apparatus 3 that performs the registration. In addition, description is abbreviate | omitted about the structure of the communication terminal device 3 already demonstrated using FIG.

  In the second embodiment, as shown in FIG. 9, the control unit 300 of the communication terminal device 3 further includes an instruction receiving image receiving unit 310 and a correspondence managing unit 311 in addition to the configuration shown in FIG. Is provided. The memory 314 stores (installs) a gesture management program corresponding to gesture registration by the instruction receiving image receiving unit 310 and the association management unit 311. The control unit 300 operates in accordance with the gesture management program, so that the control unit 300 functions as an instruction reception image reception unit 310 and a correspondence management unit 311 in addition to the units described above with reference to FIG. To do. However, the instruction receiving image receiving unit 310 and the association management unit 311 of the control unit 300 may be configured by hardware circuits without depending on the operation based on the gesture management program.

  As in the case of the first embodiment, the instruction receiving image receiving unit 310 includes an instruction receiving image designated by the operator among the instruction receiving images existing in the display data acquired by the browser 307 from the server 2. Accept.

  In addition, the association management unit 311 in the second embodiment associates the gesture detected by the gesture detection unit 303 with the instruction reception image received by the instruction reception image reception unit 310, and associates the storage unit. 302 is stored.

  FIG. 10 is a flowchart showing a second embodiment of processing for registering a gesture in the communication terminal device 3. FIG. 11 is a diagram illustrating an example of a display screen of the display unit 315 in the communication terminal device 3.

  When a gesture registration instruction is input by an operation of the input unit 319 by the operator (YES in S31), the gesture reception unit 309 displays a gesture reception screen D3 as illustrated in FIG. 11 on the display screen of the display unit 315. Display. The gesture reception unit 309 displays a registration instruction reception image d35 for receiving a registration instruction associated with the selected gesture and the instruction reception image on the gesture reception screen D3.

  Similarly to the first embodiment, the gesture receiving unit 309 displays the display data acquired by the browser 307 or the like from the server 2 in the partial area D30 on the gesture receiving screen D3, and displays the operation content of the application. Let

  When the operator performs a desired gesture by performing a contact slide operation as shown in FIG. 11 on the gesture reception screen D3 when the gesture reception screen D3 is displayed, the gesture detection unit 303 detects the gesture. (YES in S32).

  Subsequently, when the operator touches the display position of the registration instruction reception image d35 on the gesture reception screen D3, an instruction for associating an instruction reception image to be specified later with the detected gesture through the touch panel. The instruction receiving image receiving unit 310 is in a state of receiving as a receiving image.

  When the operator touches an instruction reception image as an instruction reception image in the partial area D30, for example, a “return” button d36 shown in FIG. 11, a “return” button d36 as an instruction reception image is displayed. The instruction receiving image receiving unit 310 receives an instruction to associate with the detected gesture through the touch panel (YES in S33).

  Thereafter, the association management unit 311 stores the association between the “return” button d36 as the designated instruction reception image and the gesture detected in S32 in the association storage unit 302 (S34).

  If no gesture is detected in S32 (NO in S32) and if designation of an instruction receiving image is not accepted in S33 (NO in S33), the process ends.

  Accordingly, if the operator performs a desired gesture on the touch panel of the display unit 315, the gesture is detected by the gesture detection unit 303, and if an instruction reception image associated with the gesture is further specified, the association management unit 311 Thus, the association between the desired gesture performed by the operator and the designated instruction receiving image is stored in the association storage unit 302. For this reason, it becomes possible to input an operation instruction corresponding to an instruction reception image in the application displayed on the display unit 315 by a desired gesture performed by the operator and operate the application.

  Note that after the gesture registration in the first and second embodiments, the processing shown in FIG. 3 is performed as described above.

  The present invention is not limited to the configuration of the above embodiment, and various modifications can be made. For example, in the above embodiment, the information processing apparatus according to an embodiment of the present invention is described as a mobile terminal and a mobile phone as a communication terminal apparatus. However, this is only an example, and other electronic apparatuses, for example, It may be a personal computer, car navigation, or the like.

  The configurations and processes shown in the above embodiments using FIGS. 1 to 11 are only one embodiment of the present invention, and are not intended to limit the present invention to the configurations and processes.

DESCRIPTION OF SYMBOLS 10 Communication system 2 Server 3 Communication terminal device 300 Control unit 301 Control unit 302 Corresponding storage unit 303 Gesture detection unit 304 Image reading unit 305 Determination unit 306 Transmission control unit 307 Browser 308 Gesture storage unit 309 Gesture reception unit 310 Instruction reception image Reception unit 311 Corresponding management unit 315 Display unit 319 Input unit

Claims (4)

An information processing apparatus that obtains display data related to the application from another information processing apparatus that operates the application via a network and displays the display data on the display unit.
A correspondence storage unit that stores a gesture associated in advance for each instruction reception image included in the display data;
A gesture detection unit for detecting an operator's gesture performed on the touch panel included in the display unit;
An image reading unit that reads an instruction receiving image associated with the gesture detected by the gesture detecting unit from the associated storage unit;
A determination unit that determines whether or not an instruction reception image read by the image reading unit exists in the acquired display data;
An information processing apparatus as a transmission control section that causes the communication section to transmit an operation instruction associated with an instruction receiving image determined to be present in the display data by the determination section to the other information processing apparatus Gesture management program to make it work. A gesture storage unit for storing predetermined gestures;
Among the gestures stored in the gesture storage unit, a gesture reception unit that receives a gesture specified by an operator;
An instruction receiving image receiving unit that receives an instruction receiving image specified by an operator among the instruction receiving images present in the display data;
An information processing apparatus is further provided as an association management unit that associates the gesture received by the gesture reception unit with the instruction reception image received by the instruction reception image reception unit and stores the association in the association storage unit. The gesture management program according to claim 1, which is made to function. An instruction receiving image receiving unit that receives an instruction receiving image specified by an operator among the instruction receiving images present in the display data;
An information processing apparatus is further provided as an association management unit that associates the gesture detected by the gesture detection unit with the instruction reception image received by the instruction reception image reception unit and stores the association in the association storage unit. The gesture management program according to claim 1, which is made to function. A display unit;
A communication unit that acquires display data related to the application from another information processing apparatus that operates the application via a network;
A display control unit that causes the display unit to display the display data acquired by the communication unit;
A correspondence storage unit that stores a gesture associated in advance for each instruction reception image included in the display data;
A gesture detection unit for detecting an operator's gesture performed on the touch panel included in the display unit;
An image reading unit that reads an instruction receiving image associated with the gesture detected by the gesture detecting unit from the associated storage unit;
A determination unit that determines whether or not an instruction reception image read by the image reading unit exists in the acquired display data;
Information including: a transmission control unit that causes the communication unit to transmit an operation instruction associated with an instruction receiving image determined to be present in the display data by the determination unit to the other information processing apparatus. Processing equipment.