JP6664589B1 - Display program, display device, and display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve usability. A display program according to the present application is captured in a captured image using a captured image (for example, a screen shot of a screen of a terminal device used by a user or an image captured using the terminal device). An acquisition procedure for acquiring target information that is information related to the shooting target (for example, information obtained by analyzing the shooting target, an icon image indicating the provider of the shooting target, and the target information acquired by the acquisition procedure) It is characterized by causing a computer to execute a display procedure of displaying the image in an overlapping manner. [Selection diagram] FIG.

Description

  The present invention relates to a display program, a display device, and a display method.

  2. Description of the Related Art Conventionally, a technique for displaying an acquired screen shot is known. For example, a technique has been proposed in which a screen shot displayed on a screen is acquired, and a title or URL of a web page on which the screen shot is displayed is superimposed on the screen shot and displayed.

JP 2014-533013 A International Publication No. WO 2016/017420

  However, the above-mentioned conventional technology does not always have high usability. For example, in the above-described conventional technology, an application or an IT service for which a screen shot is displayed cannot be intuitively understood, and there is room for improvement in terms of usability.

  The present application has been made in view of the above, and has as its object to provide a display program, a display device, and a display method with high usability.

  A display program according to the present application is configured to obtain, using a captured image, target information that is information on a shooting target captured in the captured image, and to superimpose the target information obtained in the obtaining procedure on the captured image. And causing the computer to execute a display procedure for causing the computer to display.

  According to an aspect of the embodiment, there is an effect that usability can be improved.

FIG. 1 is a diagram (1) illustrating an example of a terminal device according to the embodiment. FIG. 2 is a diagram (2) illustrating an example of the terminal device according to the embodiment. FIG. 3 is a diagram (3) illustrating an example of the terminal device according to the embodiment. FIG. 4 is a diagram (4) illustrating an example of the terminal device according to the embodiment. FIG. 5 is a diagram illustrating a configuration example of the display processing system according to the embodiment. FIG. 6 is a diagram illustrating a configuration example of the content distribution server according to the embodiment. FIG. 7 is a diagram illustrating an example of a content information storage unit according to the embodiment. FIG. 8 is a diagram illustrating a configuration example of the terminal device according to the embodiment. FIG. 9 is a diagram illustrating an example of the behavior information storage unit according to the embodiment. FIG. 10 is a diagram (5) illustrating an example of the terminal device according to the embodiment. FIG. 11 is a diagram (6) illustrating an example of the terminal device according to the embodiment. FIG. 12 is a flowchart (1) illustrating an example of a display processing procedure according to the embodiment. FIG. 13 is a flowchart (2) illustrating an example of a display processing procedure according to the embodiment. FIG. 14 is a flowchart (3) illustrating an example of a display processing procedure according to the embodiment. FIG. 15 is a hardware configuration diagram illustrating an example of a computer that implements the functions of the terminal device.

  An embodiment (hereinafter, referred to as “embodiment”) for implementing a display program, a display device, and a display method according to the present application will be described in detail with reference to the drawings. The display program, the display device, and the display method according to the present application are not limited by this embodiment. In the following embodiments, the same portions are denoted by the same reference numerals, and redundant description will be omitted.

[1. Embodiment)
Display processing realized by the display program and the like of the present embodiment will be described with reference to FIGS. In the following description, an application for collecting (stocking) behavior information on the behavior of the user (hereinafter, may be referred to as an “information collection application”) is mounted on the terminal device 100 used by the user. Here is an example. In the following description, it is assumed that the display processing according to the embodiment is realized by the function of the information collection application.

  The terminal device 100 shown in FIGS. 1 to 4 is a smart device such as a smartphone or a tablet used by a user, and is an arbitrary server device via a wireless communication network such as 3G (Generation) or LTE (Long Term Evolution). A portable terminal device capable of communicating with the mobile terminal. 1 to 4 show a case where the terminal device 100 is a smartphone used by the user U1. Further, it is assumed that the terminal device 100 has an output unit such as a liquid crystal display (hereinafter, sometimes referred to as “screen”) and employs a touch panel. That is, the user U1 performs various operations of the terminal device 100 by touching the display surface of the output unit with a finger or a dedicated pen. Note that the terminal device 100 is not limited to a smart device, and may be an information processing device such as a notebook PC (Personal Computer), a desktop PC, a mobile phone, or a PDA (Personal Digital Assistant).

  Hereinafter, the display processing of the information collection application performed by the terminal device 100 will be described with reference to FIGS. 1 to 4. In the following description, when the state transition of the screen of the terminal device 100 is illustrated, the state transition is represented as a first state, a second state,..., An N-th state (N is an arbitrary number). In addition, in the examples illustrated in FIGS. 1 to 4, it is assumed that the behavior information of the user U1 is collected by the information collection application mounted on the terminal device 100. Here, in the following description, the behavior information refers to a screen shot of the screen of the terminal device 100 acquired by the user U1 using the terminal device 100, an image and a moving image captured by the user U1 using the terminal device 100, Images and videos viewed by U1 using the terminal device 100, e-mails transmitted and received by the user U1 using the terminal device 100, applications used by the user U1 via the terminal device 100, bookmarked contents in web services and the like. Show.

  First, with reference to FIG. 1, a description will be given of a display process of a theme of a user's behavior estimated based on the user's behavior information. FIG. 1 is a diagram (1) illustrating an example of a terminal device according to the embodiment.

  The first state in FIG. 1 shows a state of a screen displaying a subject of the behavior of the user U1 estimated in the past. In the first state of FIG. 1, the terminal device 100 displays, in the area AR11, a content corresponding to the subject of the behavior of the user U1 estimated in the past. In the first state of FIG. 1, the terminal device 100 displays an area AR12 for instructing the terminal device 100 to perform a process of estimating the subject of the action.

  Subsequently, the second state in FIG. 1 shows a state of a screen displayed when an instruction for subject estimation processing is received from the user U1. In the second state of FIG. 1, the terminal device 100 displays a thumbnail corresponding to the behavior information of the user U1 collected by the information collection application in the area AR13. In the following description, a thumbnail indicates a screen shot itself, which is action information, an image included in a bookmarked content, and the like.

  For example, in the second state of FIG. 1, the terminal device 100 displays the thumbnails D1 to D9 corresponding to the action information of the user U1 in the area AR13. In addition, the terminal device 100 displays an area AR14 in which the user U1 performs an operation of selecting a thumbnail displayed in the area AR13 so as to be superimposed on each thumbnail. In addition, the terminal device 100 displays an area AR15 for performing the operation of confirming the thumbnail selected by the user U1.

  Subsequently, the third state in FIG. 1 shows a state of the screen on which a thumbnail selection operation has been received from the user U1. In the example of FIG. 1, it is assumed that the user U1 has selected the thumbnails D1 to D3. Here, the thumbnail D1 corresponds to the content related to the restaurant A in Sendai city bookmarked by the user U1 in the restaurant introduction service. The thumbnail D2 is a search result obtained by the user U1 searching for a transfer guide from the departure place “Tokyo” to the arrival place “Sendai” in the route search service with the departure date and time set at “7:56 on January 13”. Corresponds to the screenshot of the screen shown. The thumbnail D3 corresponds to an e-mail received by the user U1 using the terminal device 100 and relating to the reservation confirmation of the hotel B in Sendai city on "January 13". Here, the terminal device 100 may analyze information such as text information, time information, and position information included in the action information corresponding to each thumbnail by appropriately using various conventional techniques such as image analysis and morphological analysis. Extract by

  Then, the terminal device 100 changes the screen from the third state to the fourth state by selecting the area AR15.

  Subsequently, the fourth state in FIG. 1 shows a state in which the subject of the behavior of the user U1 estimated by the terminal device 100 based on the behavior information corresponding to the thumbnail selected by the user U1 is displayed on the screen. In the fourth state of FIG. 1, the terminal device 100 displays, in the area AR16, the subject of the action of the user U1 estimated based on the action information corresponding to the thumbnails D1 to D3. In addition, in the fourth state of FIG. 1, the terminal device 100 displays an area AR17 in which the user U1 performs an operation for determining the subject of the action.

  For example, in the example of FIG. 1, the terminal device 100 determines that the behavior information corresponding to the thumbnails D1 to D3 relates to Sendai City, searches for a transfer guide from the departure place “Tokyo” to the destination “Sendai”, and Since the reservation of the hotel B in Sendai is for the same date, the subject of the action of the user U1 is estimated to be “Sendai trip” on January 13, and the estimated result is displayed in the area AR16. . Then, the terminal device 100 changes the screen from the fourth state to the fifth state by selecting the area AR17.

  Subsequently, the fifth state in FIG. 1 shows a state in which content corresponding to the estimated subject of the behavior of the user U1 has been added to the screen displayed in the first state. As shown in the fifth state of FIG. 1, the terminal device 100 displays a content corresponding to “Sendai trip” in the area AR11.

  Next, a display process of a content corresponding to the subject of the user's action will be described with reference to FIG. FIG. 2 is a diagram (2) illustrating an example of the terminal device according to the embodiment. FIG. 2 illustrates an example in which the terminal device 100 in FIG. 1 edits the content C1 (content relating to the schedule of “Sendai trip”) corresponding to the subject “Sendai trip” of the behavior of the user U1 estimated. Further, the content C1 is content optimized for a smart device, and is assumed to be a web page whose horizontal display size is the same as the horizontal display size of the screen of the terminal device 100. It is assumed that the content C1 is a content whose display size in the vertical direction is longer than the display size in the vertical direction of the screen of the terminal device 100.

  Further, as shown in FIG. 2, the terminal device 100 includes the content C11, C12, and C14 indicating the action information corresponding to the thumbnails D1 to D3 selected by the user U1 in the third state of FIG. The content C1 including the content C13 for recommending information related to “Sendai travel” to the user U1 is displayed on the screen. In the example of FIG. 2, the terminal device 100 includes an area AR21 in which the contents included in the content C1 are arranged in chronological order and displayed on the screen. For example, the terminal device 100 extracts time information by appropriately analyzing the behavior information corresponding to the content included in the content C1 using various conventional techniques such as image analysis and morphological analysis. Then, the terminal device 100 arranges and displays the content at the display position corresponding to the extracted time information in the area AR21.

  Here, in the following description, the terminal device 100 sets the departure date and time “January 13 at 7:56” and the arrival date and time “January 13 at 9:30” as time information from the behavior information corresponding to the content C12. It shall be extracted. In this case, the terminal device 100 displays the area AR21 at the display position corresponding to the time information extracted from the content C12.

  Also, in the following description, it is assumed that the terminal device 100 extracts the check-in available time to the hotel B “15: 30-29: 00” from the behavior information corresponding to the content C14 as time information. In this case, in the subject of the action of the user U1 "Sendai trip", it is assumed that the action of the user U1 (check-in to the hotel B) corresponding to the content C14 is performed at the end of the schedule. The content C14 is arranged and displayed at the lowermost end of the area AR21. Then, the terminal device 100 receives a selection operation (for example, a tap) on the content C14, and changes the display position of the content C14 according to the drag operation of the finger F10 of the user U1 to the check-in available time “15” in the area AR21. 30 minutes to 29 hours ".

  In the following description, the terminal device 100 extracts only the business hours of “restaurant A” as time information from the behavior information corresponding to the content C11, and extracts a fixed time (for example, a reservation date and time). It shall not be done. In this case, the terminal device 100 determines that the time during which the action of the user U1 (visiting to the restaurant A) corresponding to the content C12 is performed is undetermined, and places the content C12 in an area other than the area AR21 (the top end of the content C1). And display it. That is, in the schedule of the “Sendai trip” in the initial state, the action related to visiting the restaurant A is not incorporated.

  Further, in the description of FIG. 2, the terminal device 100 arranges the content C13 in an area where other content is not arranged in the area AR21 (that is, a position corresponding to a blank time in which a schedule is not determined). indicate.

  The first state in FIG. 2 shows an initial state in which the terminal device 100 displays the content C1 on the screen. In the first state of FIG. 2, the terminal device 100 displays the content C11 and the content C12 on the screen from the top end of the content C1. Then, the terminal device 100 superimposes and displays the area AR22 for instructing the terminal device 100 to change the display mode of the content C1 on the content C1. In addition, the terminal device 100 displays an area AR23 for instructing the terminal device 100 to save the content C1 (board) to be superimposed on the content C1. When the area AR23 is selected by the user U1, the terminal device 100 ends the editing process of the content C1, and transmits the content arranged in the area AR21 and the information on the time corresponding to each content to the “Sendai trip” schedule. In the storage unit of the terminal device 100. Note that the terminal device 100 may set an alarm to be notified to the user U1 based on information on time corresponding to each content.

  Here, it is assumed that the user U1 desires to visit the restaurant A in the first state of FIG. In this case, the terminal device 100 accepts a selection operation on the content C11 by the finger F10 of the user U1, and moves the display position of the content C11 according to the drag operation of the finger F10 of the user U1. Then, the terminal device 100 arranges the content C11 at a position where the selection of the content C11 by the finger F10 of the user U1 is released, and displays the content C11 on the screen. In the description of FIG. 2, it is assumed that the content C11 is arranged in an area corresponding to “10:00” to “12:00” in the area AR21.

  Subsequently, the second state in FIG. 2 is a state in which the area of the content C1 that is not displayed in the first state is displayed on the screen by moving the content C1 downward by a scroll operation or the like of the user U1. Is shown. In the second state of FIG. 2, the terminal device 100 displays the content C11, the content C12, and the content C13 on the screen. Also in the second state, the terminal device 100 displays the area AR22 and the area AR23 so as to overlap the content C1.

  Here, it is assumed that in the second state of FIG. 2, the user U1 desires to incorporate a schedule into a blank time. In this case, the terminal device 100 receives the selection operation for the content C13 by the user U1, and makes the screen transition to the third state.

  Subsequently, the third state in FIG. 2 shows a state in which content indicating information related to the subject of the action “Sendai trip” is displayed on the screen instead of the content C1. In the third state of FIG. 2, the terminal device 100 searches for information corresponding to the subject of the action, and displays the content corresponding to the searched information on the screen as content recommended for the user U1. For example, the terminal device 100 searches for information corresponding to the theme of the action “Sendai trip” from the web (a web page or various applications installed on the terminal device 100). In addition, the terminal device 100 is the activity information of the user U1 other than the activity information corresponding to the content C11, the content C12, and the content C13 collected by the information collection application mounted on the terminal device 100, and the subject of the activity is “Sendai travel. Search for action information related to "." Further, the terminal device 100 searches for information corresponding to the theme of the action “Sendai trip” based on the demographic attribute and the psychographic attribute based on the action information of the user U1. Then, the terminal device 100 displays, among the searched information, contents related to actions that can be performed by the user U1 in the blank time as contents indicating information recommended to the user U1. In the third state of FIG. 2, it is assumed that the terminal device 100 displays content C15 and content C16 on the screen as content indicating information recommended to the user U1.

  In the third state of FIG. 2, it is assumed that it is desired to incorporate a schedule related to the content C15 into the blank time. In this case, the terminal device 100 receives the selection operation on the content C15 by the user U1, and makes the screen transition to the fourth state.

  Subsequently, the fourth state in FIG. 2 shows a state of the screen on which the new content is displayed in place of the content C13. In the fourth state of FIG. 2, the terminal device 100 displays the content C17 corresponding to the content C15 selected by the user U1 in the third state on the screen instead of the content C13. Also in the second state, the terminal device 100 displays the area AR22 and the area AR23 so as to overlap the content C1.

  Here, in the fourth state of FIG. 2, when the area AR22 is selected, the terminal device 100 changes the screen to the fifth state or the sixth state.

  Subsequently, the fifth state in FIG. 2 shows a state of the screen in which the content arranged in the area AR21 of the content C1 is arranged at a position corresponding to each content on the map. For example, in the example of FIG. 2, the terminal device 100 appropriately uses various conventional techniques such as image analysis and morphological analysis on the behavior information corresponding to the content C11 and the content C14 and the information regarding the content C17. By analyzing, position information is extracted. Then, the terminal device 100 arranges and displays each content at a position P1 corresponding to the content C11, a position P2 corresponding to the content C17, and a position P3 corresponding to the content C14 on the map of Sendai City.

  In addition, the terminal device 100 may arrange and display each content on the content C1 based on the position information extracted from the behavior information corresponding to the content included in the content C1. For example, the terminal device 100 sets “Sendai Station” as a departure point. Then, the terminal device 100 extracts location information from the behavior information corresponding to each content included in the content C1, and arranges each content in the content C1 according to the distance of the extracted location information from “Sendai Station”. indicate.

  In addition, the terminal device 100 determines the position P1 on the map of Sendai City based on the information on the time corresponding to each content arranged in the area AR21 (information on the time when the user U1 moves to the position corresponding to each content). Information regarding the order in which the user U1 visits may be displayed on P3.

  Subsequently, the sixth state in FIG. 2 shows a state of a screen that displays thumbnails corresponding to the content arranged in the area AR21 of the content C1 in an album format. In the sixth state of FIG. 2, the terminal device 100 displays thumbnails corresponding to the contents C11, C12, C14, and C17 arranged in the area AR21 of the content C1 on the screen in the form of an album.

  In addition, in the example of FIG. 2, as shown in the sixth state, the content C18 on the property of the user U1 on the theme of the action "Sendai trip" may be displayed. For example, the terminal device 100 accepts a selection operation on the content C18 by the user U1, and transitions the screen to an editing screen of the belongings of the user U1 on the theme of the action "Sendai trip". In addition, the terminal device 100 may display an icon image or the like for the user U1 to instruct the terminal device 100 to transition to the editing screen of the belongings in the first state to the sixth state in FIG. 2.

  In addition, in the example of FIG. 2, the terminal device 100 may search for information regarding a means of movement between positions corresponding to the content included in the content C1, and display the content corresponding to the searched information on the content C1. For example, in the second state of FIG. 2, the terminal device 100 searches for a means of transportation from Sendai Station to the location of the restaurant A by a route search service or the like, and outputs a content indicating the search result between the content C11 and the content C12. Deploy.

  In addition, in the example of FIG. 2, when the activity information related to the theme of the activity “Sendai trip” is newly collected after the display of the content C1, the terminal device 100 displays the content corresponding to the newly collected activity information. May be notified as to whether or not to place the content in the content C1.

  Next, a display process for superimposing and displaying information related to action information on a thumbnail will be described with reference to FIG. FIG. 3 is a diagram (3) illustrating an example of the terminal device according to the embodiment. In FIG. 3, when a thumbnail corresponding to the behavior information of the user U1 collected by the information collection application is displayed on the screen (corresponding to the second state in FIG. 1), information on the behavior information is included in each thumbnail (hereinafter, “target information”). Are described in some cases). Here, in the following description, the target information is information (for example, text information, time information, position information, etc.) obtained by analyzing the behavior information by appropriately using various conventional techniques such as image analysis and morphological analysis. Information, etc.), information for specifying a provider of the behavior information (for example, an application, a web service, and the like), and an icon image corresponding to the provider of the behavior information (for example, an application icon or a web service) Icon images (web click icons, etc.). In the following description, it is assumed that the information collection application has acquired the target information in association with the behavior information.

  In the following description, an example is shown in which the terminal device 100 displays thumbnails D1 to D9 corresponding to the behavior information collected by the information collection application. The thumbnails D1 to D3 are the same as those shown in FIG. The thumbnail D4 corresponds to a screen shot of a screen displaying the ticket issuing code (QR code (registered trademark)) of the ticket of the concert C purchased by the user U1 using the terminal device 100 in the ticket sales service. The thumbnail D5 corresponds to a screen shot in the map application. The thumbnails D6 to D9 correspond to products bookmarked by the user U1 in the electronic commerce service.

  The first state in FIG. 3 shows a state of a screen displaying a thumbnail corresponding to the behavior information of the user U1 collected by the information collection application. In the example of FIG. 3, the terminal device 100 displays an area AR18 for displaying the target information so as to be superimposed on each thumbnail. The areas AR13 to AR15 are the same as those shown in the second state of FIG.

  In the example of FIG. 3, the terminal device 100 displays text information acquired from the behavior information and an icon image corresponding to a provider of the behavior information in the area AR18 as target information. For example, in the example of FIG. 3, the terminal device 100 superimposes and displays text information on the thumbnails D1 and D5 together with an icon image corresponding to the provider of the behavior information. Further, when the behavior information includes time information (for example, a departure date and a reservation date and time), the terminal device 100 superimposes and displays target information including the time information as shown in thumbnails D2 to D5. Further, when the behavior information includes a product or product information such as a price of the product, the terminal device 100 superimposes and displays target information including the product information as shown in thumbnails D6 to D9.

  Note that the terminal device 100 may execute a process corresponding to the character string acquired as the target information. For example, when the target information including the time information is displayed in a manner superimposed on the thumbnail, the terminal device 100 notifies the user U1 of the alarm based on the time information (for example, if the restaurant bookmarked by the user in the restaurant introduction service is Alerts indicating that the business hours are closed or closed. The terminal device 100 may set an alarm to notify the user U1 based on the current position of the user U1. For example, when the behavior information (for example, the content related to the restaurant bookmarked by the user in the restaurant introduction service) includes the position information, the terminal device 100 determines that the current position of the user U1 is within a predetermined range from the position information (for example, , 100 meters, 500 meters, etc.), the user U1 may be notified of the action information. Further, for example, the terminal device 100 may collect information about a store near the current position of the user U1 from an external device or the like such as a service providing device that provides a service to the user. Then, if there is a store that handles a product corresponding to the user's action information (for example, a product corresponding to the thumbnails D6 to D9) in a store within a predetermined range from the current position of the user U1, the terminal device 100 determines that the user U1 May be notified of the behavior information.

  Further, the terminal device 100 may display the target information updated based on the information provided from the provider of the behavior information. For example, the terminal device 100 may superimpose and display target information including information on changes such as the business hours of the restaurant A and holidays on the thumbnail D1. Further, the terminal device 100 may display the target information including the delay information of the train superimposed on the thumbnail D2. Further, the terminal device 100 may superimpose and display target information including information on a change in the price of a product (for example, information on a discount) on the thumbnails D6 to D9. Further, the terminal device 100 may notify the user U1 that the target information has been updated based on the information provided from the provider of the behavior information.

  Subsequently, the second state in FIG. 3 shows a state of the screen displayed when the thumbnail displayed in the first state is selected by the user U1. In the example of FIG. 3, when the thumbnail displayed in the first state is selected by the user U1, the terminal device 100 is a provider of the behavior information corresponding to the thumbnail, and is a content corresponding to the behavior information. Is displayed on the screen. Here, it is assumed that the thumbnail D2 is selected in the example of FIG. In this case, the terminal device 100 displays, on the screen, the content (search result) on the route search service when the user U1 acquires the screen shot as the behavior information corresponding to the thumbnail D2. That is, the terminal device 100 displays the application or web service that is the source of the behavior information in a state where the same parameter information that was input when the behavior information was acquired is input.

  In addition, when the thumbnail corresponding to the image of the product photographed by the user U1 at the physical store using the imaging unit of the terminal device 100 is selected by the user U1, the terminal device 100 displays a predetermined electronic commerce service corresponding to the product. Content may be displayed.

  Next, with reference to FIG. 4, a description will be given of a process of displaying content indicating information related to the subject of the user's action (hereinafter, may be described as “related content”). FIG. 4 is a diagram (4) illustrating an example of the terminal device according to the embodiment. FIG. 4 illustrates an example of displaying related content in the editing process of the content C2 (content relating to the schedule of “Sendai travel”) corresponding to the subject “Sendai travel” of the behavior of the user U1 estimated by the terminal device 100 in FIG. Show. The content C2 is content optimized for a smart device, and is assumed to be a web page whose horizontal display size is the same as the horizontal display size of the screen of the terminal device 100. It is assumed that the content C2 is a content whose vertical display size is longer than the vertical display size of the screen of the terminal device 100.

  Further, as shown in FIG. 4, the terminal device 100 displays on the screen a content C2 including contents C12 and C14 indicating action information corresponding to the thumbnails D2 and D3 selected by the user U1 in the third state of FIG. I do. It should be noted that the contents C12 and C14 in the example of FIG. 4 are the same as those shown in FIG.

  4 shows an initial state in which the terminal device 100 displays the content C2 on the screen. In the first state of FIG. 4, the terminal device 100 displays the content C12 on the screen from the top end of the content C2. Then, the terminal device 100 displays the areas AR22 and AR23 so as to be superimposed on the content C2. Note that the regions AR22 and AR23 in the example of FIG. 4 are the same as those shown in FIG.

  Here, in the first state of FIG. 4, it is assumed that an upward sliding operation has been performed on the lowermost part of the screen (not the area AR23) by the finger F10 of the user U1. In this case, the terminal device 100 changes the screen to the second state.

  Subsequently, the second state in FIG. 4 shows a state in which an area for displaying related content (hereinafter, sometimes referred to as a “related content area”) is displayed so as to be superimposed on the content C2. In the example of FIG. 4, in the related content area, the action information corresponding to the thumbnail D1 selected by the user U1 in the third state of FIG. 1 (in other words, the action information of the action information selected by the user U1 in the third state of FIG. 1). Among them, the content C11 indicating the behavior information of the user U1 corresponding to the behavior information and the time at which the behavior of the user U1 is performed is undecided), and the information related to the theme of the behavior "Sendai trip", which is recommended to the user U1. Contents C19 and C20 indicating information are included. Note that the content C11 in the example of FIG. 4 is the same as that shown in FIG.

  Here, in the second state of FIG. 4, it is assumed that a selection operation on the content C11 by the finger F10 of the user U1 has been performed. In this case, the terminal device 100 changes the screen to the third state.

  Subsequently, the third state in FIG. 4 shows a state where the related content area is reduced. In the third state of FIG. 4, it is assumed that the selection operation on the content C11 by the finger F10 of the user U1 is continued. In this case, the terminal device 100 reduces the related content area in a state where the display position of the content C11 is held by the finger F10 of the user U1. That is, the terminal device 100 superimposes and displays the content C11 on the content C2 while being held by the finger F10 of the user U1.

  Subsequently, the fourth state in FIG. 4 illustrates a state in which the related content area has been reduced and related content other than the content C11 has been hidden from the screen of the terminal device 100. In the third state of FIG. 4, it is assumed that the selection operation on the content C11 by the finger F10 of the user U1 is continued. Here, the terminal device 100 moves the display position of the content C11 according to the drag operation on the content C11 by the finger F10 of the user U1.

  Subsequently, a fifth state in FIG. 4 shows a state in which the display position of the content C11 has been moved. As shown in the fifth state of FIG. 4, the terminal device 100 displays the content C11 at a position corresponding to the operation of the user U1 with the finger F10.

  Subsequently, the sixth state in FIG. 4 shows a state in which the selection operation on the content C11 by the finger F10 of the user U1 has been released. As illustrated in the sixth state of FIG. 4, the terminal device 100 incorporates the schedule related to the content C11 at a position (time zone) where the selection operation on the content C11 is released.

  Note that the terminal device 100 may display the related content area in a manner superimposed on a predetermined area in the content C2 displayed on the screen. For example, the terminal device 100 superimposes and displays the related content area on the area from the bottom of the screen to the center of the screen (that is, the lower half area of the screen) of the content C2 displayed on the screen. Further, in the first state of FIG. 4, the terminal device 100 sets the related content area to an area from the bottom of the screen to a position where the upward sliding operation on the bottom of the screen with the finger F10 of the user U1 is released. .

  Further, when receiving a predetermined operation (for example, a flick operation) on the content arranged in the content C2, the terminal device 100 may delete the content from the content C2 and display the content in the related content area.

  Further, when the terminal device 100 receives a selection operation (for example, a touch operation; hereinafter, sometimes referred to as a “movement operation”) of the user U1 with the finger F10 on the content arranged in the content C2, the terminal device 100 transmits the content. The content may be displayed at a position corresponding to the operation of the finger F10 of the user U1, and the content may be displayed at a position where the moving operation has been released in the content C2. Here, the terminal device 100 may update the behavior information (for example, time information) corresponding to the content according to the position (time zone) at which the moving operation of the content is released.

  In addition, when the terminal device 100 receives a selection operation (for example, a long tap) different from the movement operation by the user U1 with the finger F10 on the content arranged in the content C2, the terminal device 100 transmits the content to the user. The related content area may be displayed so as to be superimposed on the content C2 while being held by the finger F10 of U1. Then, when the selection of the content in the related content area is released, the terminal device 100 may delete the content from the content C2 and display the content in the related content area.

  Further, the screen shot of the screen of the terminal device 100 in the above description may be obtained by a function of the information collection application, or may be obtained by a function of the OS (Operating System) of the terminal device 100.

  Further, the behavior information of the user in the above description is not limited to information collected by the function of the information collection application, and may include information added to the information collection application in response to a user operation.

  In general, a user examines things based on some action guidelines, uses the collected information to schedule and act. At this time, when the results of the information collection are aggregated and provided to the user in the form of a schedule or the like, the user experience is improved. Here, the terminal device 100 according to the embodiment estimates the subject of the behavior of the user based on the behavior information of the user as described above, and displays the content corresponding to the subject of the behavior. That is, the terminal device 100 according to the embodiment can provide the information based on the guideline of the user's action.

  Conventionally, a technology for capturing a screen shot of a display screen while operating an application of a mobile terminal to acquire image data and a technology for superimposing and displaying a title or URL when a screen shot image is a web page are disclosed. Have been. However, in the conventional technology, the application or the IT service from which the screen shot acquired in the past is based cannot be intuitively understood, and there is room for improvement in usability. On the other hand, the terminal device 100 according to the embodiment can display the text information included in the screenshot and the information about the provider of the screenshot as superimposed on the screenshot as described above. That is, the terminal device 100 according to the embodiment can improve usability.

  Conventionally, there has been proposed a technique for adjusting the layout of content (main information and sub-information) based on the arrangement of the sub-display area, which is different from the main information display area. However, in the related art, operability when a user changes the arrangement of contents is not considered, and there is room for improvement in usability. On the other hand, the terminal device 100 according to the embodiment can arrange information (sub-information) displayed in the related content area as described above in main content by a user's intuitive operation. That is, the terminal device 100 according to the embodiment can improve usability.

[2. Configuration of display processing system]
Next, a display processing system 1 for realizing the above-described display processing will be described with reference to FIG. FIG. 5 is a diagram illustrating a configuration example of the display processing system according to the embodiment. As shown in FIG. 5, the display processing system 1 includes a terminal device 100, a content distribution server 10, and an external device 20. The terminal device 100, the content distribution server 10, and the external device 20 are communicably connected via a network N by wire or wirelessly. The network N is a communication network such as a LAN (Local Area Network), a WAN (Wide Area Network), a telephone network (a mobile telephone network, a fixed telephone network, etc.), a regional IP (Internet Protocol) network, the Internet, and the like. The network N may include a wired network or a wireless network. Further, the display processing system 1 illustrated in FIG. 5 may include a plurality of terminal devices 100, a plurality of content distribution servers 10, and a plurality of external devices 20.

  The content distribution server 10 is a server device that distributes content related to the information collection application to the terminal device 100. The content distribution server 10 may distribute the data of the information collection application itself to the terminal device 100 as the content. Note that the content distribution server 10 may function as a distribution device that distributes control information to the terminal device 100. Here, the control information is described in, for example, a script language such as JavaScript (registered trademark) or a style sheet language such as CSS (Cascading Style Sheets). Note that the information collection application distributed from the content distribution server 10 itself may be regarded as control information. The configuration of the content distribution server 10 will be described later.

  The external device 20 is a server device that distributes information of various contents to the terminal device 100. The external device 20 relates to, for example, a portal site, a news site, an auction site, a weather forecast site, a shopping site, a finance (stock price) site, a route search site, a map providing site, a travel site, a restaurant introduction site, a web blog, and the like. This is a device that distributes, to the terminal device 100, a web page in which various types of information to be placed are arranged. The external device 20 includes, for example, various applications installed on the terminal device 100 (for example, a portal application, a news application, an auction site, a weather forecast application, a shopping application, a finance (stock price) application, a route search application, and a map providing application). , A travel application, a restaurant introduction application, a blog browsing application, etc.). Further, the external device 20 may be a server that distributes application data itself. The external device 20 may function as a distribution device that distributes control information to the terminal device 100. Here, the control information is described in, for example, a script language such as JavaScript (registered trademark) or a style sheet language such as CSS. Note that the application itself distributed from the external device 20 may be regarded as the control information.

  The terminal device 100 is an information processing device used by a user. The terminal device 100 acquires information related to the information collection application from the content distribution server 10 according to a user operation. In addition, the terminal device 100 acquires information of various contents from the external device 20 according to a user operation. Further, when the terminal device 100 receives the control information for realizing the above-described display process from the content distribution server 10, the terminal device 100 realizes the display process according to the control information. The configuration of the terminal device 100 will be described later.

[3. Configuration of Content Distribution Server]
Next, the configuration of the content distribution server 10 according to the embodiment will be described with reference to FIG. FIG. 6 is a diagram illustrating a configuration example of the content distribution server according to the embodiment. As shown in FIG. 6, the content distribution server 10 includes a communication unit 11, a storage unit 12, and a control unit 14.

(About the communication unit 11)
The communication unit 11 is realized by, for example, a NIC (Network Interface Card) or the like. The communication unit 11 is connected to the network N by wire or wirelessly, and transmits and receives information to and from the terminal device 100, the external device 20, and the like.

(About the storage unit 12)
The storage unit 12 is realized by a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As illustrated in FIG. 6, the storage unit 12 includes a content information storage unit 13.

(About the content information storage unit 13)
The content information storage unit 13 stores information on content. For example, the content information storage unit 13 stores content (corresponding to “content C1” in the example of FIG. 2) corresponding to the subject of the user's action. Further, the content information storage unit 13 stores content that is information related to the subject of the behavior of the user and that indicates information recommended to the user. Further, the content information storage unit 13 stores content corresponding to the subject of the user's action to be recommended according to the user's action information and attribute information. Here, an example of information on the content stored in the content information storage unit 13 will be described with reference to FIG. FIG. 7 is a diagram illustrating an example of a content information storage unit according to the embodiment. In the example of FIG. 7, the content information storage unit 13 has items such as “content ID”, “content information”, “attribute information”, and “action history”.

  “Content ID” indicates identification information for identifying the content. “Content information” indicates information constituting the content. FIG. 7 illustrates an example in which conceptual information such as “content information 1” is stored in “content information”. However, in actuality, text information indicating the content of the content, image information, video information, or A file path name indicating the storage location is stored.

  The “attribute information” stores information such as a demographic attribute and a psychographic attribute of a user to whom the content stored in the “content information” is to be distributed. The “action history” stores the history of the action information of the user to whom the content stored in the “content information” is distributed.

  That is, in FIG. 7, “content information 1” identified by the content ID “C101” is distributed to the user who has the attribute corresponding to “attribute information 1” and has performed the action indicated in “action history 1”. Here is an example.

(About the control unit 14)
The control unit 14 is a controller. For example, various programs stored in a storage device inside the content distribution server 10 use a RAM as a work area by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). It is realized by being executed as. The control unit 14 is a controller, and is realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). As illustrated in FIG. 6, the control unit 14 according to the embodiment includes a reception unit 15, an acquisition unit 16, an estimation unit 17, and a distribution unit 18, and performs functions and operations of information processing described below. Realize or execute.

(About the reception unit 15)
The receiving unit 15 receives a content distribution request and behavior information from the terminal device 100.

(About the acquisition unit 16)
The acquisition unit 16 analyzes the behavior information received by the reception unit 15 from the terminal device 100 by appropriately using various conventional techniques such as image analysis and morphological analysis, thereby obtaining information (target information) on the behavior information. get. For example, the acquisition unit 16 acquires target information including an icon image (for example, an application icon or an icon image (web click icon) set for a web service) corresponding to the provider of the behavior information. The acquiring unit 16 acquires target information including text information, time information, position information, and the like by appropriately analyzing various types of conventional techniques such as image analysis and morphological analysis on the behavior information.

(About the estimating unit 17)
The estimating unit 17 estimates a subject corresponding to the behavior of the user based on behavior information regarding the behavior of the user. For example, the estimating unit 17 analyzes the behavior information received by the receiving unit 15 from the terminal device 100 by appropriately using various conventional techniques such as image analysis and morphological analysis, thereby obtaining text information included in the behavior information. And information such as time information and position information. Then, the estimating unit 17 estimates the subject of the user's behavior based on the information extracted from the behavior information.

(About the distribution unit 18)
The distribution unit 18 distributes the content stored in the content information storage unit 13 according to the attribute information and the action information of the user who uses the terminal device 100 when the distribution request of the content is received by the receiving unit 15. The distribution unit 18 distributes the target information acquired by the acquisition unit 16 to the terminal device 100. The distribution unit 18 distributes information about the subject to a terminal device used by the user and that performs display processing of information corresponding to the subject estimated by the estimation unit 17. For example, the distribution unit 18 distributes the information on the subject of the user's behavior estimated by the estimation unit 17 to the terminal device 100.

[4. Configuration of terminal device)
Next, the configuration of the terminal device 100 according to the embodiment will be described with reference to FIG. FIG. 8 is a diagram illustrating a configuration example of the terminal device according to the embodiment. As illustrated in FIG. 8, the terminal device 100 includes a communication unit 110, a storage unit 120, an input unit 130, an output unit 140, an imaging unit 150, and a control unit 160.

(About the communication unit 110)
The communication unit 110 is realized by, for example, an NIC or the like. The communication unit 110 is connected to the network N by wire or wirelessly, and transmits and receives information to and from the content distribution server 10, the external device 20, and the like.

(About the storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory device such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. As illustrated in FIG. 8, the storage unit 120 includes an action information storage unit 121.

(About the behavior information storage unit 121)
The behavior information storage unit 121 stores behavior information on the behavior of the user. Here, an example of the behavior information stored in the behavior information storage unit 121 will be described with reference to FIG. FIG. 9 is a diagram illustrating an example of the behavior information storage unit according to the embodiment. In the example of FIG. 9, the behavior information storage unit 121 has items such as “date”, “activity information”, and “analysis result information”.

  “Date” indicates the date on which the user using the terminal device 100 performed the action indicated by the information stored in the “action information”. “Action information” indicates information on an action performed by the user using the terminal device 100. Although FIG. 9 illustrates an example in which conceptual information such as “content information 1” is stored in “action information”, actually, a screen shot of the terminal device 100 or a user , Images and moving images taken by the user using the terminal device 100, e-mails transmitted and received by the user using the terminal device 100, applications and web services used by the user via the terminal device 100 And the like, or information such as bookmarked contents, or a file path name indicating the storage location.

  “Analysis result information” is information such as time information and position information extracted by appropriately using various conventional techniques such as image analysis and morphological analysis on information stored in “behavior information”. 4 shows information (target information in the example of FIG. 3).

  That is, in FIG. 9, an example in which the user using the terminal device 100 performs the action shown in “action information 1” on the date “20XX / 1/9” and “analysis result 1” is extracted from “action information 1” Is shown.

(About the input unit 130 and the output unit 140)
The input unit 130 is an input device that receives various operations from a user. For example, the input unit 130 is realized by a keyboard, a mouse, operation keys, and the like. The output unit 140 is a display device for displaying various information. That is, the output unit 140 corresponds to a display screen of the terminal device 100. For example, the output unit 140 is realized by a liquid crystal display or the like. When a touch panel is used for the terminal device 100, the input unit 130 and the output unit 140 are integrated.

(About the imaging unit 150)
The imaging unit 150 includes an optical system such as a lens and an imaging device such as a charge coupled device (CCD) image sensor and a complementary metal oxide semiconductor (CMOS) sensor.

(About the control unit 160)
The control unit 160 is a controller. For example, various programs (corresponding to an example of a display program) stored in a storage device inside the terminal device 100 are executed by a CPU or an MPU using a RAM as a work area. Is achieved. When the control information or the application for realizing the above-described display processing is distributed from the content distribution server 10, the control unit 160 operates according to the control information or the application and realizes the display processing.

  The control unit 160 is a controller, and is realized by, for example, an integrated circuit such as an ASIC or an FPGA. As illustrated in FIG. 8, the control unit 160 according to the embodiment includes an acquisition unit 161, a processing unit 162, an estimation unit 163, a search unit 164, a display unit 165, and a display control unit 166. The functions and actions of the information processing described below are realized or executed.

(About the acquisition unit 161)
The obtaining unit 161 obtains target information, which is information on a shooting target captured in the captured image, using the captured image. For example, in the example of FIG. 3, the acquisition unit 161 includes a screen shot of the screen of the terminal device 100, an image and a moving image captured by the user U1 using the terminal device 100, and an image viewed by the user U1 using the terminal device 100. Image analysis, morpheme analysis, and behavior information including moving images, e-mails sent and received by the user U1 using the terminal device 100, applications used by the user U1 via the terminal device 100, bookmarked contents in web services, and the like. The target information is acquired by appropriately performing analysis using various conventional techniques such as analysis.

  Further, the acquisition unit 161 may acquire target information indicating information on an application serving as a provider of a shooting target. For example, in the example of FIG. 3, the acquisition unit 161 acquires target information including information for specifying a provider of the behavior information (for example, an application or a web service).

  The acquisition unit 161 may acquire target information including an icon image indicating an application serving as a provider of a shooting target. For example, in the example of FIG. 3, the acquisition unit 161 includes a target including an icon image (for example, an application icon or an icon image (web click icon) set in a web service) corresponding to a provider of the behavior information. Get information.

  Further, the acquisition unit 161 may acquire the analysis result of the character string to be captured as the target information. For example, in the example of FIG. 3, the acquisition unit 161 analyzes text information, time information, position information, and the like by appropriately using various conventional techniques such as image analysis and morphological analysis on behavior information. Get target information including.

  The acquisition unit 161 acquires target information indicating information indicated by the two-dimensional code to be captured. Normally, a display screen of a ticket issuing code (for example, a QR code (registered trademark)) provided from a ticket sales service or the like displays time information on the date and time of the event corresponding to the ticket issuing code together with the ticket issuing code and the venue. It is assumed that information such as position information on the display is displayed. Therefore, the acquisition unit 161 analyzes the screen shot of the screen on which the two-dimensional code is displayed by using the image analysis to obtain target information such as time information indicating the information indicated by the two-dimensional code and position information. get.

  Note that the acquisition unit 161 may acquire the target information from the content distribution server 10. For example, the acquisition unit 161 transmits the behavior information to the content distribution server 10 together with the information for specifying the provider of the behavior information. The acquisition unit 161 acquires the target information by receiving from the content distribution server 10 the target information extracted by analyzing the information for identifying the provider of the behavior information and the behavior information by the content distribution server 10. You may.

(About the processing unit 162)
The processing unit 162 performs a process corresponding to the character string acquired as the target information. For example, the processing unit 162 executes a process corresponding to the character string acquired as the target information based on the current position of the user U1. For example, in the example of FIG. 3, when the behavior information (for example, the content related to the restaurant bookmarked by the user in the restaurant introduction service) includes the position information, the processing unit 162 determines the current position of the user U1 from the position information. If the distance is within a predetermined range (for example, 100 meters, 500 meters, etc.), the user U1 is notified of the action information.

  In addition, for example, in the example of FIG. 3, the processing unit 162 collects information about stores near the current position of the user U1 from an external device 20 or the like such as a service providing device that provides a service to the user U1. Then, if there is a store that handles a product corresponding to the user's action information (for example, a product corresponding to the thumbnails D6 to D9) in a store within a predetermined range from the current position of the user U1, the terminal device 100 determines that the user U1 To the action information.

  Further, the processing unit 162 may notify the user U1 that the target information has been updated based on the information provided by the provider of the behavior information. For example, in the example of FIG. 3, the processing unit 162 includes information on changes in the restaurant A corresponding to the activity information of the thumbnail D1 such as business hours and holidays, and information on the train delay corresponding to the activity information of the thumbnail D2. Alternatively, the user U1 may be notified of information on a change in the price of a product corresponding to the action information of the thumbnails D6 to D9. Further, the processing unit 162 may notify the user U1 of the information on the detour plan together with the notification of the delay information of various transportation means.

  Further, when the character string acquired as the target information indicates the time, the processing unit 162 may execute a process of notifying the user of the time. For example, in the example of FIG. 3, the processing unit 162 may perform a notification regarding the action information corresponding to the thumbnails D2 to D4 in which the target information includes the time information based on the time information.

(About the estimation unit 163)
The estimating unit 163 estimates a subject corresponding to the behavior of the user based on behavior information regarding the behavior of the user. For example, in the example of FIG. 1, the estimation unit 163 estimates a subject corresponding to the behavior of the user U1 based on the behavior information of the user U1 collected by the information collection application mounted on the terminal device 100.

  The estimating unit 163 may estimate the subject based on a part of the behavior information. For example, in the example of FIG. 1, the estimation unit 163 estimates the subject of the action of the user U1 based on the action information related to “Sendai” (action information corresponding to the thumbnails D1 to D3).

  The estimating unit 163 may estimate the subject based on the behavior information selected by the user from the behavior information. For example, in the example of FIG. 1, the estimating unit 163 estimates the subject of the behavior of the user U1 based on the behavior information corresponding to the thumbnails D1 to D3 selected by the user U1.

  Further, the estimating unit 163 may estimate the subject based on action information that is information stored according to a user operation. For example, in the example of FIG. 1, the estimation unit 163 includes a screen shot of the screen of the terminal device 100 acquired by the user U1 using the terminal device 100, an image and a moving image captured by the user U1 using the terminal device 100, Images and videos viewed by the user U1 using the terminal device 100, e-mails transmitted and received by the user U1 using the terminal device 100, applications used by the user U1 via the terminal device 100, contents bookmarked in web services, and the like. Based on the behavior information obtained by the active operation of the user U1, the theme of the behavior of the user U1 is estimated.

  Further, the estimating unit 163 may estimate the subject based on the behavior information including the information regarding the electronic mail. For example, in the example of FIG. 1, the estimation unit 163 is based on the behavior information including the e-mail regarding the reservation confirmation of the hotel B in Sendai city on “January 13” received by the user U1 using the terminal device 100. Then, the theme of the behavior of the user U1 is estimated.

  Further, the estimating unit 163 may estimate the subject based on action information including information based on an image recognition result of an image on the display screen of the application. For example, in the example of FIG. 1, the estimating unit 163 analyzes the screen shot of the display screen of the route search service on a predetermined application by appropriately using various conventional techniques to obtain text information, time information, and position information. Extract information such as information. Then, the estimating unit 163 estimates the subject of the action of the user U1 based on the action information including the extracted information.

  The estimating unit 163 may acquire behavior information including a result of a route search performed by the user, estimate a region visited by the user based on the result of the route search, and set the estimated region as a subject. For example, in the example of FIG. 1, the estimating unit 163 acquires action information including a screen shot of a display screen of a search result of the route search service performed by the user U1. Then, the estimating unit 163 sets the subject “Sendai travel” related to the destination “Sendai” extracted by analyzing the screen shot of the display screen by appropriately using various conventional techniques as the subject of the action of the user U1. I do.

  The estimating unit 163 may estimate the subject based on the behavior information selected by the user from the behavior information. For example, in the example of FIG. 1, the estimating unit 163 estimates the subject of the behavior of the user U1 based on the behavior information corresponding to the thumbnails D1 to D3 selected by the user U1.

  The estimating unit 163 may acquire information such as text information, time information, and position information included in the behavior information from the content distribution server 10. For example, the estimating unit 163 transmits the behavior information to the content distribution server 10. The estimating unit 163 receives information such as text information, time information, and position information extracted by analyzing the behavior information by the content distribution server 10 from the content distribution server 10, and based on the received information, the user's behavior. To estimate the subject. Further, the estimating unit 163 may acquire information on the subject of the user's behavior estimated by the content distribution server 10 (the estimating unit 17) based on the behavior information transmitted to the content distribution server 10. The estimating unit 163 may synchronize the subject of the user's behavior estimated by the estimating unit 163 with the subject of the user's behavior estimated by the content distribution server 10.

(About the search unit 164)
The search unit 164 searches for information corresponding to the subject estimated by the estimation unit 163. For example, in the example of FIG. 2, the search unit 164 retrieves information corresponding to the theme of the action “Sendai trip” from the action information storage unit 121 or an external device 20 such as a service providing apparatus that provides a service to the user U1. Search for.

  Further, the search unit 164 may search the behavior information for information corresponding to the subject. For example, in the example of FIG. 2, the search unit 164 searches for action information that is action information collected by the information collection application mounted on the terminal device 100 and that is related to the subject of the action, “Sendai Trip”.

  Further, the search unit 164 may search for information on the web corresponding to the subject. For example, in the example of FIG. 2, the search unit 164 searches for information corresponding to the theme of the action “Sendai trip” from a web page or various applications installed on the terminal device 100.

  In addition, the search unit 164 may search for information on a facility corresponding to the subject and within a predetermined range from a position corresponding to the behavior information selected by the user. For example, when the behavior information selected by the user includes the position information, the search unit 164 searches the facility information corresponding to the subject within a predetermined range (for example, 100 meters or 500 meters) from the position information. Search for.

  The search unit 164 searches for facility information based on the search history of the user. For example, the search unit 164 searches for facility information based on the search history of the user included in the behavior information collected by the information collection application mounted on the terminal device 100.

  The search unit 164 may search for information corresponding to the subject of the user's behavior indicated by the information, based on the information on the subject of the user's behavior acquired by the estimating unit 163 from the content distribution server 10. For example, when the information on the subject of the user's behavior acquired by the estimating unit 163 from the content distribution server 10 indicates “Sendai travel”, the search unit 164 stores information corresponding to “Sendai travel” in the behavior information storage unit. 121 or an external device 20 such as a service providing device that provides a service to the user U1.

(About the display unit 165)
The display unit 165 displays at least content based on the information searched by the search unit 164. For example, in the example of FIG. 2, the display unit 165 is configured based on the external device 20 such as a service providing device that provides a service to the user U1 or the behavior information of the user U1 collected by the information collecting application mounted on the terminal device 100. The content C1 based on the information corresponding to the theme of the action “Sendai trip” searched by the search unit 164 is displayed.

  Further, the display unit 165 may display a content based on the behavior information on the behavior of the user and the information searched by the search unit 164. For example, in the example of FIG. 2, the display unit 165 displays the content C11, the content C12, and the content C13 corresponding to the behavior information collected by the information collection application mounted on the terminal device 100, and the information searched by the search unit 164. The content C1 based on the corresponding content C17 is displayed.

  The display unit 165 may display the content in a mode according to the situation of the user. For example, in the example of FIG. 2, the display unit 165 displays the content C1 when the current time is morning (for example, from 5:00 to 8:00) or night (for example, from 18:00 to 24:00), and the user U1 Since it is assumed that the content C1 is browsed for the purpose of checking the schedule of the day or the schedule of the next day, the content C1 is displayed in a timeline format (the first state, the second state, and the fourth state in FIG. 2) or the like. , In the form of an album (the sixth state in FIG. 2).

  In addition, for example, the display unit 165 displays the content in a mode according to the situation of the user estimated based on sensor information detected by various sensors mounted on the terminal device 100. For example, in the example of FIG. 2, when the state of the user is estimated to be “walking state” by various sensors mounted on the terminal device 100, the display unit 165 converts the content C1 into a map format (the (5 states).

  The display unit 165 displays the content in a mode according to the subject. For example, in the example of FIG. 2, the display unit 165 displays the content C1 in the form of an imline corresponding to the travel and accepts an editing operation because the estimating unit 163 estimates the subject of the action of the user U1 as “Sendai travel”. . The display unit 165 may display the content in various modes according to the subject of the user's action estimated by the estimation unit 163. Here, an example in which content is displayed in the form of an album will be described with reference to FIG. FIG. 10 is a diagram (5) illustrating an example of the terminal device according to the embodiment.

  In the example of FIG. 10, in the e-commerce service, the action information on the product (shoe H to shoe K) bookmarked by the user U1 is selected, and the estimating unit 163 sets the subject of the action of the user U1 to "Purchasing shoes. (Notes you want) ". Then, in the first state of FIG. 10 (corresponding to the fifth state of FIG. 1), it is assumed that the display unit 165 displays the content corresponding to the subject “want memo” of the action of the user U1 in the area AR11.

  In the first state of FIG. 10, the display unit 165 causes the screen to transition to the second state by accepting the user U1's selection operation on the content corresponding to the theme of the action “desired memo”. The second state in FIG. 10 shows a state where information corresponding to the behavior information selected by the user is displayed. In the example of FIG. 10, the display unit 165 displays the contents C21 to C24 corresponding to the behavior information selected by the user on the screen. The display unit 165 displays an area AR22 on the screen for the user U1 to change the display mode of the contents C21 to C24.

  In the second state of FIG. 10, the display unit 165 displays contents C21 to C24 including an image of a product obtained from a provider of the behavior information, information on a price of the product, a discount, and the like. In the second state of FIG. 10, the display unit 165 relates to shoes that are to be considered for purchase in the subject of the behavior of the user U1 based on the demographic attribute and the psychographic attribute based on the behavior information of the user U1. Information may be displayed.

  In the second state of FIG. 10, when the area AR22 is selected, the display unit 165 displays map information indicating the position of an actual store that sells products corresponding to the contents C21 to C24 on the screen. Is also good. For example, the display unit 165 displays map information indicating the position of an actual store that sells products corresponding to the contents C21 to C24 within a predetermined range (for example, 100 meters, 500 meters, or the like) from the current position of the user U1. Display on the screen.

  Subsequently, the third state in FIG. 10 shows a state in which information corresponding to the action information selected by the user U1 is displayed. For example, in the example of FIG. 10, the display unit 165 displays a product page corresponding to the behavior information selected by the user U1 in the electronic commerce service on a browser included in the information collection application.

  The display unit 165 may extract time information or position information from the information corresponding to the subject, and may display a content in which information corresponding to the subject is arranged at a display position based on the time information or the position information. For example, in the example of FIG. 2, the display unit 165 extracts time information from the behavior information corresponding to the content included in the content C1, arranges the content at a display position corresponding to the extracted time information, and displays the content on the screen. . In the example of FIG. 2, the display unit 165 sets “Sendai Station” as a departure point, and extracts position information from behavior information or the like corresponding to each content included in the content C1. Then, the display unit 165 arranges each content on the content C1 according to the distance of the extracted position information from “Sendai Station” and displays the content on the screen.

  The display unit 165 may display a content including map information in which information corresponding to a subject is arranged based on the position information. For example, in the example of FIG. 2, the display unit 165 extracts position information from behavior information or the like corresponding to the content included in the content C1, and places each content at a position corresponding to the position information extracted on the map of Sendai City. Place and display on screen.

  In addition, the display unit 165 may display a content in which information corresponding to a subject is arranged in chronological order based on time information or position information. For example, in the example of FIG. 2, the display unit 165 extracts time information or position information from behavior information or the like corresponding to the content included in the content C1, and arranges each content included in the content C1 in the area AR21. Display on the screen.

  The display unit 165 may display the behavior information selected by the user and information on the web. For example, in the example of FIG. 2, the display unit 165 displays the content C11, the content C12, and the content C13 corresponding to the action information selected by the user U1, and the content C1 including the content corresponding to the information searched by the search unit 164. indicate.

  The display unit 165 may display the recommendation of the subject of the action. Here, an example in which the recommendation of the theme of the behavior is displayed with reference to FIG. FIG. 11 is a diagram (6) illustrating an example of the terminal device according to the embodiment.

  In the example of FIG. 11, it is assumed that the behavior information of the user U1 regarding the movie L has been collected by the information collection application mounted on the terminal device 100. For example, in the example of FIG. 11, as the behavior information on the movie L, an image and a moving image on the movie L viewed by the user U1 using the terminal device 100, a bookmark in a web service on the movie theater M showing the movie L, and the like. It is assumed that the information is collected by the information collection application. In the example of FIG. 11, it is assumed that the estimating unit 163 estimates the subject of the action of the user U1 as “movie L” based on the action information on the movie L. Then, in the first state of FIG. 11 (corresponding to the fifth state of FIG. 1), it is assumed that the display unit 165 displays the content corresponding to the theme “movie L” of the action recommended to the user U1 in the area AR11.

  The first state in FIG. 11 shows a state in which the recommendation of the subject of the action is displayed. In the first state of FIG. 11, the display unit 165 displays the content corresponding to the theme of the action “Movie L” in a reduced state with the text “Would you like to go to a movie this weekend?” indicate. Here, the display unit 165 causes the screen to transition to the second state by accepting the selection operation of the user U1 for the content corresponding to the theme “movie L” of the action.

  Subsequently, the second state in FIG. 11 shows a state in which content related to the theme “movie L” of the action is displayed. For example, in the example of FIG. 11, the search unit 164 searches for information related to the subject “movie L” of the action. Then, the display unit 165 extracts time information from the information searched by the search unit 164, and displays the content C25 to C27 indicating the information with reduced visibility at a display position corresponding to the extracted time information. Further, also in the first state of FIG. 11, the terminal device 100 displays the area AR22 and the area AR23. Here, the content C25 indicates information on transfer guidance from the residence area “Shimokitazawa” of the user U1 to the location “Shinjuku” of the movie theater M in the route search service. In addition, the content C26 indicates information on the duration of the movie L in the movie theater M. The content C27 is a result of the search unit 164 searching for information related to the area “Shinjuku” visited by the user U1 in the theme “Movie L” of the action (information about the restaurant N whose location is “Shinjuku”). Show.

  In the second state of FIG. 11, when a predetermined operation (for example, a tap or a long tap) is performed on the contents C25 to C27, an application corresponding to each content, a web service, a content, and the like are displayed. Is also good. For example, when a predetermined operation is performed on the content C25, the display unit 165 starts the route search service in a state where parameter information corresponding to the content C25 is input. Then, when the parameter information is changed (for example, the arrival time is changed), the display unit 165 displays the content C25 at the display position corresponding to the changed parameter information in the second state of FIG. Further, for example, when a predetermined operation is performed on the content C26 or C27, the display unit 165 displays a page related to the reservation of the movie L or the restaurant N in the facility reservation service. Further, for example, when a predetermined operation is performed on the content C27, the display unit 165 corresponds to information related to the area “Shinjuku” visited by the user U1 and different from information related to the restaurant N. The content is displayed on the screen as content recommended for the user U1.

  Subsequently, the third state in FIG. 11 shows a state in which the content displayed in the second state is selected by the user U1. In the example of FIG. 11, it is assumed that the contents C25 to C27 have been selected by the user U1. In this case, the display unit 165 displays the contents C25 to C27 with enhanced visibility. Then, when the area AR23 is selected in the third state of FIG. 11, the display unit 165 stores the content related to the theme “movie L” of the action in the storage unit 120, and shifts the screen to the fourth state. .

  Subsequently, the fourth state in FIG. 11 illustrates a state in which the recommendation of the subject of the action is selected by the user U1. As shown in the fourth state in FIG. 11, the display unit 165 displays the content corresponding to the theme “movie L” of the action in a state in which the visibility is enhanced.

  The display unit 165 may display the target information acquired by the acquisition unit 161 in a manner superimposed on the captured image. For example, in the example of FIG. 3, the display unit 165 displays an area AR18 for displaying the target information acquired by the acquisition unit 161 so as to be superimposed on each thumbnail.

  In addition, the display unit 165 may display target information updated based on information provided from an application serving as a provider of a shooting target so as to be superimposed on the shot image. For example, in the example of FIG. 3, the display unit 165 displays information on changes in the business hours and holidays of the restaurant provided from the provider of the shooting target, delay information on various types of transportation, and changes in prices of products. The target information updated by the information about the thumbnails is superimposed on each thumbnail and displayed.

  In addition, when a captured image is selected, the display unit 165 may display a screen corresponding to the imaging target, which is provided by an application that provides the imaging target. For example, in the example of FIG. 3, the display unit 165 displays the same information as the parameter information input when the user U1 acquires the action information corresponding to the selected thumbnail, and provides the source of the action information. Display applications and web services that are.

  Further, the display unit 165 may display the third content in which the second content is arranged so as to be superimposed on the first content. For example, in the example of FIG. 4, the display unit 165 displays a related content area in which related content is displayed so as to be superimposed on the content C2.

  The display unit 165 may display the third content in which a plurality of second contents are arranged so as to overlap the first content. For example, in the example of FIG. 4, the display unit 165 superimposes and displays the related content area including the content C19 and the content C20 indicating the information recommended for the user U1 on the content C2.

  In addition, when the fourth content included in the first content is selected, the display unit 165 may display the third content. For example, in the example of FIG. 4, when the display unit 165 receives a selection operation performed by the finger F10 of the user U1 on the content arranged in the content C2, the display unit 165 stores the content in the state of being held by the finger F10 of the user U1. The area is superimposed on the content C2 and displayed.

  The display unit 165 may display the third content while maintaining the position of the selected fourth content. For example, in the example of FIG. 4, when the display unit 165 receives a selection operation by the finger F10 of the user U1 with respect to the content arranged in the content C2 and is different from the moving operation, the display unit 165 displays the content in the user U1. The related content area is displayed while being superimposed on the content C2 while being held by the finger F10.

(About the display control unit 166)
When the second content is selected, the display control unit 166 reduces the area where the third content is displayed, while displaying the first content and the selected second content. For example, in the example of FIG. 4, the display control unit 166 reduces the related content area while displaying the content C11 on which the selection operation has been performed by the finger F10 of the user U1.

  In addition, the display control unit 166 may display the second content in a manner superimposed on the first content. For example, in the example of FIG. 4, the display control unit 166 superimposes and displays the content C11 on which the selection operation has been performed by the finger F10 of the user U1 on the content C2.

  In addition, the display control unit 166 may superimpose the selected second content on the first content in a state where the display mode of the selected second content is changed. For example, the display control unit 166 superimposes and displays the second content in a state where vibration, blinking, coloring, size up, size down, brightness up, brightness down, addition of a marking, and the like have been performed on the first content. Let it.

  Further, when the second content is selected, the display control unit 166 may display the first content and the third content in a changed state. For example, the display control unit 166 displays the first content and the third content in a state where vibration, blinking, coloring, size up, size down, brightness up, brightness down, addition of marking, and the like have been performed.

  In addition, the display control unit 166 may display the selected second content so as to be superimposed on the first content in a state where the visibility of the selected second content is enhanced. For example, in the example of FIG. 4, the display control unit 166 displays the content C11 on which the selection operation has been performed with the finger F10 of the user U1 while superimposing the content C11 on the content C2 in a state where the visibility is enhanced.

  In addition, the display control unit 166 may superimpose the selected second content on the first content in a state where the visibility of the selected second content is reduced. For example, in the example of FIG. 4, the display control unit 166 superimposes and displays the content C2 on which the selection operation has been performed with the finger F10 of the user U1 with the content C11 reduced.

  Further, the display control unit 166 may display the first content and the third content in a state where the visibility is reduced. For example, in the example of FIG. 4, the display control unit 166 displays the content C2 and the related content area in a reduced visibility state after the selection operation on the content C11 is performed by the finger F10 of the user U1.

  Further, the display control unit 166 may gradually reduce the area where the third content is displayed. For example, in the example of FIG. 4, the display control unit 166 gradually reduces the related content area after the selection operation on the content C11 is performed by the finger F10 of the user U1.

  In addition, the display control unit 166 may gradually enlarge an area where the first content is displayed. For example, in the example of FIG. 4, the display control unit 166 gradually enlarges the content C2 after the selection operation on the content C11 is performed by the finger F10 of the user U1.

  In addition, the display control unit 166 may reduce an area where the third content is displayed while holding the position of the second content. For example, in the example of FIG. 4, the display control unit 166 gradually reduces the related content area while the content C11 is held by the finger F10 of the user U1.

  Further, when the second content is continuously selected, the display control unit 166 may reduce the area where the third content is displayed. For example, in the example of FIG. 4, when an operation such as a long tap is performed on the content C11 with the finger F10 of the user U1, the display control unit 166 reduces the related content area.

  In addition, when the selection position at which the second content is selected is moved in a predetermined direction, the display control unit 166 may reduce the area where the third content is displayed. For example, in the example of FIG. 4, when the selection operation on the content C11 is performed by the finger F10 of the user U1 and the content C11 is moved in a predetermined direction, the display control unit 166 reduces the related content area.

  In addition, the display control unit 166 may move the display position of the second content according to the movement of the selected position where the second content is selected. For example, in the example of FIG. 4, the display control unit 166 moves the display position of the content C11 according to a drag operation on the content C11 by the finger F10 of the user U1.

  In addition, when the selection of the second content is released, the display control unit 166 may display the second content by superimposing the second content at a position corresponding to the position where the selection of the second content is released in the first content. Good. For example, in the example of FIG. 4, the display control unit 166 displays the content C11 at a position in the content C2 where the selection operation on the content C11 by the finger F10 of the user U1 is released.

  When any of the second contents is selected, the display control unit 166 reduces the area in which the third content is displayed while keeping the first content and the selected second content displayed. Is also good. For example, in the example of FIG. 4, when any one of the content C11, the content C19, and the content C20 displayed in the related content area is selected, the display control unit 166 sets the related content while displaying the selected content. Reduce the content area.

  Further, when a predetermined operation is performed on the fourth content included in the first content, the display control unit 166 may hide the four content in an area where the first content is displayed. For example, in the example of FIG. 4, when receiving a predetermined operation (for example, a flick operation) on the content arranged in the content C2, the display control unit 166 deletes the content from the content C2.

  Also, when the selection of the fourth content is released in the area where the third content is displayed, the display control unit 166 hides the fourth content in the area where the first content is displayed, and displays the third content. The fourth content may be displayed in the region to be displayed. For example, in the example of FIG. 4, the display control unit 166 determines that the content is selected on the related content area displayed by the display unit 165 in response to the selection operation of the user U1 with the finger F10 on the content arranged in the content C2. When released, the content is deleted from the content C2 and displayed in the related content area.

  Also, when the selection of the fourth content is released in the area where the third content is displayed, the display control unit 166 hides the fourth content in the area where the first content is displayed, and displays the third content. The area in which the third content is displayed may be reduced while the fourth content is displayed in the area to be displayed. For example, in the example of FIG. 4, when the selection of the content arranged in the content C2 by the finger F10 of the user U1 is released on the related content area, the display control unit 166 deletes the content from the content C2, The related content area is reduced while being displayed in the content area.

[5. Display processing flow)
A procedure of a content display process of the display processing system 1 according to the embodiment will be described with reference to FIG. FIG. 12 is a flowchart (1) illustrating an example of a display processing procedure according to the embodiment.

  As shown in FIG. 12, the terminal device 100 estimates the subject of the user's behavior based on the user's behavior information (step S101). Subsequently, the terminal device 100 searches for information corresponding to the subject of the user's action (Step S102). For example, the terminal device 100 may determine the subject of the user's behavior on the web (a web page or various applications installed on the terminal device 100) or from the user's behavior information collected by the information collecting application mounted on the terminal device 100. Search for information corresponding to.

  Subsequently, the terminal device 100 displays the content based on the searched information (step S103), and ends the processing.

  Next, a procedure of display processing of target information of the display processing system 1 according to the embodiment will be described with reference to FIG. FIG. 13 is a flowchart (2) illustrating an example of a display processing procedure according to the embodiment.

  As illustrated in FIG. 13, the terminal device 100 acquires target information from behavior information (Step S201). For example, the terminal device 100 specifies the source of the text information, the time information, the position information, and the behavior information by appropriately analyzing the behavior information using various conventional techniques such as image analysis and morphological analysis. Information, icon images, etc. corresponding to the sources of the behavior information are acquired.

  Subsequently, the terminal device 100 displays the target information superimposed on the captured image (Step S202). For example, the terminal device 100 displays the target information superimposed on a screen shot or the like of the terminal device 100 acquired as the behavior information.

  Subsequently, the terminal device 100 executes a process corresponding to the target information (Step S203), and ends the process. For example, when the target information includes time information, the terminal device 100 sets an alarm to notify the user based on the time information.

  Next, a procedure of a content display process of the display processing system 1 according to the embodiment will be described with reference to FIG. FIG. 14 is a flowchart (3) illustrating an example of a display processing procedure according to the embodiment.

  As shown in FIG. 14, the terminal device 100 displays the first content (Step S301). For example, in the example of FIG. 4, the terminal device 100 displays the content C2. Subsequently, when an operation of instructing a display process of the third content is performed (Step S302; Yes), the terminal device 100 displays the third content in which the second content is arranged so as to be superimposed on the first content. (Step S303). For example, in the example of FIG. 4, the terminal device 100 displays the related content area in which the related content is arranged so as to be superimposed on the content C2. On the other hand, when the operation of instructing the display processing of the third content is not performed (Step S302; No), the terminal device 100 displays the first content as it is. Then, when the operation of instructing the storage processing of the first content is performed (Step S306; Yes), the terminal device 100 stores the first content (Step S307), and ends the processing. In addition, when the operation of instructing the storage processing of the first content is not performed (Step S306; No), the terminal device 100 continuously displays the first content.

  Subsequently, when the second content is selected (Step S304; Yes), the terminal device 100 reduces the area in which the third content is displayed while the first content and the second content are displayed (Step S305). . On the other hand, when the second content is not selected (Step S304; No), the terminal device 100 reduces the area in which the third content is displayed and displays the first content. For example, when a predetermined operation (for example, a swipe operation) is performed on the third content, the terminal device 100 reduces the area in which the third content is displayed and displays the first content. Then, when the operation of instructing the storage processing of the first content is performed (Step S306; Yes), the terminal device 100 stores the first content (Step S307), and ends the processing. In addition, when the operation of instructing the storage processing of the first content is not performed (Step S306; No), the terminal device 100 continuously displays the first content.

  Subsequently, when an operation for instructing a storage process of the first content is performed (Step S306; Yes), the first content is stored (Step S307), and the process ends. For example, the terminal device 100 stores the first content in which the second content is arranged. On the other hand, when the operation of instructing the storage processing of the first content is not performed (Step S306; No), the terminal device 100 continuously displays the first content in which the second content is arranged.

[6. Modification)
The above-described embodiment is an example, and various modifications and applications are possible.

[6-1. About action information)
In the above-described embodiment, the behavior information is a screen shot of the screen of the terminal device 100 acquired by the user U1 using the terminal device 100, an image and a moving image captured by the user U1 using the terminal device 100, Examples of images and moving images browsed using the terminal device 100, e-mails transmitted and received by the user U1 using the terminal device 100, applications used by the user U1 via the terminal device 100, contents bookmarked in web services and the like. However, the action information is not limited to the example described above. The behavior information may be any information as long as it is information on the behavior of the user that can be obtained by the information processing device or the like.

[6-2. Display Unit 165 and Display Control Unit 166]
In the example of FIG. 4 described above, an example in which the display unit 165 and the display control unit 166 superimpose and display the related content area for displaying the related content on the screen has been described, but the functions of the display unit 165 and the display control unit 166 are described. Is not limited to the example described above. The second content may be any information as long as it can be added to the first content. For example, the display unit 165 displays the message application as the first content, and superimposes the third content in which the second content, which is information such as a text, an image, and a moving image, which can be added to the message, is superimposed on the first content. indicate. Then, the display control unit 166 performs the above-described display control in the message application.

[6-3. Display of Content Based on User Behavior Pattern]
The display unit 165 may display content based on the user's behavior pattern. For example, when behavior information indicating that the user is continuously visiting the region O is collected, the estimation unit 163 estimates the subject of the user's behavior as “visit to the region O”. Then, the display unit 165 displays a content including information about the area O searched by the search unit 164 (facilities such as restaurants around the area O). Further, the display unit 165 may change and display the content of the content every time the content related to the predetermined area is displayed. The display unit 165 may display the content to the user based on information about the evaluation of the content after the user has actually performed an action corresponding to the content displayed on the display unit 165.

[6-4. Display of content related to behavior information)
The display unit 165 may display content related to the collected user behavior information. For example, when the user's behavior information includes information such as concert viewing and event participation, the display unit 165 displays content related to the next work of the concert and content related to holding of the event in the next fiscal year. .

[7. effect〕
As described above, the terminal device 100 according to the embodiment includes the acquisition unit 161, the processing unit 162, the estimation unit 163, the search unit 164, the display unit 165, and the display control unit 166. The obtaining unit 161 obtains target information, which is information on a shooting target captured in the captured image, using the captured image. The processing unit 162 performs a process corresponding to the character string acquired as the target information. The estimating unit 163 estimates a subject corresponding to the behavior of the user based on behavior information regarding the behavior of the user. The search unit 164 searches for information corresponding to the subject estimated by the estimation unit 163. The display unit 165 displays at least content based on the information searched by the search unit 164. In addition, the display unit 165 displays the target information acquired by the acquisition unit 161 so as to be superimposed on the captured image. In addition, the display unit 165 displays the third content in which the second content is arranged so as to overlap the first content. When the second content is selected, the display control unit 166 reduces the area where the third content is displayed, while displaying the first content and the selected second content.

  Thereby, the terminal device 100 according to the embodiment can provide information based on the guideline of the user's behavior, because the terminal device 100 displays the content based on the information corresponding to the subject of the user's behavior. Further, the terminal device 100 according to the embodiment displays the target information on the shooting target so as to be superimposed on the shot image, so that usability can be improved. Further, the terminal device 100 according to the embodiment controls the display of the content according to the case where the content dynamically changes, so that usability can be improved.

  In the terminal device 100 according to the embodiment, for example, the estimation unit 163 estimates a subject based on a part of the behavior information. Then, the search unit 164 searches the behavior information for information corresponding to the subject.

  Accordingly, the terminal device 100 according to the embodiment estimates the subject of the user's behavior based on the relevant behavior information among the behavior information, and searches for information to be provided to the user from the user's behavior information. And the information with high appealing power can be provided to the user.

  In the terminal device 100 according to the embodiment, for example, the estimating unit 163 estimates a subject based on the behavior information selected by the user from the behavior information.

  Thereby, the terminal device 100 according to the embodiment can provide information according to the user's request because the subject of the user's behavior is estimated based on the behavior information selected by the user.

  Further, in the terminal device 100 according to the embodiment, for example, the display unit 165 displays a content based on the behavior information on the behavior of the user and the information searched by the search unit 164.

  Accordingly, the terminal device 100 according to the embodiment can provide the user with highly appealing information because the content corresponding to the subject of the user's behavior is displayed.

  Further, in the terminal device 100 according to the embodiment, for example, the display unit 165 displays the content in a mode according to the situation of the user.

  Accordingly, the terminal device 100 according to the embodiment displays the content in accordance with the time zone in which the user browses the content and the moving state of the user, so that usability can be improved.

  Further, in the terminal device 100 according to the embodiment, for example, the display unit 165 displays the content in a mode according to the subject.

  Accordingly, the terminal device 100 according to the embodiment can display the content in a mode according to the subject of the user's action, and thus can improve usability.

  In the terminal device 100 according to the embodiment, for example, the display unit 165 extracts time information or position information from the information corresponding to the subject, and outputs the information corresponding to the subject to the display position based on the time information or the position information. Display the content where is placed.

  Accordingly, the terminal device 100 according to the embodiment can display the content based on the time information or the position information, and thus can improve the usability.

  Further, in the terminal device 100 according to the embodiment, for example, the display unit 165 displays a content including map information in which information corresponding to a subject is arranged based on the position information.

  Accordingly, the terminal device 100 according to the embodiment provides information to the user along with the position on the map, and thus can improve usability.

  Further, in the terminal device 100 according to the embodiment, for example, the display unit 165 displays a content in which information corresponding to a subject is arranged in a time-series order based on time information or position information.

  Accordingly, the terminal device 100 according to the embodiment displays information in chronological order, so that usability can be improved.

  In the terminal device 100 according to the embodiment, for example, the estimating unit 163 estimates a subject based on action information that is information stored according to a user operation.

  Accordingly, the terminal device 100 according to the embodiment can provide information according to the user's request because the terminal device 100 estimates the subject of the user's behavior based on the behavior information stored by the user.

  In the terminal device 100 according to the embodiment, for example, the estimating unit 163 estimates a subject based on action information including information on electronic mail.

  Accordingly, the terminal device 100 according to the embodiment estimates the subject of the user's action including the e-mail, so that usability can be improved.

  In the terminal device 100 according to the embodiment, for example, the estimating unit 163 estimates a subject based on action information including information based on an image recognition result of an image on a display screen of an application.

  Accordingly, the terminal device 100 according to the embodiment estimates the subject of the user's behavior including the display screen of the application, and thus can improve usability.

  In the terminal device 100 according to the embodiment, for example, the estimation unit 163 acquires behavior information including a result of a route search performed by the user, estimates a region visited by the user based on the result of the route search, The estimated area is the subject.

  Accordingly, the terminal device 100 according to the embodiment can provide the user with information having a high appealing power, because the terminal device 100 provides the information on the visit area of the user.

  In the terminal device 100 according to the embodiment, for example, the estimating unit 163 estimates a subject based on the behavior information selected by the user from the behavior information. Then, the search unit 164 searches for information on the web corresponding to the subject. Then, the display unit 165 displays the action information selected by the user and the information on the web.

  Accordingly, the terminal device 100 according to the embodiment searches for and displays information related to the subject of the user's action on the web, and thus can improve usability.

  In addition, in the terminal device 100 according to the embodiment, for example, the search unit 164 searches for information on facilities corresponding to the subject and within a predetermined range from a position corresponding to the behavior information selected by the user. I do.

  Accordingly, the terminal device 100 according to the embodiment displays facilities related to the subject of the user's behavior, and thus can improve usability.

  In the terminal device 100 according to the embodiment, for example, the search unit 164 searches for facility information based on the search history of the user.

  Thereby, the terminal device 100 according to the embodiment displays the facility based on the search history of the user, so that usability can be improved.

  In addition, in the terminal device 100 according to the embodiment, for example, the acquisition unit 161 acquires target information indicating information on an application serving as a provider of a shooting target. Further, for example, the acquisition unit 161 acquires target information including an icon image indicating an application serving as a provider of a shooting target. In addition, for example, the display unit 165 displays target information updated based on information provided from an application that is a provider of a shooting target so as to be superimposed on a shot image.

  Accordingly, the terminal device 100 according to the embodiment displays the captured image in a state where the information regarding the provider of the imaging target can be visually recognized, thereby improving usability.

  In addition, in the terminal device 100 according to the embodiment, for example, when a captured image is selected, the display unit 165 displays a screen corresponding to the capturing target, which is provided by an application that is a provider of the capturing target. I do.

  Thereby, the terminal device 100 according to the embodiment displays the screen of the application that is the provider of the shooting target, so that usability can be improved.

  In the terminal device 100 according to the embodiment, for example, the acquisition unit 161 acquires an analysis result of a character string to be captured as target information. Further, for example, the acquisition unit 161 acquires target information indicating information indicated by a two-dimensional code to be captured.

  Accordingly, the terminal device 100 according to the embodiment displays the captured image in a state where the information of the imaging target can be visually recognized, thereby improving usability.

  In the terminal device 100 according to the embodiment, for example, when the character string acquired as the target information indicates a time, the processing unit 162 performs a process of notifying the user of the time.

  Accordingly, the terminal device 100 according to the embodiment executes the notification regarding the imaging target based on the time information included in the imaging target, and thus can improve usability.

  In the terminal device 100 according to the embodiment, for example, the display control unit 166 causes the second content to be displayed while being superimposed on the first content.

  Accordingly, the terminal device 100 according to the embodiment displays the content according to the user's intuitive operation, and thus can improve usability.

  In the terminal device 100 according to the embodiment, for example, the display control unit 166 superimposes and displays the selected second content on the first content in a changed state. For example, when the second content is selected, the display control unit 166 displays the first content and the third content in a changed state. Further, for example, the display control unit 166 superimposes and displays the selected second content on the first content in a state where visibility is enhanced. Further, for example, the display control unit 166 superimposes and displays the selected second content on the first content in a state where the visibility of the selected second content is reduced. Further, for example, the display control unit 166 displays the first content and the third content in a state where the visibility is reduced.

  Accordingly, the terminal device 100 according to the embodiment displays the content according to the user's intuitive operation, and thus can improve usability.

  Further, in the terminal device 100 according to the embodiment, for example, the display control unit 166 gradually reduces the area where the third content is displayed. Further, for example, the display control unit 166 gradually enlarges an area where the first content is displayed.

  Accordingly, the terminal device 100 according to the embodiment displays the content according to the user's intuitive operation, and thus can improve usability.

  In the terminal device 100 according to the embodiment, for example, the display control unit 166 reduces the area where the third content is displayed while holding the position of the second content. Further, for example, when the second content is continuously selected, the display control unit 166 reduces the area where the third content is displayed. Further, for example, when the selection position where the second content is selected is moved in a predetermined direction, the display control unit 166 reduces the area where the third content is displayed.

  Accordingly, the terminal device 100 according to the embodiment displays the content according to the user's intuitive operation, and thus can improve usability.

  In the terminal device 100 according to the embodiment, for example, the display control unit 166 moves the display position of the second content according to the movement of the selected position where the second content is selected. Further, for example, when the selection of the second content is released, the display control unit 166 superimposes and displays the second content at a position corresponding to the position at which the selection of the second content is released among the first contents. .

  Accordingly, the terminal device 100 according to the embodiment displays the content according to the user's intuitive operation, and thus can improve usability.

  In the terminal device 100 according to the embodiment, for example, the display unit 165 displays the third content in which a plurality of second contents are arranged so as to overlap the first content. Then, when any of the second contents is selected, the display control unit 166 reduces the area in which the third content is displayed, while displaying the first content and the selected second content.

  Accordingly, the terminal device 100 according to the embodiment displays the content according to the user's intuitive operation, and thus can improve usability.

  Further, in the terminal device 100 according to the embodiment, for example, when a predetermined operation is performed on the fourth content included in the first content, the display control unit 166 determines whether the fourth content is displayed in the area where the first content is displayed. Hide the content. Further, for example, when the fourth content included in the first content is selected, the display unit 165 displays the third content. Then, when the selection of the fourth content is released in the area where the third content is displayed, the display control unit 166 hides the fourth content in the area where the first content is displayed, and displays the third content. The fourth content is displayed in the region to be displayed. Further, for example, the display unit 165 displays the third content while maintaining the position of the selected fourth content. Further, for example, the display control unit 166 hides the fourth content in an area where the first content is displayed, and displays the third content while displaying the fourth content in an area where the third content is displayed. Reduce the area.

  Accordingly, the terminal device 100 according to the embodiment displays the content according to the user's intuitive operation, and thus can improve usability.

  Further, as described above, the content distribution server 10 according to the embodiment includes the reception unit 15, the acquisition unit 16, the estimation unit 17, and the distribution unit 18. The receiving unit 15 receives a content distribution request and behavior information from the terminal device 100. The acquisition unit 16 acquires information on the behavior information by appropriately analyzing the behavior information received by the reception unit 15 from the terminal device 100 using various conventional techniques such as image analysis and morphological analysis. The estimating unit 17 estimates a subject corresponding to the behavior of the user based on behavior information regarding the behavior of the user. The distribution unit 18 distributes information about the subject to a terminal device used by the user and that performs display processing of information corresponding to the subject estimated by the estimation unit 17.

  Thereby, the content distribution server 10 according to the embodiment can provide information according to the user's request in order to estimate the subject of the user's behavior based on the user's behavior information.

[8. Hardware configuration)
Further, the terminal device according to each embodiment described above is realized by, for example, a computer 1000 having a configuration as shown in FIG. Hereinafter, the terminal device 100 will be described as an example. FIG. 15 is a hardware configuration diagram illustrating an example of a computer that implements the functions of the terminal device. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

  The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400, and controls each unit. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 starts up, a program that depends on hardware of the computer 1000, and the like.

  The HDD 1400 stores a program executed by the CPU 1100, data used by the program, and the like. The communication interface 1500 receives data from another device via the communication network 500 (corresponding to the network N of the embodiment) and sends the data to the CPU 1100, and also transmits data generated by the CPU 1100 via the communication network 500 to another device. Send to device.

  The CPU 1100 controls output devices such as a display and a printer and input devices such as a keyboard and a mouse via the input / output interface 1600. The CPU 1100 obtains data from an input device via the input / output interface 1600. Further, CPU 1100 outputs data generated through input / output interface 1600 to an output device.

  The media interface 1700 reads a program or data stored in the recording medium 1800 and provides the program or data to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc), a PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. And so on.

  For example, when the computer 1000 functions as the terminal device 100, the CPU 1100 of the computer 1000 implements the function of the control unit 160 by executing a program loaded on the RAM 1200. Further, the HDD 1400 stores various data in the storage device of the terminal device 100. Although the CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, as another example, these programs may be obtained from another device via a predetermined communication network.

[9. Others)
As described above, some of the embodiments of the present application have been described in detail with reference to the drawings. However, these are exemplifications and various modifications based on the knowledge of those skilled in the art, including the aspects described in the section of the disclosure of the invention. The invention can be implemented in other modified forms.

  The configuration of the terminal device 100 described above can be flexibly changed, for example, by realizing an external platform or the like by calling it with an API (Application Programming Interface) or network computing depending on the function.

  Further, “parts” described in the claims can be read as “means” or “circuit”. For example, the acquiring unit can be replaced with an acquiring unit or an acquiring circuit.

Reference Signs List 10 content distribution server 11 communication unit 12 storage unit 13 content information storage unit 14 control unit 15 reception unit 16 acquisition unit 17 estimation unit 18 distribution unit 20 external device 100 terminal device 110 communication unit 120 storage unit 121 action information storage unit 130 input unit 140 output unit 150 imaging unit 160 control unit 161 acquisition unit 162 processing unit 163 estimation unit 164 search unit 165 display unit 166 display control unit

Claims (12)

Using a captured image that is a screen shot of the screen, an acquisition procedure for acquiring target information that is information about the imaging target captured in the captured image,
A display procedure for displaying the target information acquired by the acquisition procedure in a manner superimposed on the captured image, and
The display procedure includes:
A display program for displaying the target information, which is updated based on information provided from an application serving as a provider of the shooting target, superimposed on the shot image . Using a captured image that is a screen shot of the screen, an acquisition procedure for acquiring target information that is information about the imaging target captured in the captured image,
The target information acquired by the acquisition procedure is displayed by being superimposed on the photographed image, and when the photographed image is selected, the screen when the photographed object is photographed, and the provider of the photographed object and Display procedure to display the screen provided by another application and
A display program for causing a computer to execute the following. The acquisition procedure includes:
Display program according to claim 1 or 2, characterized in that obtaining the target information indicating the information about the application to be the imaging target of the provider. The acquisition procedure includes:
The display program according to any one of claims 1 to 3, wherein the target information includes an icon image indicating an application serving as a provider of the shooting target. The acquisition procedure includes:
The display program according to any one of claims 1 to 4 , wherein an analysis result of the character string to be captured is acquired as the target information. The display program according to claim 5 , further comprising: a processing procedure for executing a process based on information indicated by a character string acquired as the target information. The processing procedure includes:
7. The display program according to claim 6 , wherein when the character string obtained as the target information indicates a time, a process of notifying the user of the time is executed. The acquisition procedure includes:
The display program according to any one of claims 1 to 6 , wherein the target information indicating information indicated by the two-dimensional code to be captured is acquired. Using a captured image that is a screen shot of the screen, an obtaining unit that obtains target information that is information on a shooting target captured in the captured image,
A display unit for displaying the target information obtained by the obtaining unit in a manner superimposed on the captured image ,
The display unit,
A display device , wherein the target information updated based on information provided from an application serving as a provider of the shooting target is displayed so as to be superimposed on the shot image . A display method performed by a computer,
Using a captured image that is a screen shot of the screen, an acquisition step of acquiring target information that is information about an imaging target captured in the captured image,
Look including a display step of displaying the target information acquired by the acquisition step superimposed on the captured image,
The display step includes:
A display method , wherein the target information updated based on information provided from an application serving as a provider of the shooting target is displayed so as to be superimposed on the shot image . Using a captured image that is a screen shot of the screen, an acquisition unit that acquires target information that is information about the imaging target captured in the captured image,
The target information obtained by the obtaining unit is displayed so as to be superimposed on the photographed image, and when the photographed image is selected, a screen when the photographed object is photographed, and a provider of the photographed object and A display unit that displays a screen provided by another application
A display device comprising: A display method performed by a computer,
Using a captured image that is a screen shot of the screen, an acquisition step of acquiring target information that is information about an imaging target captured in the captured image,
The target information obtained in the obtaining step is displayed so as to be superimposed on the photographed image, and when the photographed image is selected, a screen when the photographed object is photographed, and a provider of the photographed object. Display process to display the screen provided by the different application
A display method comprising:

Images