JP7561686B2 - Portable information terminal device and its program - Google Patents

Description

The present invention relates to a mobile information terminal device equipped with a camera on the back side of the screen and a program for use therewith, and is particularly applicable to smartphones and the like.

Recent mobile information terminal devices are equipped with touch panels. A touch panel is a transparent film-type pointing device placed on a display, and is sometimes called a touch screen, touch display, contact screen, etc. In such mobile information terminal devices equipped with a touch panel, as disclosed in Patent Document 1, for example, a user places a finger or stylus in contact with the display (or more precisely, the touch panel on the display) and then moves the finger or the like in a sliding motion in a predetermined direction, thereby moving content displayed on the display in that direction or causing the device to perform a predetermined operation. Such an operation is called a swipe or slide gesture.

Smartphones are a typical example of today's portable information terminal devices, and in addition to their original telephone functions, they also have the ability to access the Internet and take pictures of landscapes and other images using an attached camera, making them highly useful as multi-function devices. In particular, when using the Internet, it is possible to view the content of a searched website, and if the website is an EC (Electronic Commerce) site, to purchase products that are on sale. For this reason, it is desirable for smartphone displays to be able to display more information, but there is a limit to the size of the display when portability is taken into account.

In response to this, in smartphones and the like, technology such as that disclosed in Patent Document 1 below is used to move some of multiple pieces of content that cannot be displayed on the display at once due to the display size in the direction of the swipe operation, moving them out of the display range, making it possible to display other content. This allows, for example, a smartphone user to swipe left/right or up/down on the display to scroll left/right or up/down on a large number of pieces of content or product information displayed as reduced images on the display like thumbnails, and to visually check them in order. In addition, for content or products about which the user wishes to know more, detailed information can be displayed on the display by tapping on the reduced image.

Special Publication No. 2010-515978

In this way, mobile information terminal devices such as smartphones allow users to swipe or tap the display with their finger or the like, making it possible to display a large amount of information even on small displays that prioritize portability over display size. However, there are problems with operation, such as the following. For example, if a thumbnail image of a certain piece of content is selected from among many thumbnail images of content, etc., and detailed information is displayed, but the detailed information is different from what was expected, it becomes necessary to tap the display again with a finger or the like to perform a "back operation" that displays many thumbnail images, or to tap a thumbnail image of another piece of content to display detailed information.

In other words, with conventional mobile information terminal devices, users must repeatedly swipe or tap the display to select or go back until they find the desired content, product, or other information, which can be cumbersome and bothersome for users as the operations required to reach the desired information are complicated.

In addition, because the touch panel is placed on the display, when the display is swiped or tapped with a finger or a stylus, part of the content or information displayed on the display may be hidden by the hand doing the swiping, making it difficult for the user to see. This can cause a problem in that the visibility of the display may be impaired.

The present invention has been made to solve the above-mentioned problems, and aims to provide a mobile information terminal device and a program therefor that can improve operability while ensuring visibility.

In order to achieve the above object, the mobile information terminal device of the first invention employs the technical means of claim 1. According to these means, in the first invention , when a series of image information captured by a camera during a predetermined period includes an image of a human's (typically, a user of the mobile information terminal device ) hand or finger (hereinafter referred to as "hand, etc." in the "Means for solving the problem" and "Effects of the invention" columns) and the motion determining means determines that the motion of the hand, etc. during the predetermined period is a predetermined motion based on the series of image information, the content switching means switches the content currently being displayed on the display to another content.

This allows the user of the portable information terminal device to switch the content currently displayed on the display to another content by moving the hand, etc. captured by the camera, in a predetermined motion, instead of swiping or tapping on the display. For example, if the user is holding the portable information terminal device in the left hand, the user can hold the right hand over the back side of the portable information terminal device and perform a predetermined motion to switch content without the content displayed on the display being obscured by the hand, etc. Note that in this specification, "camera" mainly refers to a digital camera that forms an image of a subject on a solid-state imaging element.

Furthermore, when there are a plurality of other contents and the content switching means judges multiple times in succession that the movement of the hand or the like is a predetermined movement and switches the display of the plurality of other contents in sequence for each judgment, when the next display content to be displayed next among the plurality of other contents has the same content as the displayed content already displayed on the display, the content switching means switches to the other content to be displayed after the next display content instead of the next display content and displays it on the display. This prevents content with the same content as the already displayed content from being displayed on the display, making it possible to prevent a situation in which the user of the mobile information terminal device feels disappointed or feels that time has been wasted by showing the content that has already been viewed again.

The device may also be configured to include an image determination means for determining whether or not a series of image information captured by the rear camera within a predetermined period of time includes an image (image information, specific image) that satisfies a control condition. If the image determination means determines that an image of a human hand or finger (typically a user of the mobile information terminal device) is included in the series of image information captured by the camera within a predetermined period of time, and the motion determination means determines based on the series of image information that the motion of the hand or finger within the predetermined period of time is a predetermined motion, the content switching means may be configured to switch the content currently being displayed on the display to another content.

Here, "an image that satisfies the control conditions (specific image)" refers to an image (series of image information) that the camera on the back side of the mobile information terminal device recognizes as an appropriate image within the imaging area set for executing the display control of the content of the present invention. Furthermore, if the image determination means or the motion determination means recognizes an image within a predetermined imaging area, it can recognize it as a target image even if it is not an image of a human hand or finger (for example, even if it is an image of a stylus or other object), and can execute display control of the content to be displayed on the display. However, it is preferable that at least an image of a human hand or finger can be recognized as a target image. In addition to this configuration, certain conditions can be added to the image determination means (for example, shape conditions, temperature conditions, etc.) so that the display control of the content to be displayed on the display can be performed only when a human hand or finger, or something similar to this (i.e., something that satisfies certain conditions common to a human hand or finger) is recognized.

The present invention also assumes that an image of a person's hand or finger is captured (detected) by a camera on the back side of the mobile information terminal device, and the display of the content displayed on the display is controlled. Therefore, the imaging area in which the captured image can be recognized as a proper image is set to a distance (depth distance) from the camera on the back side of the mobile information terminal device between about 0 cm and about 25 cm. More preferably, the imaging area may be set between about 1 cm and about 15 cm.

This imaging area can be set appropriately depending on the performance of the camera on the back side of the mobile information terminal device, and the present invention is not limited to this specific range (approximately 0 cm to approximately 25 cm). However, if the camera is too close, it becomes difficult to recognize (identify) various movements such as hand movements, and if the camera is too far away, there is a risk that the user will recognize other image information that is not intended by the user (for example, movements of someone other than the user), so it is preferable to limit the imaging area to the specific range (approximately 0 cm to approximately 25 cm) as described above, for example.

According to the mobile information terminal device of the present invention and the computer program for controlling it, the user can, for example, move the hand or other hand imaged by the camera so as to perform a predetermined motion or other predetermined motion, instead of swiping or tapping on the display, thereby switching the currently displayed content to another content, scrolling the display vertically or horizontally, or enlarging or reducing the display, and controlling the display mode. For example, when the user holds the mobile information terminal device in one hand (e.g., the left hand), the user can hold the other hand (e.g., the right hand) over the back side of the mobile information terminal device and perform a predetermined motion or other predetermined motion, thereby enabling the user to control the content switching operation and the display mode of the content without the content or hand being hidden on the display. Therefore, the operation to reach the desired information can be performed only by the movement of the hand or other hand behind the display, thereby ensuring visibility and improving operability.

1 is an explanatory diagram showing a smartphone (hereinafter referred to as a "smartphone") according to one embodiment of the present invention to which a mobile information terminal device and its program are applied, and the hands and fingers of a user holding and operating it. FIG. 2 is a block diagram showing an example of a functional configuration of the smartphone according to the present embodiment. 13 is a flowchart showing the flow of processing of a browsing operation assistance application program executed by the smartphone of this embodiment. 4 is a flowchart showing the flow of a subroutine of the browsing operation assist application program shown in FIG. 3, in which (A) shows a first operation process and (B) shows a second operation process. An explanatory diagram showing an example of content displayed on the display of a smartphone according to the present embodiment. (A) shows a display example of the result of searching for "chair", (B) shows a display example when a content of a certain chair is selected from the search result of (A), (C) shows a display example when the content of related products is shifted from (B) by a first operation process, and (D) shows a display example when the display of (B) is scrolled upward by a second operation process. The figures show the state in which the scroller is attached to the smartphone. (A) is a plan view of the smartphone and the scroller from the front side of the smartphone, (B) is a cross-sectional view showing the state in which the smartphone and the scroller are cut along the line 6B-6B shown in (A), (C) is a side view seen from the direction of the arrow 6C shown in (A) and a bottom view of the smartphone and the scroller, and (D) is a bottom view showing the state in which the scroller is being attached to or removed from the smartphone in the side view of (C). 6(B) is an explanatory diagram showing an example of operation of the roller unit constituting the scroller, in a cross-sectional view equivalent to that shown in FIG. 6(A) shows a case where a user rotates the roller unit counterclockwise in the figure with his/her finger, (B) shows the state immediately after a corner of the roller unit comes into contact with a touch panel from the state of (A), (C) shows the state in which the corner is pressed against the touch panel and crushed as a result of the roller unit being rotated further from the state of (B), and (D) shows the state immediately after the crushed corner returns to its original shape as a result of the roller unit being rotated further from the state of (C).

Next, the various embodiments to which the mobile information terminal device and its program of the present invention are applied will be described with reference to the accompanying drawings. First, the configuration of the smartphone 10 of this embodiment will be described with reference to Figs. 1 and 2.

As shown in FIG. 1, the smartphone 10 is a commercially available product and is made up of a thin, rectangular body case 10a that can be held by a user in one hand (the left hand 90L in the figure), a display 11 provided on the inside and outside of the body case 10a, a touch panel 12, a control unit 20, and the like.

On the front side 10F of the smartphone 10, a display 11 is attached to a rectangular frame hole formed in the main body case 10a, and a speaker 14 and a front camera 15 are also provided. A rear camera 16 is provided on the back side 10B, and a microphone 13 is provided on the lower end side. The touch panel 12 is provided on the surface of the display 11, almost integrally with the display 11. Therefore, please note that in the same figure, the display 11 and the touch panel 12 are depicted overlapping in the same rectangular shape (parts colored gray).

As shown in FIG. 2, the display 11, touch panel 12, front camera 15, etc. are electrically connected to and controlled by a control unit 20 housed in the main body case 10a. The control unit 20 is a main board that has an MPU 21, memory 25, etc., and in addition to the display 11, etc., a wireless unit 17, various sensors 18, a battery 19, and even a memory card and a SIM card, etc. (not shown), are electrically connected to the control unit 20.

The display 11 is, for example, a 6-inch liquid crystal display device with an aspect ratio of 16:9, and is configured to be able to display image information output from the control unit 20. The touch panel 12 is a transparent film-type pointing device that is roughly the same size as the display 11 and is provided almost on the front of the display surface, and is configured to be able to output position information on the display 11 where a user touches it with a finger or a stylus to the control unit 20. The touch panel 12 is sometimes called a touch screen or contact screen. For position detection, for example, a resistive film method, a capacitive method, an ultrasonic surface acoustic wave method, etc. are used.

The microphone 13 and speaker 14 are used when making calls using the telephone function, and for recording and audio output by application programs (also called application software; hereafter collectively referred to as "apps"). The microphone 13 is configured to be able to collect the user's voice and output acoustic information to the control unit 20, and the speaker 14 is configured to be able to output acoustic information output from the control unit 20 to the outside.

The front camera 15 and the rear camera 16 are imaging devices having a solid-state imaging element such as a CCD or CMOS and an optical lens, and are configured to be able to output captured image information to the control unit 20. The front camera 15 is provided on the front side 10F of the smartphone 10, and is configured to be able to capture, for example, the user's face. The rear camera 16 is provided on the back side 10B of the smartphone 10, and is configured to be able to capture, for example, landscapes. The front camera 15 is sometimes called the in-camera, and the rear camera 16 is sometimes called the out-camera.

The wireless unit 17 is a wireless transmitter/receiver configured to enable voice and data communication via the wireless communication network of a mobile network operator (MNO), and the control unit 20 controls the calls and data communication. A SIM card (not shown) stores identification information unique to the smartphone 10 that is used during wireless communication. The various sensors 18 include a GPS sensor, an acceleration sensor, an illuminance sensor, a temperature sensor, etc., and all of them are configured to be able to output detection data to the control unit 20.

The battery 19 is a secondary battery (e.g., a lithium ion battery) that can be charged with DC power supplied from an external source, and is configured to be able to supply driving power to the control unit 20, the display 11, etc. Information about the charging and discharging of the battery 19 is also output to the control unit 20, and it is configured so that, for example, the control unit 20 can manage charging and discharging.

The control unit 20 is composed of an MPU 21, a GPU (Graphics Processing Unit) 23, a memory 25, etc. The MPU 21 is a microcomputer, and is configured to control hardware such as the display 11 described above, and to execute a browsing application and a browsing operation assistance application described below, by executing an OS (operating system software) such as Android (registered trademark) or iOS (registered trademark) stored in the memory 25. The GPU 23 is used for image data analysis processing, etc., which will be described later. As described in the section [Problems to be solved by the invention], a typical smartphone 10 configured in this way has problems such as the complicated operations required to reach the desired information, which can be bothersome for the user, and the visibility of the display 11 can be impaired.

In the smartphone 10 of this embodiment, a browsing operation assistance app that can solve these problems is installed. With this browsing operation assistance app, the smartphone 10 is controlled so that, for example, when the rear camera 16 captures (takes an image of) a hand movement such as shaking the right hand 90R from side to side (first movement), the smartphone 10 switches the content currently displayed on the display 11 to another display content, or when the rear camera 16 captures a hand movement such as shaking the right hand 90R up and down (second movement), the smartphone 10 scrolls the content currently displayed on the display 11 upwards (or downwards). From here on, such a browsing operation assistance app will be described with reference to Figures 3 to 5.

The file of the browsing operation assistance app may be downloaded by data communication via a wireless communication network performed by the function of the wireless unit 17 and installed in the memory 25 of the control unit 20, or may be pre-installed in the memory 25. The browsing operation assistance app may be executed independently, or may function in conjunction with, for example, a web browser or a file browser (file manager). That is, the browsing operation assistance app may function as a subroutine of a general-purpose plug-in compatible browsing app (for example, Google Chrome (registered trademark), Safari (registered trademark), Microsoft Edge (registered trademark), Internet Explorer (registered trademark), etc.). In this case, the browsing operation assistance app is an executable file that conforms to a specified plug-in format, and is plugged into these browsing apps and executed each time it is called.

In addition, prior to execution of the browsing operation assistance app, it is necessary to register the above-mentioned first and second actions and the right hand 90R and left hand 90L that perform these actions in the memory 25 of the control unit 20 in advance. Although the flow of the registration process is omitted, it is performed by a function of the browsing operation assistance app. For example, as a first action for switching the content currently displayed on the display 11 to another display content, the action of the user shaking his/her right hand 90R (or left hand 90L) from side to side and the right hand 90R (or left hand 90L) that performs this action are registered. In addition, as a second action for controlling the display mode of the content currently displayed on the display 11 (for example, scrolling upward), the action of shaking the right hand 90R (or left hand 90L) up and down and the right hand 90R (or left hand 90L) that performs this action are registered.

The direction in which the right hand 90R (or left hand 90L) is swung follows the up, down, left and right directions that are set (defined) in advance in the rear camera 16. For example, as shown in FIG. 1, when the rear camera 16 is located above the smartphone 10 and the display 11 faces the user, the smartphone 10 sets the upper side of the smartphone 10 as "up", the lower side of the smartphone 10 as "down", the right side of the smartphone 10 as "right", and the left side of the smartphone 10 as "left". Therefore, for example, when the smartphone 10 is held by the user in a state rotated 90 degrees clockwise from the state shown in FIG. 1, the movement of the user waving the right hand 90R above (below) himself is analyzed as the movement of waving the right hand 90R to the left (right), and the movement of waving the right hand 90R to the left (right) is analyzed as the movement of waving the right hand 90R to the bottom (up) by the motion analysis process (S105) described later.

In addition, instead of the right hand 90R and the left hand 90L, the movements of the fingers (e.g., index finger, thumb, etc.) and those fingers may be registered, and instead of the right hand 90R and the left hand 90L, for example, the movements of a stylus or touch pen held by the right hand 90R and the stylus that causes that movement may be registered. Note that the right hand 90R, the left hand 90L, those fingers, and the stylus or touch pen held by those hands are collectively referred to as "hands, etc." in the section below titled "Embodiment for carrying out the invention."

As shown in FIG. 3, the browsing operation assistance app executed by the control unit 20 first performs a predetermined initialization process in step S101. In this initialization process, for example, the work area and flags in the memory 25 are cleared, and video capture by the rear camera 16 is started. The captured image data is video data within a predetermined period (for example, 1 to 2 seconds).

In the next step S102, image data acquisition processing is performed. This image data is a series of image information captured by the backup camera 16 within a predetermined period of time, and is a series of still image information consisting of a plurality of still images captured (extracted) from video data within the predetermined period of time by frame advance, frame drop, or frame skip. Note that, if video analysis is possible in the image data analysis processing of step S103, the series of image information may be video-captured video image information for a predetermined period of time.

For example, as shown in FIG. 1, when a user is looking at the display 11 while holding the smartphone 10 in the left hand 90L, the user often swipes or taps the touch panel 12 with the right hand 90R. Therefore, the browsing operation assistance app uses the rear camera 16 to capture an image of the movement of the free right hand 90R (or left hand 90L) when the smartphone 10 is held in the left hand 90L (or right hand 90R).

When the user turns the palm of their hand toward the rear camera 16 and, for example, shakes their right hand 90R from side to side (in the direction of the arrow shown in FIG. 1), a series of still image information capturing the movement of the right hands 90R, 90R' is acquired in step S102. Therefore, in the image data analysis process in the following step S103, an analysis is performed to determine whether or not the series of still image information includes image information of the right hand 90R. This image data analysis process (S103) is mainly performed by the GPU 23.

In the image data analysis process in step S103, for example, pattern matching is used as a known image analysis method. In image analysis using pattern matching, for example, image information of the user's right hand 90R is registered in advance in memory 25 as template image information, and a match determination is made between each of a series of still image information to be compared and the template image. In this image data analysis process, for example, the proportion of still image information among the series of still image information that is determined to match the template image information is obtained.

In step S104, a process is performed to determine whether the series of still image information includes the user's hand (right hand 90R in the example of Figure 1). In the previous step S103, the proportion of still image information that is determined to match the template image information is obtained, and here, if the proportion is equal to or greater than a predetermined value, it is determined that the series of still image information includes a hand, and if it is less than the predetermined value, it is determined that the hand is not included. The predetermined value is determined in advance based on the analysis accuracy of the image data analysis process in step S103, and is set arbitrarily in the range of 70 to 90%, for example.

If it is determined by the determination process of step S104 that the series of still image information includes a hand or the like (S104; YES), the motion analysis process is performed in the following step S105. On the other hand, if it is determined that the series of still image information does not include a hand or the like (S104; NO), since there is a high probability that the image captured by the rear camera 16 includes something other than the right hand 90R, for example, the process proceeds again to step S102 to obtain image data captured within a predetermined period from this time onwards.

In step S105, a motion analysis process is performed. In this process, an analysis is performed to determine what movement (action) is being made in the images of the right hand 90R and the like included in the series of still image information acquired in step S102. This process, like the image data analysis process in step S103, is performed mainly by the GPU 23 using, for example, pattern matching as a well-known image analysis method. Note that, if the image data analysis process in step S103 also includes an analysis process with the same content as the motion analysis process in step S105, then this motion analysis process (S105) may be omitted.

As described above, in this embodiment, the first and second actions are registered in advance in the memory 25 of the control unit 20. Therefore, for example, if the user's action of shaking his/her right hand 90R etc. from side to side is registered as the first action, in this action analysis process, a plurality of still images captured by frame-by-frame advancement of the movement of the right hand 90R etc. are used as template image information, and a matching determination is made between each of the series of still image information to be compared and the template image. Also, for example, if the user's action of shaking his/her right hand 90R etc. up and down is registered as the second action, in this process, a plurality of still images captured by frame-by-frame advancement of the movement of the right hand 90R etc. are used as template image information, and a matching determination is made between each of the series of still image information to be compared and the template image.

In this motion analysis process (S105), the matching judgment result is obtained as to whether the motion matches either the first motion or the second motion, or whether the motion does not match either the first motion or the second motion. Therefore, the motion judgment process in the following step S106 judges whether the motion is already registered (the first motion or the second motion), and if it is already registered, judges whether it is the first motion or the second motion.

In addition, in the motion analysis process in step S105, the direction and speed of the right hand 90R or the like are analyzed when the hand is waving. For example, when the right hand 90R or the like is waving from side to side, directional information of "left and right" is obtained by analysis as the direction of the hand. Also, by comparing the speed of the hand moving from left to right (rightward) with the speed of the hand moving from right to left (leftward), information on the magnitude relationship of the speed in the left and right directions is also obtained by analysis. Similarly, when the right hand 90R or the like is waving up and down, directional information of "up and down" is obtained by analysis as the direction of the hand, and by comparing the speed of the hand moving up and the speed of the hand moving down, information on the magnitude relationship of the speed in the up and down directions is also obtained by analysis. This information on the direction and speed of the hand waving is used in the second motion process (S108, FIG. 4(B)) described later.

If the motion determination process of step S106 determines that the motion based on the image of the right hand 90R, etc., matches the first motion (e.g., motion of shaking the right hand 90R from side to side) (S106; YES (first motion)), the process proceeds to the first motion process of step S107, and if the motion based on the image of the right hand 90R, etc., matches the second motion (e.g., motion of shaking the right hand 90R up and down) (S106; YES (second motion)), the process proceeds to the second motion process of step S108. On the other hand, if it is determined that the motion based on the image of the right hand 90R, etc. (e.g., motion of shaking the right hand 90R diagonally, etc.) does not match either the first motion or the second motion, the motion captured by the rear camera 16 is different from the predetermined motion, etc. that has been determined in advance. In other words, there is a high possibility that the motion of the user's right hand 90R, etc. is incorrect. Therefore, the process returns to step S102 to obtain image data captured within a predetermined period from this time onwards.

The flowcharts of the first operation process (S107) and the second operation process (S108) are shown in FIG. 4. Also, an explanatory diagram showing an example of the content displayed on the display 11 by these processes is shown in FIG. 5.

As shown in FIG. 4 (A), in the first operation process (S107), first, in step S201, a content information acquisition process is performed. Content refers to the substance or substance of information, and content information refers to information related to the content (for example, address information such as a URL where the content exists, file name, file type, file size, creation date, information on other content that is linked (associated), etc.). Here, information on the content currently being displayed on the display 11 is acquired from the viewing app.

For example, as shown in FIG. 5(A), when "chair" is entered into a search box 70 of a search engine provided via a browsing app on the Internet on the smartphone 10, the search results are displayed on the screen of the display 11. In this example, reduced image information 71a, 71b, 71c of multiple recommended products are displayed side-by-side from the left side of the screen in order of recommendation, from the right side to the left side of the screen. In the figure, some of the reduced image information 71c such as 71a that cannot be displayed on the screen due to the display size and hidden information 79 that cannot be displayed are shown in light gray on the outer right side of the display 11. The hidden information 79 can be displayed by scrolling the screen display to the left.

In addition, summary information 73 of mail-order sites selling "chairs" is displayed as text information below the reduced image information 71a to 71c on the screen. Although not shown in FIG. 5(A), multiple pieces of summary information 73 of mail-order sites not shown on the display 11 are obtained from the Internet as search results so that they can be displayed in order of ranking, and summary information 73 not shown in the figure can be displayed by scrolling the screen display upwards.

After the reduced image information 71a-71c and summary information 73 are displayed on the display 11, if the user selects the reduced image information 71a of "Office stacking chair" via the touch panel 12 with a finger, a stylus, or a touch pen, detailed information 74a of the stacking chair is displayed as content, for example, as shown in FIG. 5(B). In this case, in step S201, information on the detailed information 74a of the stacking chair displayed on the display 11 is acquired as content information. Then, based on this acquired content information, the process of determining whether there is the next content, i.e., content that has not yet been viewed this time, is performed in the following step S202.

In the example of the stacking chair content shown in FIG. 5(B), when scrolling upward, multiple related product information 75a, 75b, 75c ("transport cart", "set of 2 chairs", and "set of 5 chairs") displayed in the "★★★Related Products★★★" column of detailed information 74a' that subsequently appears on display 11 are linked as other content information, as shown in FIG. 5(D). In this way, when other content information is linked to the currently viewed content or content that can be viewed by scrolling (hereinafter referred to as "viewed content, etc."), the next content exists. Therefore, in step S202, it is determined that the next content exists (S202; YES), and in the next step S203, information on the next content is obtained from the viewing app or from the Internet via the viewing app.

Then, in the following step S205, a process is performed to determine whether the content is the same as the already displayed content. In other words, if the next content has been viewed before, i.e., the previous time, and that already viewed content is displayed again on the display 11, the user will see content that he or she has already viewed again, which may cause the user to feel disappointed or give the impression that time has been wasted. Furthermore, if the content α being viewed is linked to the next content β, an infinite loop of viewing will be created, such as content α → content β → content α → content β → ....

For this reason, in this embodiment, in step S205, a process is performed to determine whether the next content is the same as the content previously viewed. If the next content is the same as the previously displayed (viewed) content (S205; YES), the process returns to step S202 to determine whether another next content exists, and if they are not the same (S205; NO), the process proceeds to step S206.

Whether these contents are the same or not is determined based on content information such as the content's address information, file name, file size, and creation date, and if this content information is the same, it is determined that these are the same content. Also, if previously viewed content is stored in the cache memory or cookie file of the control unit 20, and the same content exists in the cache memory, it is determined that these are the same content.

As a result, previously viewed content will not be displayed again on the display 11, making it possible to avoid the occurrence of problems such as those described above. Note that if there is an advantage to configuring the previously displayed content to be viewable again rather than the occurrence of such problems, this determination process (S205) will be unnecessary. Therefore, in this case, the determination process of step S205 is deleted.

On the other hand, if it is determined in step S202 that there is no next content (S202; NO), i.e., if other content information such as related product information 75a-75c of the viewed content is not linked, the process proceeds to step S204. Then, after a process is performed to obtain end information (predetermined data or a predetermined flag) indicating that there is no next content from memory 25, the process proceeds to step S206.

In addition, when other content information is not linked to the content information acquired in step S201, for example, the content currently displayed on the display 11 and other content that exists in the same example in the display hierarchy (or data hierarchy) may be set as the next content, and it may be determined in step S202 that there is a next content.

For example, if no other content information is linked to the detailed information 74a of the stacking chair shown in FIG. 5(B), the content may be returned to the previous content (or one level above in the display hierarchy), and other reduced image information 71b, 71c displayed in the same row as the reduced image information 71a of the stacking chair shown in FIG. 5(A) may be selected as the next content. For example, if the content of "GAMING CHAIR" of reduced image information 71b displayed to the right of reduced image information 71a in accordance with the recommended order (from the left to the right of the screen) is selected, the detailed information of "GAMING CHAIR" is displayed in the same display format as the detailed information 74a of the stacking chair shown in FIG. 5(B) and the detailed information 74b of the transport cart shown in FIG. 5(C), just as when the reduced image information 71 of "GAMING CHAIR" is selected on the same screen.

In step S206, a display information output process is performed. In this process, the next content information acquired in step S203 or the end information acquired in step S204 is output to the viewing application. When the next content information is output to the viewing application, the viewing application acquires data of the other content from the Internet based on the next content information (or from memory 25 if the data of the other content has already been downloaded by pre-reading, etc.), and performs a process of switching the content currently displayed on display 11 to the display of the other content. In other words, the viewing application receives an instruction from the viewing operation assistance application and switches the display content of display 11 to the other content in response to the first operation performed by the user.

For example, in the detailed information 74a' of the stacking chair shown in FIG. 5(D), thumbnail images of the contents "transport cart", "set of 2 chairs", and "set of 5 chairs" are linked as multiple related product information 75a-75c, so the viewing application receives instructions from the viewing operation assistance application and displays these contents as other contents on the display 11 in the order of display on the display 11 (from the left side of the screen to the right side). In this example, the related product information 75a displayed on the left side of the screen is a thumbnail image of the "transport cart" content, so detailed information 74b of "transport cart for stacking chairs" is displayed on the display 11 as the linked content (next content) in response to the first action performed by the user (see FIG. 5(C)).

In contrast, when end information is output to the viewing app, there is no next content linked to the viewed content, etc., and the viewing app cannot obtain other content information, so it cannot switch the display on display 11 to other content. Therefore, in such a case, for example, even if the user performs the first action, the viewing app may maintain the content currently displayed on display 11 as is, or may display information on display 11 to the effect that there is no content to switch to.

When the process in step S206 ends, the first operation process series is completed and the process returns to the main routine of the browsing operation assistance app shown in FIG. 3. Then, the process proceeds again to step S102 to obtain image data captured within a predetermined period of time from this time onward. In other words, the process prepares for the next hand or other movement made by the user.

As shown in FIG. 4B, in the second motion process (S108), first, hand information acquisition process is performed in step S301. The hand information is information on the direction in which the hand is waved and the speed at which it is waved, obtained by the motion analysis process (S105) shown in FIG. 3 described above. Generally, when a person waves their hand while being conscious of the direction, the speed at which they wave in that direction is faster than the speed at which they wave in the opposite direction to return the hand after waving. Therefore, from this information, it is possible to know that the hand is being waved in the direction with the greater speed (upward if up > down, downward if up < down), in other words, the direction of the hand motion.

Therefore, if the judgment process in the next step S302 judges that the hand is being waved upward (S302; upward), the process proceeds to step S303, where an upward scroll setting process is performed. Also, if the judgment process judges that the hand is being waved downward (S302; downward), the process proceeds to step S304, where a downward scroll setting process is performed. In these setting processes (S303, S304), for example, a scroll command including the scroll direction (upward or downward) and the scroll amount is set in the output buffer. The scroll amount is a predetermined amount that is set in advance, and is set, for example, to a value that is approximately the same as or slightly smaller than the width (vertical width in this case) of the display 11 in the scroll direction (vertical direction in this case), that is, approximately equivalent to one page in the vertical direction of the screen.

In step S305, a scroll command output process is performed. In this process, the scroll command set in step S303 or step S304 is output to the viewing application. When the scroll command is output to the viewing application, the viewing application scrolls the content currently displayed on the display 11 upward or downward by a predetermined amount. This makes it possible to scroll the content displayed on the display 11 upward or downward by approximately one page in the vertical direction of the screen.

For example, in the example of the stacking chair content shown in FIG. 5(B), the viewing app scrolls the detailed information 74a displayed on the display 11 downward by approximately one page in the vertical direction of the screen. As a result, as shown in FIG. 5(D), in the detailed information 74a' displayed on the display 11 after scrolling, the display of "★★★ Related Products ★★★", which was at the bottom of the screen before scrolling, is now displayed at the top of the screen, and multiple related product information 75a to 75c, which were not displayed before scrolling, are displayed below "★★★ Related Products ★★★". In addition, the model number, size, weight, color, material, functions, precautions, and part of "<<< Product Reviews >>>" are also displayed as "<<< Product Details >>>".

When the process in step S305 is completed, the series of second operation processes is completed and the process returns to the main routine of the browsing operation assistance app shown in FIG. 3, and the process again proceeds to step S102 to obtain image data captured within a predetermined period from this time onward. In other words, it prepares for the next hand or other movement made by the user.

In the present embodiment, the browsing operation assistance app has been described as being plugged into a general-purpose browsing app and functioning as a subroutine thereof, but this is not limited thereto. For example, the browsing app may be configured to incorporate the functions of the browsing operation assistance app (FIGS. 3 and 4) in advance as one function of a dedicated browsing app for the smartphone 10.

As described above, in the smartphone 10 of this embodiment, the browsing operation assistance app installed on the smartphone 10 controls the control unit 20 via the browsing app or directly, so that the control unit 20 determines that a series of image information captured by the rear camera 16 within a predetermined period of time includes an image of the hand or the like of the user of the smartphone 10 (S104; YES), and if the control unit 20 determines that the movement of the hand or the like within the predetermined period of time is a predetermined first action (predetermined action) based on the series of image information (S106; YES (first action)), the control unit 20 switches the detailed information 74a currently displayed on the display 11 to the display of related product information 75a to 75c (S206).

As a result, for example, instead of swiping or tapping on the display 11, the user of the smartphone 10 can switch the detailed information 74a currently displayed on the display 11 to the display of related product information 75a to 75c by moving the hand, etc. so that the movement of the hand, etc. captured by the rear camera 16 becomes the first movement. For example, when the user holds the smartphone 10 in the left hand 90L (or right hand 90R), the user can hold the right hand 90R (or left hand 90L) over the back side 10B of the smartphone 10 and perform a predetermined movement, thereby enabling the content switching operation to be performed without the content, etc. displayed on the display 11 being hidden by the right hand 90R (or left hand 90L). Therefore, the operation until the desired information is reached can be performed only by the movement of the right hand 90R (or left hand 90L) behind the display 11, thereby improving operability while ensuring visibility.

In addition, in the smartphone 10 of this embodiment, when the control unit 20 determines that the movement of the hand or the like is a second action (another specified action) (S106; YES (second action)), it scrolls the screen display on the display 11 upwards or downwards as a control related to the display mode of the content. The control related to the display mode of the content may, for example, scroll the content currently displayed on the display 11 to the left or right, or change the display magnification of the content (enlarged or reduced display). The content displayed on the display 11 may also be returned to the content at the previous point in time (or one level above in the display hierarchy).

As a result, for example, instead of swiping the display 11, the user of the smartphone 10 can control various display modes, such as scrolling the content currently displayed on the display 11 in the vertical or horizontal directions of the display 11, or enlarging or reducing the content, by moving the hand or other object captured by the rear camera 16 in a manner that corresponds to a second predetermined motion.

Furthermore, in the smartphone 10 of this embodiment, when a plurality of related product information 75 (other contents) are displayed on the display 11 and the control unit 20 judges multiple times in succession that the movement of the hand or the like is the first movement and switches the plurality of related product information 75a to 75c in sequence for each judgment, for example, when the content of the related product information 75b to be displayed next on the display 11 among the plurality of related product information 75a to 75c is the same as the displayed content already displayed on the display 11, the control unit 20 switches to the content of the related product information 75c to be displayed next (other content to be displayed after the next displayed content) instead of the content of the related product information 75b (next display content) and displays it on the display 11. This makes it possible to prevent detailed information with the same content as the displayed content from being displayed on the display 11. In other words, it is possible to prevent a situation in which the user feels disappointed or feels like time has been wasted by showing the content that has already been viewed again.

In the above-described embodiment of the browsing operation assistance app, when the browsing content, etc. is a recommended product (recommended product), related product information 75a to 75c is exemplified as other content information linked to it. However, the linked other content information may be, for example, company information selling the recommended product, or other product information sold by the company. In addition, the browsing content, etc. is not limited to recommended products provided by an Internet search engine using the browsing app, but may be text data, image data, video data, etc. provided from the Internet in association with information on the search results by the search engine. In addition, as long as it is electronic data that can be viewed on the display 11 using the browsing app, it may be, for example, text data, image data, video data, etc. stored in the memory 25 of the smartphone 10.

In the above-described embodiment of the browsing operation assistance app, an example has been described in which the screen display of the display 11 is scrolled upward or downward in response to the second action. However, if the control is related to the display mode of the content corresponding to the second action, the content currently displayed on the display 11 may be scrolled to the left or right, or the display magnification of the content may be changed (enlarged or reduced). The browsing operation assistance app may be configured to return the content displayed on the display 11 in response to the second action to the content at the previous point in time (or the content one level above in the display hierarchy). Note that the scrolling in the vertical or horizontal directions may be configured to be continuous at a constant speed, or the scrolling speed may be changed in response to the distance of the movement of the hand or the like in the second action. For example, the scrolling may be configured to be faster as the vertical movement distance of the hand or the like increases, and slower as the vertical movement distance decreases.

Furthermore, in the above-described embodiment of the browsing operation assistance app, an example has been described in which the display on the display 11 is switched to other content in response to a first action, and an example in which the screen display on the display 11 is scrolled upwards or downwards in response to a second action. However, in addition to these, a third action, a fourth action, a fifth action, etc. may be registered, and the display mode of the content being displayed on the display 11 may be controlled in response to these actions in the action determination process (S106).

For example, the third action is a diagonal movement of the right hand 90R (or the left hand 90L) from the upper right to the lower left, the fourth action is a diagonal movement of the right hand 90R (or the left hand 90L) from the upper left to the lower right, and the fifth action is a movement of moving the right hand 90R (or the left hand 90L) back and forth away from or toward the smartphone 10. The browsing operation assistance app may be configured to scroll the content displayed on the display 11 to the left or right in response to the third action, change the display magnification of the content in response to the fourth action, or return the display on the display 11 to the content from the content at the previous point in time (or one level above in the display hierarchy) in response to the fifth action. In addition, when the content displayed on the display 11 is video data, the browsing operation assistance app may be configured to play the video in response to the second action, stop the video being played in response to the third action, fast-forward the video in response to the fourth action, or rewind the video in response to the fifth action.

[Other inventions]
In conventional mobile information terminal devices such as smartphones, a user can swipe or tap the display with a finger, stylus, or touch pen (hereinafter referred to as "finger, etc.") to display a large amount of information even on a small display that prioritizes portability over display size, but there are problems with operation, such as the following: In general, the surface of the display of a mobile information terminal device is gloss-treated to make images such as photos and videos look beautiful, or is non-glare-treated to reduce eye fatigue caused by reflection, but in either case, it often feels like a flat surface when touched with a finger, etc. (even non-glare feels only rough to the touch).

For this reason, for example, when swiping the display with a finger or the like, it is not possible to give the user the clicking sensation that is transmitted to the finger when the scroll wheel of a mouse is rotated. In other words, in conventional mobile information terminal devices, for example, it is not possible to give the user an intuitive sense of the amount of swiping when swiping the display. Therefore, in [Other Inventions], an operation device (also called a "scroll operation device") is provided that can give the user an intuitive sense of the amount of swiping when swiping the display.

[Another Invention 1]
A scroll operation device that is attached to a mobile information terminal device having a touch panel on the surface of a display,
a roller portion having elastic protrusions on its outer periphery at substantially equal intervals in the circumferential direction;
a frame portion that rotatably supports both ends of the roller portion;
a support portion for mounting both ends or one end of the frame portion on the portable information terminal device so that the protrusion comes into contact with the touch panel when the roller portion rotates;
A scroll operation device comprising:
[Another Invention 2]
In the scroll operation device according to the above-mentioned another invention 1, the convex portion is present in a linear shape in the axial direction of the roller portion.
[Other Invention 3]
In the above other invention 1 or other invention 2, the frame body portion is provided with a mechanism for imparting a clicking sensation to the tactile sensation of the user's finger touching the roller portion as the roller portion rotates, or a mechanism for imparting a clicking sound to the user's hearing.

The above-mentioned other invention 1 is a scroll operation device that is attached to a mobile information terminal device having a touch panel on the surface of the display, and includes a roller section, a frame section, and a support section. The roller section has elastic convex sections on its outer periphery at approximately equal intervals in the circumferential direction, and both ends of the roller section are supported by the frame section so that it can rotate freely. The frame section is attached to the mobile information terminal device by the support section at both ends or one end of the frame section so that the convex sections of the roller section come into contact with the touch panel when the roller section rotates. Note that rotation means that the roller section can rotate in both the forward direction and the reverse direction. Note that the scroll operation device of this other invention is also simply called an "operation device" because it is capable of operations other than scrolling (screen switching operations).

As a result, when a user of a portable information terminal device rotates the roller part with his/her finger in the same way as rotating the scroll wheel of a mouse, "the convex parts that the roller part has on its outer circumference at approximately equal intervals in the circumferential direction" discontinuously come into contact with the touch panel of the portable information terminal device. Therefore, the roller part comes into contact with the touch panel as if the finger or the like were swiping in short, predetermined strokes, and the user's finger is given a sensation of contact of the convex parts that occurs in proportion to the amount of rotation as the roller part rotates. The amount of rotation of the roller part is the total amount of short swipes (total amount of short swipes), and can correspond to the amount of swipe. The user can also intuitively grasp the amount of rotation of the roller part from the sensation of contact of the intermittent convex parts. Therefore, when swiping the display, the amount of swipe can be given to the user intuitively.

In the above-mentioned other invention 2, the convex portion of the roller portion exists in a linear manner in the axial direction of the roller portion, so that it is possible to swipe a wide range on the touch panel. As a result, even if the scrolling operation device is attached to a mobile information terminal device in an inappropriate state, for example, where a part of the roller portion is excessively floating above the touch panel in the axial direction of the roller portion, it is possible to abut the convex portion on the remaining part of the roller portion against the touch panel. Therefore, even in such an inappropriate attachment state, the amount of swipe can be intuitively imparted to the user when swiping the display.

In the third aspect of the present invention, the frame portion includes a mechanism for imparting a clicking sensation to the tactile sensation of the user's finger touching the roller portion, or a mechanism for imparting a clicking sound to the user's hearing as the roller portion rotates. This allows the user to obtain a clicking sensation or clicking sound through their sense of touch or hearing as the roller portion rotates, in addition to the sensation of contact with the protrusions, so that the amount of swiping can be more reliably imparted to the user in a sensory manner.

The scrolling device of [Other Invention 1] to [Other Invention 3] above can be embodied as, for example, a scroller 30 as shown in Figures 6 and 7 as one embodiment. The scroller 30 is used by attaching it to a smartphone 50 that has a touch panel 52 on the surface of a display 51, and mainly comprises a roller unit 35, a frame 31, and a click mechanism unit 36.

The display 51 of the smartphone 50 is, for example, a 6-inch liquid crystal display device with an aspect ratio of 16:9. The touch panel 52 is a transparent film-type pointing device that is approximately the same size as the display 51 and is provided on almost the front of the display surface, and is configured to be able to output position information on the display 51 where the user touches it with a finger or a stylus to the control unit of the smartphone 50. The position detection of the touch panel 52 is, for example, a resistive film method, a capacitive method, an ultrasonic surface acoustic wave method, etc.

The roller portion 35 is made of a material (e.g., conductive rubber) that is elastic and has good electrical conductivity and is formed in a polygonal prism shape. In this embodiment, the roller portion 35 is formed in a regular hexagonal prism shape with an axial length set shorter than the width direction length of the display 51. As long as the cross-sectional shape is a regular polygon, it may be, for example, a regular octagonal prism shape or a regular decagonal prism shape. The outer periphery of the roller portion 35, i.e., the peripheral surface portion 35a, has corners 35b extending in the axial direction in a linear manner and formed at approximately equal intervals in the circumferential direction. In this embodiment, since the roller portion 35 is in the shape of a regular hexagonal prism, the corners 35b are formed at six points on the circumference. A shaft member is provided so as to penetrate the center of the roller portion 35, and both ends thereof protrude as shafts 35c and 35d. In this embodiment, the corners 35b are formed as stripes in the axial direction between both ends of the roller part 35. Therefore, even if the scroller 30 is attached to the smartphone 50 in an inappropriate state, such as when one end of the roller part 35 is excessively raised from the touch panel 52, the corners 35b on the other side of the roller part 35 can be brought into contact with the touch panel 52.

The frame 31 is composed of an upper frame 32 and a lower frame 33, which are configured to sandwich the smartphone 50 in both the front and back directions and the width (left and right) direction of the smartphone 50. The upper frame 32 is configured to support the roller part 35 so that it can rotate freely. The upper frame 32 and the lower frame 33 are made of, for example, high-density polyethylene, polypropylene, ABS resin, etc. The separation distance between the upper frame 32 and the lower frame 33 when the smartphone 50 is sandwiched is such that, as shown in FIG. 7(A), when the roller part 35 of the scroller 30 and the touch panel 12 of the smartphone 50 face each other, the peripheral part 35a of the roller part 35 has a gap SP, and the corner part 35b of the roller part 35 is pressed against the roller part 35 to the extent that it is slightly crushed.

The upper frame 32 is formed in a thin plate shape having a rectangular shape longer than the width direction length of the smartphone 50, and a rectangular long hole (roller accommodating section 32a) that can accommodate the roller section 35 in a state where it is laid down along the longitudinal direction of the upper frame 32 is formed in almost the center. A hinge piece 32b that constitutes a hinge mechanism is formed on one end side of the upper frame 32, and a rotation shaft 34 penetrates the hinge piece 32b. The rotation shaft 34 may be integrally formed with the hinge piece 32b. A claw portion 32c that protrudes in the longitudinal direction of the upper frame 32 is formed on the other end side of the upper frame 32. A click mechanism portion 36 described later is accommodated in the side wall of the roller accommodating section 32a located on one end side of the upper frame 32, and the shaft portion 35c of the roller section 35 is connected to the click mechanism portion 36. Additionally, a bearing hole 32d is formed in the side wall of the roller housing portion 32a located at the other end, which can rotatably support the shaft portion 35d of the roller portion 35.

The lower frame 33 is formed in a rectangular shape with L-angle portions 33b and 33c at both ends that can hold both longitudinal ends of the upper frame 32. A cushioning material 39 is attached to the inside of the lower frame 33, that is, the inner surface of the smartphone storage portion 33a and the L-angle portions 33c and 33d. The cushioning material 39 is, for example, a sheet-like rubber sponge, and is attached to the inner surface of the lower frame 33 via double-sided tape or the like. The short-side width of the lower frame 33 is set to be almost the same as that of the upper frame 32. The L-angle portion 33b on one end functions as a hinge piece that constitutes a hinge mechanism together with the hinge piece portion 32b of the upper frame 32. The L-angle portion 33c on the other end is formed with a groove portion 33d that can engage the claw portion 32c of the upper frame 32. The claw portion 32c and the groove portion 33d constitute a locking mechanism.

The click mechanism 36 is a mechanism that gives a clicking sensation to the tactile sense of the user's fingers touching the roller part 35 as the roller part 35 rotates and gives a clicking sound to the user's hearing, and has a known configuration that is used, for example, in the wheel of a mouse. Although not shown in the figure, specifically, it is composed of a disk part that has multiple linear grooves radially from the center and is connected to the shaft part 35c on one end side of the roller part 35, which serves as a rotation axis, a bearing hole that supports the disk part rotatably, and a flexible claw that is fixed so as to be able to engage with the multiple linear grooves of the rotating disk part. When the disk part rotates integrally with the roller part 35, the flexible claw overcomes the unevenness so as to cross the multiple linear grooves of the disk part one after another as the disk part rotates. As a result, the click mechanism 36 generates a clicking sound and vibration, so that the user is given a clicking sound and clicking sensation associated with the rotation of the roller part 35.

By configuring the scroller 30 in this manner, as shown in Figure 6 (D), the hinge mechanism configured on one end of the upper frame 32 and the lower frame 33 allows the upper frame 32 and the lower frame 33 to rotate relatively around the pivot 34, and the locking mechanism configured on the other end of the upper frame 32 and the lower frame 33 makes it possible to maintain the smartphone 50 in a clamped state. This makes it possible for the scroller 30 to be attached to the smartphone 50 in a positional relationship in which the roller portion 35 can come into contact with the touch panel 52 (Figures 6 (A) to (C)).

The scroller 30 attached to the smartphone 50 functions as shown in FIG. 7. As shown in FIG. 7(A), when the user of the smartphone 50 rotates the roller part 35 by pulling it downward with his/her finger 91, the roller part 35 rotates in the direction of the arrow in the figure. Then, in the state of FIG. 7(A), a gap SP is formed between the peripheral surface part 35a of the roller part 35 and the touch panel 52, and the two (roller part 35 and touch panel 52) are not in contact with each other, but as the roller part 35 rotates, the corner part 35x of the roller part 35 comes into contact with the touch panel 52 (FIG. 7(B)). Then, by rotating the roller part 35 in the same direction, the corner part 35x' is pressed against the touch panel 52 and crushed (FIG. 7(C)), and the crushed corner part 35x' slides in the direction of the arrow, and by further rotating, the crushed corner part 35x' returns to its original shape (corner part 35x) (FIG. 7(D)).

Since multiple corners 35b are formed at equal intervals in the circumferential direction on the peripheral surface 35a of the roller portion 35, these multiple corners 35b come into intermittent (discontinuous) contact (contact) with the touch panel 52 through three contact processes of point → surface → point, such as corner 35x → crushed corner 35x' → corner 35x, and the planar crushed corner 35x' moves on the touch panel 52 to slide upward on the smartphone 50. As a result, the roller portion 35 comes into surface contact with the touch panel 52 as if the finger or the like were swiping toward the top of the smartphone 50 in short strokes with a predetermined short stroke, and the user's finger is given a contact sensation of the corners 35b that occurs in proportion to the amount of rotation as the roller portion 35 rotates. The amount of rotation of the roller portion 35 is the total amount of swipes made in short swipes (total amount of short swipes) and can correspond to the amount of swiping. In addition, the user can intuitively grasp the amount of rotation of the roller part 35 from the feeling of the intermittent contact of the corners 35b.

Therefore, when swiping the display 51, the user can sense the amount of swiping. Furthermore, the click mechanism 36 allows the user to feel a click or hear a click through the sense of touch or hearing as the roller unit 35 rotates, in addition to the sensation of contact with the corner 35b, so the amount of swiping can be more reliably sensed. Furthermore, since the corner 35b of the roller unit 35 is formed as a line in the axial direction between both ends of the roller unit 35, even if the scroller 30 is attached to the smartphone 50 in an inappropriate state in which one side of the roller unit 35 is excessively raised from the touch panel 12, the amount of swiping can be sensed to the user when swiping the display 51.

The corners 35b of the roller part 35 may be continuous (striated) or discontinuous (dot-like) in the axial direction of the roller part 35. The roller part 35 may also be a cylinder with a curved circumferential surface, in which case it has a continuous (striated) or discontinuous (dot-like) convex part in the axial direction. The support part does not have to be connected at both ends like the lower frame 33. It may be made up of two members with a U-shape facing each other on the opening side (open square bracket symbol "[" and closed square bracket symbol "[]" facing each other). It may also be configured so that these two members are connected to both ends of the upper frame 32.

In the example shown in FIG. 7, the case where the user of the smartphone 50 pulls the roller unit 35 downward on the smartphone 50 with the finger 91 has been described, but even when the roller unit 35 is rotated so as to be pushed upward on the smartphone 50, the roller unit 35 and the click mechanism unit 36 function in the same way as when "pulled downward." However, in this case, the roller unit 35 rotates in the opposite direction to the arrow in the figure, so the crushed corner 35x' slides downward on the touch panel 52 on the smartphone 50. As a result, the roller unit 35 comes into surface contact with the touch panel 52 as if the finger or the like were swiping downward on the smartphone 50 in short, predetermined strokes.

In the other aspect of the invention described above, a predetermined display control is performed by abutting the corner of the roller part against the touch panel (by detecting that the roller part has come into contact with the touch panel). In place of this aspect, a scroller may be provided at a position where a camera (front camera) can detect (take an image of) the rotation (amount of rotation, angle of rotation) of the roller part (for example, the camera may be provided below the roller part), so that the amount of rotation of the roller part can be recognized and the display can be controlled even if the roller part and the touch panel (display screen) are not in contact. In addition, by making the roller part smaller in diameter than that shown in FIG. 6 and preventing the roller part from coming into contact with the touch panel (display screen) when the roller part is rotated, it is possible to prevent damage to the touch panel (display screen) due to the rotation operation of the roller part.

In the above-mentioned other aspect of the invention, by rotating the roller unit, it is possible to scroll the displayed content vertically on the display, or to switch the displayed content to another content. In addition, by performing a predetermined input operation (predetermined input) using an app or the like installed on the smartphone, it is possible to associate the "rotation operation of the roller unit" with either a content scrolling operation (first display control) or a content switching operation (second display control). This allows the user to switch which operation is performed by the "rotation operation of the roller unit".

10, 50 Smartphone (mobile information terminal device), 10B Back side, 10F Front side (surface having a display), 11, 51 Display, 12, 52 Touch panel, 15 Front camera, 16 Back camera (camera on the back side), 20 Control unit (image determination means, operation determination means, content switching means, display control means, computer that controls the mobile information terminal device), 30 Scroller (scroll operation device), 32 Upper frame (frame body portion, support portion), 33 Lower frame (support portion), 35 Roller portion, 35b, 35x, 35x' Corner portion (convex portion), 35c, 35d Shaft portion (both ends of the roller portion), 36 Click mechanism portion (mechanism for imparting a clicking sensation, mechanism for imparting a clicking sound), 71a, 71b, 72 Reduced image information, 73 Summary information, 74a, 74b, 74a' Detailed information (content) 75a, 75b, 75c Related product information (other content), 90R right hand 90R (human hand), 90L left hand 90L (human hand), 91 finger 91, S103, S104 (image determination means), S105, S106 (motion determination means), S107 (content switching means), S108 (display control means).

Claims (1)

A mobile information terminal device having a camera on the back side of a surface having a display,
a motion determination means for determining whether or not a motion of a human hand or finger (hereinafter referred to as "hand, etc.") within a predetermined period of time is a predetermined motion based on a series of image information captured by the camera when the series of image information includes an image of the human hand or finger (hereinafter referred to as "hand, etc.");
a content switching means for switching the content currently being displayed on the display to another content when the motion determining means determines that the motion of the hand or the like is the predetermined motion;
Equipped with
In a case where there are a plurality of the other contents, and the content switching means determines a plurality of times in succession that the movement of the hand or the like is the predetermined movement, and the display of the plurality of the other contents is switched in sequence for each determination,
When a next display content to be displayed next on the display among the plurality of other contents has the same content as a content already displayed on the display, the content switching means switches to the other content to be displayed next to the next display content instead of the next display content and displays the same on the display.
A portable information terminal device comprising: