JP7244458B2 - Information processing device, information processing method and information processing program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

A technique is disclosed for constructing a sound statistical database in which sounds in a predetermined space are accumulated for each type.

JP 2016-180791 A

However, the above conventional technology assumes fixed-point observation using a recording device or the like installed in a predetermined space. Therefore, although the environment of a specific place such as a building room can be analyzed, it is not possible to analyze the surrounding environment and situations of moving users and classify users based on the results.

The present application has been made in view of the above, and aims to classify users based on environmental information.

An information processing apparatus according to the present application includes an acquisition unit that acquires sensor information acquired at the same time by a terminal device of a user having a sensor and identification information of the user, and based on the sensor information acquired by the terminal device, the An estimating unit for estimating a context related to the surrounding environment and situation of the terminal device; a classifying unit for grouping the users according to a classification related to the content of the sensor information of the same type based on the estimated context; and a group to which the user belongs. a recommendation unit that makes a recommendation to the user according to the above, the estimation unit further estimates the attribute of the user based on the sensor information, and the classification unit estimates the estimated attribute of the user and the context, the users are grouped according to the classification related to the content of the sensor information of the same type based on the context .

According to one aspect of an embodiment, users can be classified based on environmental information.

FIG. 1 is an explanatory diagram showing an outline of an information processing method according to an embodiment. FIG. 2 is a diagram illustrating a configuration example of an information processing system according to the embodiment; FIG. 3 is a diagram illustrating a configuration example of a terminal device according to the embodiment; FIG. 4 is a diagram illustrating a configuration example of an information providing apparatus according to the embodiment; FIG. 5 is a diagram showing an example of the user information database. FIG. 6 is a diagram showing an example of a sensor information database. FIG. 7 is a diagram showing an example of a recommendation information database. FIG. 8 is a flow chart showing a processing procedure according to the embodiment. FIG. 9 is a diagram illustrating an example of a hardware configuration;

Hereinafter, modes for implementing an information processing apparatus, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail with reference to the drawings. The information processing apparatus, information processing method, and information processing program according to the present application are not limited to this embodiment. Also, in the following embodiments, the same parts are denoted by the same reference numerals, and overlapping descriptions are omitted.

[1. Outline of information processing method]
First, an outline of an information processing method performed by an information processing apparatus according to an embodiment will be described with reference to FIG. FIG. 1 is an explanatory diagram showing an outline of an information processing method according to an embodiment. Note that in FIG. 1, an example will be described in which users are classified, grouped, and recommended based on environment information.

As shown in FIG. 1 , the information processing system 1 includes a terminal device 10 and an information providing device 100 . The terminal device 10 and the information providing device 100 are communicably connected to each other by wire or wirelessly via a network N (see FIG. 2).

The terminal device 10 is a smart device such as a smartphone or tablet used by the user U, and can communicate with any server device via a wireless communication network such as 4G (Generation) or LTE (Long Term Evolution). It is a portable terminal device that can The terminal device 10 has a screen such as a liquid crystal display, which has a touch panel function. accepts the operation of An operation performed on an area where content is displayed on the screen may be an operation on the content. In addition, the terminal device 10 may be an information processing device such as a desktop PC (Personal Computer) or a notebook PC as well as a smart device. Here, a smartphone is used as an example.

The information providing device 100 acquires sensor information (sensor data) output from various sensors (sensors) mounted on the terminal device 10, user information, and position information from the terminal device 10 of the user U. It is a processing device, and is realized by a server device, a cloud system, or the like. Here, the information providing device 100 acquires sensor information, user information, and location information from each of an unspecified number of terminal devices 10, associates the sensor information, user information, and location information, and stores them in a database. . Then, the context and attributes of the user U are estimated based on the sensor information. Also, users are classified (categorized) based on sensor information. Specifically, the information providing apparatus 100 groups users based on sensor information. For example, users listening to the same music as environmental sounds (users listening to the same music) are grouped by classification (category) related to the music. Based on these results, the information providing apparatus 100 makes various recommendations (recommendations of goods, services, etc.) to the user U. FIG. The user information is identification information or the like for uniquely identifying the user U or the terminal device 10 .

[1-1. Outline of information processing method from data collection to recommendation]
First, with reference to FIG. 1, an outline of an information processing method from data collection to recommendation will be described.

As shown in FIG. 1, the information providing device 100 receives sensor information, user information, and location information from each of the terminal devices 10 of a plurality of users U located in various places (step S1).

At this time, the terminal device 10 may be moving or staying at one place. Also, the information providing apparatus 100 may receive sensor information, user information, and location information from the same terminal apparatus 10 that moves from place to place. The sensor information is, for example, environmental sounds, atmospheric pressure, illuminance, and the like. At this time, the information providing apparatus 100 extracts environmental sounds from the sounds included in the sensor information, and classifies (categorizes) the environmental sounds by sound type. Also, the position information is the position information at the time when the terminal device 10 acquires the sensor information. Position information also corresponds to sensor information in a broad sense. The sensor information and location information also include information about the date and time when the terminal device 10 acquired the sensor information and location information.

In the example of FIG. 1, the information providing device 100 receives sensor information, user information, and location information from each of the terminal devices 10A to 10E possessed by the users U1 to U5. Note that users U1 to U5 may be the same person. Also, the terminal devices 10A to 10E may be the same device.

Then, the information providing apparatus 100 associates and stores the sensor information, the user information, and the position information received from the terminal device 10 (step S2).

For example, the information providing apparatus 100 associates sensor information, user information, and location information, accumulates them in a database, and holds them for a predetermined period (for example, one week/one month/half a year/one year). Note that, in practice, accumulation may be continued without specifying a period.

When the environmental sounds are classified by sound type, the information providing apparatus 100 stores the user information and the position information in association with each sound type.

Here, when the information providing apparatus 100 cannot acquire appropriate sensor information from the terminal device 10, the information providing apparatus 100 estimates sensor information based on the position information (step S3).

For example, if the information providing apparatus 100 cannot acquire sensor information due to an obstructive factor or sensor failure of the terminal device 10, or if the accuracy or reliability of the sensor information acquired from the terminal device 10 is considered to be low, Estimate sensor information from location information. For example, based on the position information, the information providing apparatus 100 uses sensor information acquired in the past at that position or sensor information acquired from the terminal device 10 of another user existing at the same or near position, Estimate the sensor information expected at the location. Alternatively, the information providing apparatus 100 causes a model to learn a set of position information and sensor information expected to be obtained at that position as correct data. Then, the information providing apparatus 100 acquires sensor information expected to be obtained at that position by inputting the position information to the learned model. That is, the information providing apparatus 100 acquires the sensor information output by the model when position information is input as an estimation result.

Conversely, when the information providing apparatus 100 cannot acquire appropriate position information from the terminal device 10, the information providing apparatus 100 may estimate position information based on sensor information. For example, the information providing apparatus 100 may cause the model to learn a set of sensor information and position information expected to be obtained from the sensor information as correct data. Then, the information providing apparatus 100 acquires position information from which the sensor information is expected to be obtained by inputting the sensor information into the trained model. That is, the information providing apparatus 100 may acquire the position information output by the model when the sensor information is input as the estimation result. The information providing apparatus 100 may also estimate whether the user U is indoors or outdoors based on environmental sounds, atmospheric pressure, illuminance, and the like.

Based on the sensor information, the information providing device 100 then estimates the context regarding the environment and situation in which the user U is placed (step S4).

At this time, the information providing apparatus 100 may estimate the environment and situation in which the user U is placed from a combination of sensor information and position information. For example, when the sound of flowing water is heard as the environmental sound and the location information indicates that the user is near a river, it can be estimated with higher accuracy that the user U is near the river. That is, the positional information supports the reliability of the sensor information.

Then, the information providing device 100 estimates attributes of the user U based on the sensor information (step S5).

For example, the information providing apparatus 100 estimates the user U's hobby, lifestyle (lifestyle), behavior pattern, group to which the user U belongs, and the like from sensor information. The group to which the user U belongs includes, for example, an organization such as a company, a group of people with similar interests in sports and hobbies (clubs, circles, etc.), users of the same facility or store (members, regular customers, etc.), a community (community). , local communities, etc.) and other organizations.

In addition, for example, the information providing device 100 uses environmental sounds, conversation content, atmospheric pressure, illuminance, and other information to determine whether the user U is playing sports, playing games, watching videos/idols, watching sports, playing musical instruments, listening to music, driving, touring, It can be estimated that hiking, cycling, mountain climbing, camping, BBQ (barbecue), fishing, canoeing/kayaking, skiing, and the like are performed.

Note that the information providing apparatus 100 may estimate a user's attribute from the user's context estimated based on sensor information.

Then, the information providing device 100 classifies the users U based on the sensor information (step S6).

For example, the information providing apparatus 100 extracts only environmental sounds from the sounds included in the sensor information, and groups users with similar characteristics of the environmental sounds. Note that "the characteristics of the environmental sounds are similar" means that the background music is similar (for example, the same singer's song, the same bird song, the same mechanical sound, etc.), as mere sensor numerical time-series information (for example, similarity of volume during noise, similarity of difference between quiet and noisy times, similarity of switching timing between quiet and noisy times, etc.) is high. etc.

Further, the information providing apparatus 100 may classify the users U based on the user U's context and attributes estimated based on the sensor information.

Then, the information providing device 100 makes a recommendation to the terminal device 10 of the user U based on the sensor information (step S7).

For example, the information providing device 100 may transmit recommendation information corresponding to sensor information to the user U's terminal device 10 . In addition, the information providing apparatus 100 should recommend to the user U the terminal apparatus 10 of the user U according to the classification and belonging group of the user U, or according to the result of the above estimation (context and attributes of the user U). Information about goods, services, etc. may be transmitted. In addition, the information providing apparatus 100 may recommend a product or the like that has already been purchased by another user belonging to the same classification or group but has not yet been purchased by the user U.

[2. Configuration example of information processing system]
Next, the configuration of the information processing system 1 including the information providing device 100 according to the embodiment will be described with reference to FIG. FIG. 2 is a diagram showing a configuration example of the information processing system 1 according to the embodiment. As shown in FIG. 2, the information processing system 1 according to the embodiment includes a terminal device 10 and an information providing device 100. As shown in FIG. These various devices are communicatively connected via a network N by wire or wirelessly. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

Further, the number and types of devices included in the information processing system 1 shown in FIG. 2 are not limited to those illustrated. For example, in FIG. 2, only one type of terminal device 10 is shown for simplification of illustration, but this is only an example and is not limited, and two or more types may be used.

The terminal device 10 is an information processing device used by the user U, and is a small portable or mobile device that has various sensors and can be carried or moved. For example, the terminal device 10 is a smart device such as a smart phone or a tablet terminal. In fact, feature phones, PCs (Personal Computers), PDAs (Personal Digital Assistants), car navigation systems, wearable devices such as smart watches and head-mounted displays, smart glasses, smart speakers, 360-degree cameras etc.

In addition, the terminal device 10 is compatible with wireless communication networks such as LTE (Long Term Evolution), 4G (4th Generation), 5G (5th Generation: fifth generation mobile communication system), Bluetooth (registered trademark), wireless LAN (Local It is possible to communicate with the information providing apparatus 100 by connecting to the network N via short-range wireless communication such as Area Network).

The information providing device 100 is, for example, a PC, a server device, a mainframe, a workstation, or the like. Note that the information providing apparatus 100 may be realized by cloud computing.

[3. Configuration example of terminal device]
Next, the configuration of the terminal device 10 will be described using FIG. FIG. 3 is a diagram showing a configuration example of the terminal device 10. As shown in FIG. As shown in FIG. 3, the terminal device 10 includes a communication unit 11, a display unit 12, an input unit 13, a positioning unit 14, a sensor unit 20, a control unit 30 (controller), and a storage unit 40. Prepare.

(Communication unit 11)
The communication unit 11 is connected to the network N (see FIG. 2) by wire or wirelessly, and transmits and receives information to and from the information providing apparatus 100 via the network N. FIG. For example, the communication unit 11 is implemented by a NIC (Network Interface Card), an antenna, or the like.

(Display unit 12)
The display unit 12 is a display device that displays various information such as position information. For example, the display unit 12 is a liquid crystal display (LCD) or an organic EL display (Organic Electro-Luminescent Display). Also, the display unit 12 is a touch panel display, but is not limited to this.

(Input unit 13)
The input unit 13 is an input device that receives various operations from the user U. FIG. The input unit 13 has, for example, buttons and the like for inputting characters, numbers, and the like. Moreover, when the display unit 12 is a touch panel display, a part of the display unit 12 functions as the input unit 13 . Note that the input unit 13 may be a microphone or the like that receives voice input from the user U. FIG. The microphone may be wireless.

(Positioning unit 14)
The positioning unit 14 receives signals (radio waves) transmitted from GPS (Global Positioning System) satellites, and based on the received signals, position information (for example, latitude and longitude). That is, the positioning unit 14 positions the position of the terminal device 10 . GPS is merely an example of GNSS (Global Navigation Satellite System).

Also, the positioning unit 14 can measure the position by various methods other than GPS. For example, the positioning unit 14 may measure the position using various communication functions of the terminal device 10 as described below as auxiliary positioning means for position correction and the like.

(Wi-Fi positioning)
For example, the positioning unit 14 measures the position of the terminal device 10 using the Wi-Fi (registered trademark) communication function of the terminal device 10 or the communication network provided by each communication company. Specifically, the positioning unit 14 performs Wi-Fi communication or the like and measures the position of the terminal device 10 by measuring the distance to a nearby base station or access point.

(beacon positioning)
The positioning unit 14 may also use the Bluetooth (registered trademark) function of the terminal device 10 to measure the position. For example, the positioning unit 14 measures the position of the terminal device 10 by connecting with a beacon transmitter connected by the Bluetooth function.

(geomagnetic positioning)
Further, the positioning unit 14 positions the position of the terminal device 10 based on the geomagnetism pattern of the structure measured in advance and the geomagnetic sensor provided in the terminal device 10 .

(RFID positioning)
Further, for example, if the terminal device 10 has an RFID (Radio Frequency Identification) tag function equivalent to a contactless IC card used at station ticket gates, stores, etc., or has a function of reading an RFID tag In this case, the location used is recorded together with the information that the payment was made by the terminal device 10 . The positioning unit 14 may measure the position of the terminal device 10 by acquiring such information. Also, the position may be measured by an optical sensor provided in the terminal device 10, an infrared sensor, or the like.

The positioning unit 14 may measure the position of the terminal device 10 using one or a combination of the positioning means described above, if necessary.

(Sensor unit 20)
The sensor unit 20 includes various sensors mounted on the terminal device 10 . In the example shown in FIG. 3, the sensor unit 20 includes an acceleration sensor 21, a gyro sensor 22, an atmospheric pressure sensor 23, an air temperature sensor 24, a sound sensor 25, an optical sensor 26, a magnetic sensor 27, and an image sensor ( camera) 28.

The sensors 21 to 28 described above are only examples and are not limited. That is, the sensor unit 20 may be configured to include a part of the sensors 21 to 28, or may include other sensors such as a humidity sensor in addition to or instead of the sensors 21 to 28. .

The acceleration sensor 21 is, for example, a three-axis acceleration sensor, and detects physical movements of the terminal device 10 such as movement direction, speed, and acceleration of the terminal device 10 . The gyro sensor 22 detects physical movements of the terminal device 10 such as inclination in three axial directions based on the angular velocity of the terminal device 10 and the like. The atmospheric pressure sensor 23 detects the atmospheric pressure around the terminal device 10, for example.

Since the terminal device 10 includes the above-described acceleration sensor 21, gyro sensor 22, barometric pressure sensor 23, etc., techniques such as pedestrian dead-reckoning (PDR: Pedestrian Dead-Reckoning) using these sensors 21 to 23, etc. , the position of the terminal device 10 can be determined. This makes it possible to acquire indoor position information that is difficult to acquire with a positioning system such as GPS.

For example, a pedometer using the acceleration sensor 21 can calculate the number of steps, walking speed, and distance walked. In addition, by using the gyro sensor 22, it is possible to know the traveling direction, the direction of the line of sight, and the inclination of the body of the user U. Also, from the atmospheric pressure detected by the atmospheric pressure sensor 23, the altitude at which the terminal device 10 of the user U is present and the number of floors can be known.

The temperature sensor 24 detects the temperature around the terminal device 10, for example. The sound sensor 25 detects sounds around the terminal device 10, for example. The optical sensor 26 detects the illuminance around the terminal device 10 . The magnetic sensor 27 detects, for example, geomagnetism around the terminal device 10 . The image sensor 28 captures an image around the terminal device 10 .

The atmospheric pressure sensor 23, the temperature sensor 24, the sound sensor 25, the optical sensor 26, and the image sensor 28 described above detect the atmospheric pressure, temperature, sound, and illuminance, respectively, or capture an image of the surroundings to detect the terminal device 10. It is possible to detect the surrounding environment and situations. In addition, it is possible to improve the accuracy of the location information of the terminal device 10 based on the surrounding environment and situation of the terminal device 10 .

(control unit 30)
The control unit 30 includes, for example, a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM, an input/output port, and various circuits. Also, the control unit 30 may be configured by hardware such as an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 30 includes a transmission unit 31 , a reception unit 32 and a processing unit 33 .

(Sending unit 31)
The transmission unit 31 receives, for example, various information input by the user U using the input unit 13, various information detected by the sensors 21 to 28 mounted on the terminal device 10, and the terminal device positioned by the positioning unit 14. 10 can be transmitted to the information providing device 100 via the communication unit 11 .

(Receiver 32)
The receiving unit 32 can receive various types of information provided by the information providing device 100 and requests for sensor information from the information providing device 100 via the communication unit 11 .

(Processing unit 33)
The processing unit 33 controls the entire terminal device 10 including the display unit 12 and the like. For example, the processing unit 33 can output various types of information transmitted by the transmitting unit 31 and various types of information received by the receiving unit 32 from the information providing apparatus 100 to the display unit 12 for display.

(storage unit 40)
The storage unit 40 is realized by, for example, a semiconductor memory device such as RAM (Random Access Memory) or flash memory, or a storage device such as HDD (Hard Disk Drive), SSD (Solid State Drive), or optical disk. be. Various programs, various data, and the like are stored in the storage unit 40 .

[4. Configuration example of information providing device]
Next, the configuration of the information providing device 100 according to the embodiment will be described using FIG. FIG. 4 is a diagram showing a configuration example of the information providing device 100 according to the embodiment. As shown in FIG. 4, the information providing device 100 has a communication section 110, a storage section 120, and a control section .

(Communication unit 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. Also, the communication unit 110 is connected to the network N (see FIG. 2) by wire or wirelessly.

(storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 4 , the storage unit 120 has a user information database 121 , a sensor information database 122 and a recommendation information database 123 .

(User information database 121)
The user information database 121 stores various information about the user U. FIG. For example, the user information database 121 stores various information such as user U attributes. FIG. 5 is a diagram showing an example of the user information database 121. As shown in FIG. In the example shown in FIG. 5, the user information database 121 has items such as "user ID (Identifier)", "age", "sex", "home", "work place", and "terminal ID".

“User ID” indicates identification information for identifying the user U. FIG. "Age" indicates the age of the user U identified by the user ID. Note that the "age" may be a specific age of the user U identified by the user ID, such as 35 years old. "Gender" indicates the gender of the user U identified by the user ID.

"Home" indicates location information of the home of the user U identified by the user ID. In the example shown in FIG. 5, "home" is represented by an abstract code such as "LC11", but may be latitude/longitude information or the like. Also, for example, "home" may be an area name or an address.

Further, "place of work" indicates location information of the place of work of the user U identified by the user ID (school in the case of a student). In the example shown in FIG. 5, the "place of work" is illustrated as an abstract code such as "LC12", but may be latitude/longitude information or the like. Also, for example, the "place of work" may be an area name or an address.

"Terminal ID" indicates identification information for identifying the terminal device 10 currently used by the user U identified by the user ID (terminal device in operation). It should be noted that the number of "terminal IDs" may be plural. That is, one user U may use a plurality of terminal devices at the same time. Also, the “terminal ID” may include information indicating the type of the terminal device 10 .

For example, in the example shown in FIG. 5, the age of the user U identified by the user ID "U1" is "twenties" and the gender is "male". Also, for example, user U identified by user ID "U1" indicates that home is "LC11". Also, for example, the user U identified by the user ID "U1" indicates that the place of work is "LC12". Also, for example, it indicates that the user U identified by the user ID "U1" uses the terminal device 10 identified by the terminal ID "D1".

Here, in the example shown in FIG. 5, abstract values such as "U1", "LC11", "LC12" and "D1" are used, but "U1", "LC11", "LC12" and " D1” stores information such as specific character strings and numerical values. Hereinafter, abstract values may also be illustrated in diagrams relating to other information.

The user information database 121 is not limited to the above, and may store various types of information depending on the purpose. For example, the user information database 121 may store various information about the user U's terminal device 10 . In addition, the user information database 121 is related to attributes of the user U, such as demographics (demographic attributes), psychographics (psychological attributes), geographic (geographical attributes), and behavioral (behavioral attributes). Information may be stored. For example, the user information database 121 includes name, family structure, occupation, job title, income, qualification, residence type (detached house, condominium, etc.), presence or absence of car, commute time, commute route, commuter pass section (station , routes, etc.), stations with high frequency of use (other than the nearest station to home/work place), behavior history (location information with high frequency of use), lessons (place, time of day, etc.), hobbies, interests, lifestyle, etc. Information may be stored. The user information database 121 may also store information on search queries (search keywords) that the user U has entered into a search engine or the like.

(Sensor information database 122)
The sensor information database 122 stores various information regarding sensor outputs. For example, the sensor information database 122 associates and stores sensor information, user information, and position information from the user U's terminal device 10 . FIG. 6 is a diagram showing an example of the sensor information database 122. As shown in FIG. In the example shown in FIG. 6, the sensor information database 122 has items such as "user ID", "acquisition date and time", "sensor information", "position information", "context", "attribute", and "classification".

“User ID” is user information acquired from the terminal device 10 and indicates identification information for identifying the user U who uses the terminal device 10 . Note that the “user ID” may be identification information for identifying the terminal device 10 . That is, the "user ID" may be the "terminal ID".

“Acquisition date and time” indicates the date and time when the terminal device 10 acquired the sensor information, the user information, and the position information. Note that when the information providing apparatus 100 acquires the sensor information, the user information, and the location information from the terminal device 10 in real time, the information providing apparatus 100 acquires the sensor information, the user information, and the location information from the terminal device 10. It may be the date and time of acquisition. Also, the "acquisition date and time" is not limited to the year, month, day, hour, minute, and second, and may include information related to the day of the week, daytime/nighttime, and the like.

“Sensor information” indicates outputs (detection results) of various sensors of the terminal device 10 . "Sensor information" is, for example, environmental sound, air pressure, illuminance, and the like. The environmental sounds may be classified (categorized) by sound type and stored.

“Positional information” indicates positional information at the time when the terminal device 10 acquires the sensor information. "Positional information" is positional information determined by GPS, PDR, or the like, for example.

“Context” indicates the environment or situation in which the user U is placed. The “context” is, for example, that the user U is near a river or on top of a mountain, that the user U is in the city or inside a store, that the user U is playing a sport or playing a game, or that the user U is driving a car. or that the user U is in a dark room where the right to sunlight is violated.

"Attribute" indicates an attribute of the user U based on sensor information. The “attribute” may be information about a user's attribute estimated from sensor information, for example, the user's hobby, lifestyle (lifestyle), behavior pattern, group to which the user U belongs, and the like. Note that the same user U may correspond to different attributes for each piece of sensor information. In other words, there may be multiple "attributes". For example, the same user U may have multiple hobbies.

"Classification" indicates the classification (category) of the user U based on the sensor information. For example, "classification" may be identification information (group ID) for identifying a group obtained by grouping users based on sensor information. Note that the same user U may belong to multiple groups. That is, the same user U may be assigned multiple group IDs. In addition, the "classification" may store the user ID of each user (member) belonging to the same group.

For example, in the example shown in FIG. 6, the terminal device 10 of the user U identified by the user ID "U1" has the sensor information "sensor information #11" and the location information "location information" in the acquisition date "acquisition date #11". #11” is acquired. Also, the user U identified by the user ID "U1" has the context "context #11" and the attribute "attribute #11" and is classified into the classification "classification #11".

Note that the sensor information database 122 may store various types of information, not limited to the above, depending on the purpose. For example, the sensor information database 122 may store information regarding expiration dates for storing sensor information. In addition, the sensor information database 122 may store information related to sensor information analysis results (environmental evaluation results). Further, the sensor information database 122 may store information related to comments about the situation at the time of sensor information acquisition input by the user U of the terminal device 10 . The comment may be text or voice.

(Recommendation information database 123)
The recommendation information database 123 stores various information related to recommendations for the user U. FIG. For example, the recommendation information database 123 stores information on goods, services, etc. recommended to the user U. FIG. FIG. 7 is a diagram showing an example of the recommendation information database 123. As shown in FIG. In the example shown in FIG. 7, the recommendation information database 123 has items such as "recommendation ID", "subject", and "recommendation content".

“Recommendation ID” indicates identification information for identifying each recommendation.

"Target" indicates the target of the recommendation. The “subject” may be identification information (user ID) for identifying the user U who is the target of the recommendation. Further, the “target person” may be information indicating the classification of the user U who is the target of the recommendation, or identification information (group ID) for identifying the group which is the target of the recommendation.

"Recommendation content" indicates information about goods, services, etc. recommended to the target person. Note that the “recommendation content” may be, for example, “recommendation” or “best-selling ranking”. Also, the “recommendation content” may be, for example, an advertisement, a sales promotion site, guidance, coupon information, or the like.

For example, in the example shown in FIG. 7, the recommendation identified by the recommendation ID “recommendation #1” is to provide the target person “subject #1” with the goods, services, etc. indicated by the recommendation content “recommendation content #1”. Indicates a recommendation.

It should be noted that the recommendation information database 123 may store various types of information, not limited to the above, depending on the purpose. For example, the recommendation information database 123 may store information related to word-of-mouth, reputation, evaluation results based on predetermined evaluation indexes, etc. regarding goods, services, etc. recommended to the target person.

(control unit 130)
Returning to FIG. 4, the description is continued. The control unit 130 is a controller, and for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like controls the information providing apparatus 100. Various programs (corresponding to an example of an information processing program) stored in the internal storage device are executed by using a storage area such as a RAM as a work area. In the example shown in FIG. 4 , the control unit 130 has an acquisition unit 131 , an accumulation unit 132 , an estimation unit 133 , a classification unit 134 , a recommendation unit 135 and an output unit 136 .

(Acquisition unit 131)
The acquisition unit 131 acquires sensor information (sensor data), user information, and location information from each of the terminal devices 10 of multiple users U located in various locations via the communication unit 110 .

(Collecting unit 132)
The accumulation unit 132 associates the sensor information, the user information, and the location information received from the terminal device 10 and accumulates them in a database. Note that the accumulation unit 132 may be part of the acquisition unit 131 .

Further, when the sensor information received from the terminal device 10 includes sound, the accumulation unit 132 extracts only the environmental sound from the sound. For example, the accumulation unit 132 may filter sounds included in sensor information to extract only environmental sounds. At this time, the accumulation unit 132 may classify (categorize) the environmental sounds by sound type. That is, the accumulation unit 132 also functions as an environmental sound extraction unit and a classification unit.

Note that the accumulation unit 132 may extract environmental sounds from sounds included in the sensor information received from the terminal device 10 using a model based on machine learning. In this case, the accumulation unit 132 may cause the model to learn sets of sounds included in the sensor information received from the terminal device 10 and environmental sounds categorized by type of sound, for example, as correct data. Then, the accumulation unit 132 extracts environmental sounds classified by sound type by inputting sounds included in the sensor information to the trained model. That is, the accumulation unit 132 may use the environmental sound output by the model when the sound included in the sensor information is input as the extraction result.

Any type of model can be adopted as the model. For example, the accumulation unit 132 may adopt SVM (Support Vector Machine) or DNN (Deep Neural Network) as a model. Here, the DNN may be a CNN (Convolutional Neural Network) or an RNN (Recurrent Neural Network). Also, the RNN may be LSTM (Long short-term memory) or the like. That is, any model can be adopted as the model. The model may also be a model realized by combining a plurality of models, such as a model combining CNN and RNN.

Learning is, for example, deep learning using a deep neural network (DNN: Deep Neural Network). Data mining and other machine learning algorithms may also be used. The accumulation unit 132 learns the model using the various learning methods described above.

(Estimation unit 133)
The estimation unit 133 estimates attributes of the user U based on sensor information from the user U's terminal device 10 . For example, the estimation unit 133 estimates the user's hobby, lifestyle (lifestyle), behavior pattern, group to which the user belongs, and the like from the sensor information. The group to which the user belongs includes, for example, an organization such as a company, a group of people with similar interests in sports and hobbies (clubs, circles, etc.), users of the same facility or store (members, regular customers, etc.), a community (community, local communities, etc.) and other organizations. Note that the estimation unit 133 may be part of the classification unit 134 .

Note that the estimation unit 133 may estimate attributes of the user U using a model based on machine learning. In this case, the estimating unit 133 inputs, for example, sensor information received from the terminal device 10 into the model, and learns the user's hobby, lifestyle (lifestyle), behavior pattern, group to which the user belongs, etc. as correct data.

Moreover, the estimation part 133 estimates sensor information based on a positional information, when suitable sensor information cannot be acquired from the terminal device 10. FIG. For example, the estimating unit 133 uses sensor information acquired in the past at the position based on the position information, or sensor information acquired from the terminal device 10 of another user existing at the same or close position to determine the position Estimate the sensor information expected to be obtained in Alternatively, the estimation unit 133 may estimate sensor information using a model based on machine learning. In this case, the estimation unit 133 causes the model to learn a set of position information and sensor information expected to be obtained at that position as correct data. Then, the estimation unit 133 estimates sensor information by inputting position information to the trained model. That is, the estimation unit 133 uses the sensor information output by the model when the position information is input as the estimation result.

On the contrary, the estimation unit 133 estimates the position information based on the sensor information when the appropriate position information cannot be acquired from the terminal device 10 . For example, the estimation unit 133 causes the model to learn a set of sensor information and position information linked to the sensor information as correct data. Then, the estimation unit 133 estimates the position information by inputting the sensor information to the learned model. That is, the estimation unit 133 uses the position information output by the model when the sensor information is input as the estimation result.

The estimation unit 133 may also estimate whether the user U is indoors or outdoors based on environmental sounds, atmospheric pressure, illuminance, and the like.

(Classification unit 134)
The classification unit 134 classifies (categorizes) the user U based on the sensor information from the user U's terminal device 10 . Also, the classification unit 134 groups users based on sensor information or classification.

Note that the classification unit 134 may classify and group users using a model based on machine learning. In this case, the classification unit 134 inputs, for example, sensor information received from the terminal device 10 into the model, and performs learning using the user's classification and belonging group as correct data.

(Recommendation unit 135)
The recommendation unit 135 makes a recommendation to the terminal device 10 of the user U based on the sensor information. For example, the recommendation unit 135, according to the sensor information, according to the user U's classification or belonging group, or according to the result of the above estimation, uses information regarding goods, services, etc. to be recommended to the user U as recommendation information. decide. In addition, the recommendation unit 135 may determine, as the recommendation information, information related to a product or the like that has already been purchased by another user belonging to the same classification or group but has not yet been purchased by the user U.

(Output unit 136)
The output unit 136 transmits recommendation information to the terminal device 10 of the user U via the communication unit 110 . Note that the output unit 136 may be part of the recommendation unit 135 .

[5. Processing procedure]
Next, a processing procedure performed by the information providing apparatus 100 according to the embodiment will be described with reference to FIG. FIG. 8 is a flow chart showing a processing procedure according to the embodiment. Note that the processing procedure described below is repeatedly executed by the control unit 130 of the information providing apparatus 100 .

As shown in FIG. 8, the information providing device 100 acquires sensor information (sensor data), user information, and location information from each of the terminal devices 10 of a plurality of users U located in various places (step S101).

The information providing apparatus 100 associates the sensor information, the user information, and the position information and stores them in the database (step S102). For example, the information providing apparatus 100 extracts environmental sounds from the sounds included in the sensor information, classifies (categorizes) the extracted environmental sounds for each sound type, and classifies the classified environmental sounds for each sound type for the user. Information and location information are linked and stored.

At this time, the information providing device 100 determines whether or not appropriate sensor information has been acquired from the terminal device 10 of the user U (step S103).

If the information providing apparatus 100 cannot acquire appropriate sensor information from the terminal device 10 of the user U (step S103; No), the information providing apparatus 100 estimates sensor information based on the position information (step S104).

When the information providing device 100 can acquire appropriate sensor information from the terminal device 10 of the user U (step S103; Yes), or when the sensor information is estimated (step S104), based on the sensor information , the context regarding the environment and situation in which the user U is placed is estimated (step S105). For example, the information providing device 100 estimates that the user U is near a river from the sound of flowing water, the chirping of birds, and the like.

Also, the information providing apparatus 100 estimates attributes of the user U based on the sensor information (step S106). For example, the information providing apparatus 100 estimates the user's hobby, lifestyle (lifestyle), behavior pattern, group to which the user belongs, and the like from sensor information.

The information providing device 100 also classifies the users U based on the sensor information (step S107). For example, the information providing apparatus 100 extracts only environmental sounds from the sounds included in the sensor information, and groups users with similar characteristics of the environmental sounds.

Then, the information providing device 100 makes a recommendation to the terminal device 10 of the user U based on the sensor information. For example, the information providing apparatus 100 may provide information on goods, services, etc. to be recommended to the user U as recommendation information, according to sensor information, according to the classification or belonging group of the user U, or according to the above estimated result. , and the recommendation information is transmitted to the terminal device 10 of the user U (step S108).

[6. Modification]
The terminal device 10 and the information providing device 100 described above may be implemented in various different forms other than the above embodiments. So, below, the modification of embodiment is demonstrated.

In the above-described embodiment, the terminal device 10 may be put away in the pocket of the user U's clothes, bag (bag, rucksack, pouch), or the like. Therefore, the terminal device 10 may transmit sensor information to the information providing device 100 only when it is being used by the user U (operation, video viewing, etc.). For example, the information providing apparatus 100 acquires sensor information when the terminal device 10 is running some application program (app), and does not acquire sensor information when no app is running. to Alternatively, the information providing apparatus 100 acquires sensor information when the terminal device 10 is in a non-sleep state, and does not acquire sensor information when the terminal device 10 is in a sleep state. Note that "does not acquire sensor information" includes "does not store sensor information even if acquired".

In addition, the information providing apparatus 100 acquires the operation history of the terminal device 10 and the like together with the sensor information. is stored in the pocket of user U's clothes, bag, or the like, and the sensor information from the terminal device 10 may not be acquired.

Further, the information providing apparatus 100 presumes that the terminal device 10 is stored in the user U's clothing pocket, bag, or the like, based on sensor information (environmental sound, air pressure, illuminance, etc.) or changes thereof. In some cases, the sensor information from the terminal device 10 may not be acquired.

Note that the information providing device 100 may be implemented as one function in the user U's terminal device 10 . For example, the information providing device 100 may be one of the user U's terminal devices 10 .

[7. effect〕
As described above, the information providing device 100 according to the present application includes the acquisition section 131 and the classification section 134 . The acquisition unit 131 acquires sensor information and user identification information acquired at the same time by a user's terminal device having a sensor. The classification unit 134 classifies users based on sensor information. The classification unit 134 groups users with similar environmental sound characteristics. Furthermore, the information providing device 100 includes an estimating unit 133 and a recommending unit 135 . The estimation unit 133 estimates user attributes from sensor information. The recommendation unit 135 makes a recommendation to the user based on the user's classification. In this way, the information providing apparatus 100 according to the present application can classify users based on environmental information by utilizing HaaS (Hardware as a Service).

[8. Hardware configuration]
Also, the terminal device 10 and the information providing device 100 according to the above-described embodiments are implemented by a computer 1000 configured as shown in FIG. 9, for example. The information providing apparatus 100 will be described below as an example. FIG. 9 is a diagram illustrating an example of a hardware configuration; The computer 1000 is connected to an output device 1010 and an input device 1020, and an arithmetic device 1030, a primary storage device 1040, a secondary storage device 1050, an output I/F (Interface) 1060, an input I/F 1070, and a network I/F 1080 are buses. It has a form connected by 1090.

The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The arithmetic unit 1030 is implemented by, for example, a CPU (Central Processing Unit), MPU (Micro Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or the like.

The primary storage device 1040 is a memory device such as a RAM (Random Access Memory) that temporarily stores data used for various calculations by the arithmetic device 1030 . The secondary storage device 1050 is a storage device in which data used for various calculations by the arithmetic device 1030 and various databases are registered. State Drive), flash memory, or the like. The secondary storage device 1050 may be an internal storage or an external storage. Also, the secondary storage device 1050 may be a removable storage medium such as a USB memory or an SD (Secure Digital) memory card. Also, the secondary storage device 1050 may be a cloud storage (online storage), a NAS (Network Attached Storage), a file server, or the like.

The output I/F 1060 is an interface for transmitting information to be output to the output device 1010 that outputs various information such as a display, a projector, and a printer. (Digital Visual Interface), HDMI (registered trademark) (High Definition Multimedia Interface), and other standardized connectors. Also, the input I/F 1070 is an interface for receiving information from various input devices 1020 such as a mouse, keyboard, keypad, buttons, scanner, etc., and is realized by, for example, USB.

Also, the output I/F 1060 and the input I/F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, output device 1010 and input device 1020 may be wireless devices.

Also, the output device 1010 and the input device 1020 may be integrated like a touch panel. In this case, the output I/F 1060 and the input I/F 1070 may also be integrated as an input/output I/F.

Note that the input device 1020 includes, for example, optical recording media such as CDs (Compact Discs), DVDs (Digital Versatile Discs), PDs (Phase change rewritable discs), magneto-optical recording media such as MOs (Magneto-Optical discs), and tapes. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like.

Network I/F 1080 receives data from other devices via network N and sends the data to arithmetic device 1030, and also transmits data generated by arithmetic device 1030 via network N to other devices.

Arithmetic device 1030 controls output device 1010 and input device 1020 via output I/F 1060 and input I/F 1070 . For example, arithmetic device 1030 loads a program from input device 1020 or secondary storage device 1050 onto primary storage device 1040 and executes the loaded program.

For example, when the computer 1000 functions as the information providing device 100 , the arithmetic device 1030 of the computer 1000 implements the functions of the control unit 130 by executing a program loaded on the primary storage device 1040 . Further, arithmetic device 1030 of computer 1000 may load a program acquired from another device via network I/F 1080 onto primary storage device 1040 and execute the loaded program. Further, the arithmetic unit 1030 of the computer 1000 may cooperate with another device via the network I/F 1080, and call functions, data, etc. of the program from another program of the other device for use.

[9. others〕
Although the embodiments of the present application have been described above, the present invention is not limited by the contents of these embodiments. In addition, the components described above include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those within the so-called equivalent range. Furthermore, the components described above can be combined as appropriate. Furthermore, various omissions, replacements, or modifications of components can be made without departing from the gist of the above-described embodiments.

Further, among the processes described in the above embodiments, all or part of the processes described as being automatically performed can be manually performed, or the processes described as being performed manually can be performed manually. All or part of this can also be done automatically by known methods. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the illustrated one, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

For example, the information providing apparatus 100 described above may be implemented by a plurality of server computers, and depending on the function, may be implemented by calling an external platform or the like using an API (Application Programming Interface), network computing, or the like. can be changed flexibly.

Also, the above-described embodiments and modifications can be appropriately combined within a range that does not contradict the processing content.

Also, the above-mentioned "section, module, unit" can be read as "means", "circuit", or the like. For example, the acquisition unit can be read as acquisition means or an acquisition circuit.

1 information processing system 10 terminal device 14 positioning unit 20 sensor unit 100 information providing device 110 communication unit 120 storage unit 121 user information database 122 sensor information database 123 recommendation information database 130 control unit 131 acquisition unit 132 accumulation unit 133 estimation unit 134 classification unit 135 recommendation section 136 output section

Claims (15)

an acquisition unit that acquires sensor information and identification information of the user simultaneously acquired by a terminal device of a user having a sensor;
an estimating unit that estimates a context related to the environment and situation around the terminal device based on the sensor information acquired by the terminal device;
a classifying unit that groups the users by classifying the content of the sensor information of the same type based on the estimated context;
a recommendation unit that makes a recommendation to the user according to the group to which the user belongs;
with
The estimation unit further estimates attributes of the user based on the sensor information,
The classifying unit groups the users by classifying the content of the sensor information of the same type based on the estimated attribute of the user and the context.
An information processing device characterized by: an extraction unit that extracts an environmental sound from the sound included in the sensor information;
further comprising
The information processing apparatus according to claim 1, wherein the classifying unit groups the users having similar characteristics of the environmental sounds. The information processing apparatus according to claim 2, wherein the classification unit groups the users with similar background sounds among the environmental sounds. The information processing apparatus according to claim 2, wherein the classifying unit groups the users having a high similarity as sensor numerical value time-series information to background sounds among the environmental sounds. The information processing apparatus according to any one of claims 1 to 4, wherein the estimation unit estimates a hobby of the user as an attribute of the user from the sensor information. The information processing apparatus according to any one of claims 1 to 5, wherein the estimation unit estimates the user's lifestyle as the user's attribute from the sensor information. The information processing apparatus according to any one of claims 1 to 6 , wherein the estimation unit estimates the user's action pattern as the user's attribute from the sensor information. The information processing apparatus according to any one of claims 1 to 7 , wherein the estimation unit estimates a group to which the user belongs as an attribute of the user from the sensor information. The acquisition unit further acquires location information from the terminal device,
The information processing apparatus according to any one of claims 1 to 8 , wherein the estimation unit estimates the sensor information based on the position information when the sensor information cannot be acquired. 10. Of claims 1 to 9 , wherein the recommendation unit recommends to the user products that other users belonging to the same belonging group have already purchased and the user has not yet purchased. The information processing device according to any one of the above. an acquisition unit that acquires sensor information and identification information of the user simultaneously acquired by a terminal device of a user having a sensor;
an estimating unit that estimates a context related to the environment and situation around the terminal device based on the sensor information acquired by the terminal device;
a classifying unit that groups the users by classifying the content of the sensor information of the same type based on the estimated context;
a recommendation unit that makes a recommendation to the user according to the group to which the user belongs;
with
The recommendation unit recommends to the user products that other users belonging to the same belonging group have already purchased and the user has not yet purchased.
An information processing device characterized by: An information processing method executed by an information processing device,
an acquisition step of acquiring sensor information and identification information of the user simultaneously acquired by a terminal device of a user having a sensor;
an estimating step of estimating a context related to the surrounding environment and situation of the terminal device based on the sensor information acquired by the terminal device;
a classification step of grouping the users by classification related to the content of the sensor information of the same type based on the estimated context;
a recommendation step of making a recommendation to the user according to the group to which the user belongs;
including
In the estimating step, further estimating attributes of the user based on the sensor information,
In the classifying step, the users are grouped by classifying contents of the sensor information of the same type based on the estimated attribute of the user and the context.
An information processing method characterized by: an acquisition procedure for acquiring sensor information and identification information of the user simultaneously acquired by a terminal device of a user having a sensor;
An estimation procedure for estimating a context related to the surrounding environment and situation of the terminal device based on the sensor information acquired by the terminal device;
a classification procedure for grouping the users by classification related to the content of the sensor information of the same type based on the estimated context;
A recommended procedure for making a recommendation to the user according to the group to which the user belongs;
on the computer , and
In the estimation step, further estimating attributes of the user based on the sensor information,
In the classification step, the users are grouped by classification related to the contents of the sensor information of the same type based on the estimated attributes of the users and the context.
An information processing program characterized by : An information processing method executed by an information processing device,
an acquisition step of acquiring sensor information and identification information of the user simultaneously acquired by a terminal device of a user having a sensor;
an estimating step of estimating a context related to the surrounding environment and situation of the terminal device based on the sensor information acquired by the terminal device;
a classification step of grouping the users by classification related to the content of the sensor information of the same type based on the estimated context;
a recommendation step of making a recommendation to the user according to the group to which the user belongs;
including
In the recommendation step, a product that has already been purchased by another user belonging to the same belonging group and has not yet been purchased by the user is recommended to the user.
An information processing method characterized by: an acquisition procedure for acquiring sensor information and identification information of the user simultaneously acquired by a terminal device of a user having a sensor;
An estimation procedure for estimating a context related to the surrounding environment and situation of the terminal device based on the sensor information acquired by the terminal device;
a classification procedure for grouping the users by classification related to the content of the sensor information of the same type based on the estimated context;
A recommended procedure for making a recommendation to the user according to the group to which the user belongs;
on the computer, and
In the recommendation procedure, a product that has already been purchased by another user belonging to the same belonging group and has not yet been purchased by the user is recommended to the user.
An information processing program characterized by:

Images