JP6850215B2 - Information processing equipment, information processing methods, and programs - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program.

For the purpose of grasping service trends, various data are aggregated and provided to service providers. Aggregation of various data includes batch processing that aggregates data for a certain period of time and stream processing that aggregates data in real time, but in recent years, "breaking news" has been required as a business requirement. Stream processing is becoming more and more popular (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2008-139980

By using stream processing, it is possible to grasp the trend of the service in real time, but in reality, there may be a delay in data transfer due to a failure of the device that provides the service, etc. In some cases, there was a problem with reliability. In the conventional technique, since there is a possibility of such a delay in data transfer, it is not possible to provide the aggregated result of various data as a definite value until a certain period of time has passed. Also, in batch processing, fixed values can be aggregated by considering the effect of data transfer delays at the stage of collecting various data, but since aggregation takes time, it is possible to meet the demand for "quick news". I couldn't.

The present invention has been made in consideration of such circumstances, and provides an information processing device, an information processing method, and a program capable of providing an information processing device, an information processing method, and a program capable of providing a fixed rate of the aggregated results while maintaining breaking news. One of the purposes is to provide.

One aspect of the present invention is an acquisition unit that acquires stream data to be aggregated and first operation status data indicating an operation status of a first apparatus that outputs the stream data, and the acquisition unit acquired by the acquisition unit. It includes an aggregation unit that aggregates stream data, and a confirmation rate calculation unit that calculates the confirmation rate of the aggregated value aggregated by the aggregation unit based on the first operation status data acquired by the acquisition unit. It is an information processing device.

According to one aspect of the present invention, it is possible to provide the finalization rate while maintaining the breaking news of the aggregated results.

It is a figure which shows an example of the use environment of the stream processing server which concerns on embodiment. It is a figure which shows an example of the access log which concerns on embodiment. It is a functional block diagram which shows the functional structure of the log collection server which concerns on embodiment. It is a functional block diagram which shows the functional structure of the stream processing server which concerns on embodiment. It is a figure which shows an example of the delay actual data which concerns on embodiment. It is a figure which shows an example of the delay actual data which concerns on embodiment. It is a figure which shows an example of the delay actual data which concerns on embodiment. It is a sequence diagram which shows the operation of the system including the stream processing server which concerns on embodiment. It is a figure which shows an example of the aggregation result by the stream processing server which concerns on embodiment. It is a figure which shows an example of the management screen which shows the aggregated value and the confirmation rate which concerns on embodiment.

Hereinafter, embodiments of the information processing apparatus, information processing method, and program of the present invention will be described with reference to the drawings.

[Overview]
The stream processing server (an example of an information processing device) of the embodiment is a device that aggregates various data in real time by stream processing and provides the aggregation result. In addition to the aggregation result, the stream processing server provides a confirmation rate of the aggregation result.

In the present embodiment, "stream processing" means sequential processing (real-time processing) of stream data to be processed. Unlike batch processing, which collectively processes data to be processed for a certain period of time, stream processing can maintain breaking news (real-time performance) of aggregated results. Further, the "stream data" is data (real-time data) in chronological order that is sequentially output by the apparatus.

Further, in the present embodiment, the "determination rate" is an index indicating the accuracy (reliability) of the aggregation result. For example, the confirmation rate is the aggregation result (data delay) that is finally obtained after the current aggregation result eliminates the data delay when there is a data transfer delay (hereinafter referred to as "data delay"). It is an index showing how much it matches with the aggregated result (equivalent to the aggregated result obtained when) does not occur.

For example, the confirmation rate of the aggregation result for the "total value" such as the number of accesses calculated based on the access log of the web server is obtained at the time of aggregation with respect to the original number of accesses obtained when there is no data delay. Indicates the ratio of the number of accesses (progress rate of aggregation processing of the number of accesses). For example, when the confirmation rate of the number of accesses is "0.5", it means that the number of accesses of 100 is originally supposed to be obtained as the aggregated value, but at the time of aggregation, only the number of accesses for 50 is aggregated.

Further, for example, the confirmation rate of the aggregated value with respect to the "ratio" such as the click rate calculated based on the access log is the click rate currently obtained with respect to the original click rate obtained when there is no data delay. It is an index value indicating the accuracy of. This index value is calculated based on a predetermined standard.

The data to be processed in this embodiment includes, for example, an access log output by a web server, activity data output by various activity meters, and various devices used in the field of IoT (Internet of Things). Data to be used is included. In the following, an example in which the access log collected and output by the web server is processed will be described.

[overall structure]
FIG. 1 is a diagram showing an example of a usage environment of the stream processing server 40. The stream processing server 40 is connected to the network NW. In addition to the stream processing server 40, the network NW includes, for example, at least one terminal device 10, at least one service providing server 20 (an example of the first device), and at least one log collecting server 30 (of the second device). (Example) and at least one report providing server 50 are connected. The network NW includes, for example, a part or all of the Internet, WAN (Wide Area Network), LAN (Local Area Network), provider equipment, wireless base station, dedicated line, and the like.

[Terminal device]
The terminal device 10 is operated by a user who uses the service provided by the service providing server 20. The terminal device 10 is, for example, a personal computer, a mobile phone such as a smartphone, a tablet terminal, or a computer device such as a PDA (Personal Digital Assistant).

When the terminal device 10 receives a predetermined operation from the user, the terminal device 10 displays the web page provided by the service providing server 20 by the browser installed in advance. Further, when the terminal device 10 receives a predetermined operation from the user, the terminal device 10 may communicate with the service providing server 20 via the application installed in advance to acquire the content to be displayed or reproduced on the application. The content is, for example, moving image data, image data, audio data, text data, or the like. In the following, the service providing server 20 will be described as providing a web page.

[Service provider server]
The service providing server 20 is a web server device that provides a web page referenced by the browser of the terminal device 10 via the Internet. This web page constitutes a search site, a news site, a shopping site, an auction site, a weather forecast site, an SNS (Social Networking Service) site, a game site, and the like.

The service providing server 20 outputs the access log A in response to the access by the terminal device 10 to the web page provided by the service providing server 20. For example, one line of data is added to the access log A every time the terminal device 10 accesses the web page. FIG. 2 is a diagram showing an example of the access log A. The access log A includes, for example, a "user ID" that identifies a user who uses the terminal device 10, a "content ID" that identifies content associated with a web page (including an advertisement displayed on the web page), and a terminal. The "date" indicating the date when the access from the device 10 was processed, the "target URL" indicating the URL (Uniform Resource Locator) of the web page accessed from the terminal device 10, and the web page of the transition source to the target URL. A "transition source URL" or the like indicating a URL is included.

The "user ID" is, for example, a login ID used when logging in to a website provided by the service providing server 20, a cookie (HTTP cookie) managed for each web browser provided in the terminal device 10, and the terminal device 10. IP address, etc.

Further, the service providing server 20 outputs a service operation status report B (first operation status data) indicating its own operation status. The service operation status report B is, for example, "normal" indicating that the service is operating normally, "delay" indicating that the processing is delayed, and "stop" indicating that the operation is stopped due to a failure or the like. Includes data indicating the operating status of such as. In the case of "delay", the service operation status report B may include data indicating the degree of delay (delay time, etc.).

[Log collection server]
The log collection server 30 collects the access log A and the service operation status report B from the service providing server 20 at a predetermined cycle (for example, every second). The log collection server 30 outputs the data included in the collected access log A and the service operation status report B to the stream processing server 40. The log collection server 30 may be arranged between the service providing server 20 and the stream processing server 40. FIG. 3 is a functional block diagram showing a functional configuration of the log collection server 30. The log collection server 30 includes, for example, a collection unit 31, a monitoring unit 33, and an output unit 35.

The collecting unit 31 includes, for example, a communication interface such as a NIC. The collection unit 31 communicates with the service providing server 20 via the network NW, and collects the access log A and the service operation status report B at a predetermined cycle (for example, every second).

The monitoring unit 33 monitors the operation status of the log collection server 30 and generates a log collection operation status report C (second operation status data) showing the operation status. The log collection operation status report C is, for example, "normal" indicating that the operation is normal, "delay" indicating that the processing is delayed, and "stop" indicating that the operation is stopped due to a failure or the like. ”, Etc., which includes data indicating the operating status. In the case of "delay", the log collection operation status report C may include data indicating the degree of delay (delay time, etc.).

The output unit 35 includes, for example, a communication interface such as a NIC. The output unit 35 is data that has not been transmitted to the stream processing server 40 among the data included in the access log A collected by the collection unit 31 (data of at least one line in the access log A. Hereinafter, “stream data A1””. Is output to the stream processing server 40 via the network NW. Further, the output unit 35 outputs the service operation status report B and the log collection operation status report C to the stream processing server 40 via the network NW. The output unit 35 may share a communication interface with the above-mentioned collection unit 31.

[Stream processing server]
The stream processing server 40 aggregates various data in real time by stream processing and generates an aggregation result (aggregated value). Further, the stream processing server 40 calculates the determination rate of the aggregation result in addition to the aggregation result. FIG. 4 is a functional block diagram showing a functional configuration of the stream processing server 40. The stream processing server 40 includes, for example, an acquisition unit 41, an aggregation unit 43, a confirmation rate calculation unit 45, and a storage unit 47.

The acquisition unit 41 includes, for example, a communication interface such as a NIC. The stream data A1, the service operation status report B, and the log collection operation status report C output by the log collection server 30 are acquired.

The aggregation unit 43 aggregates the stream data A1 according to a predetermined aggregation standard. The aggregation unit 43 stores the aggregation value E, which is the aggregation result, in the storage unit 47. For example, the aggregation unit 43 aggregates the number of accesses (impressions) for each content ID as the aggregation value E. When the total number of accesses as the total value E is already stored in the storage unit 47, the totaling unit 43 adds the newly totaled number of accesses to the number of accesses stored in the storage unit 47. Further, the totaling unit 43 may total the number of accesses for each predetermined period (for example, for each hour).

In the aggregation unit 43, for example, when the web page is an advertisement page and the click log, conversion log, etc. are acquired by the service providing server 20, the number of clicks (click rate) and the number of conversions are obtained in addition to the number of accesses. (Number of conversions) and the like may be totaled.

The confirmation rate calculation unit 45 calculates the confirmation rate F of the aggregated value E aggregated by the aggregation unit 43. The confirmation rate F is, for example, a value from 0.0 to 1.0, and the closer it is to 0.0, the lower the accuracy of the aggregation result, and the closer it is to 1.0, the higher the accuracy of the aggregation result. .. For example, the confirmation rate calculation unit 45 calculates the operation status (ratio of the number of operating units, load status, etc.) of the service providing server 20 based on the service operation status report B and the log collection operation status report C, and calculates the current log. Calculate the delay status. For example, the confirmation rate calculation unit 45 estimates the amount of delayed logs by performing a load test in advance, and calculates the current log delay status based on the estimated amount of delayed logs. Further, the confirmation rate calculation unit 45 multiplies the calculated log delay status by a predetermined actual coefficient based on the past delay actual. This makes it possible to calculate the operating status on the safer side.

Further, the confirmation rate calculation unit 45 predicts the delay status of the log based on the service operation status report B, the log collection operation status report C, and the delay actual data D (actual data) stored in the storage unit 47. The confirmation rate F is calculated. The delay record data D is data indicating the past delay record. That is, the confirmation rate calculation unit 45 supplies both the service operation status report B and the log collection operation status report C and the delay actual data D regarding the processing delay of at least one of the service providing server 20 and the log collection server 30. Based on this, the confirmation rate F is calculated. The confirmation rate calculation unit 45 may calculate the confirmation rate F based on the operation status report of other route devices and the delay actual data in addition to the service providing server 20 and the log collection server 30.

5 to 7 are diagrams showing an example of delay actual data D. The delay record data D shown in FIG. 5 includes data in which the delay time (time) and the fluctuation range of the confirmation rate (an example of the amount of fluctuation) are defined for each type of failure based on the past delay record. .. For example, in the case of the failure type “H”, the delay time is “5.2” time, and the fluctuation range of the confirmation rate is “−0.50”. When the confirmation rate calculation unit 45 determines that the failure type "H" has occurred based on the service operation status report B and the log collection operation status report C, the confirmation rate is subtracted by 0.50 from 1. The confirmation rate F is calculated. Further, the confirmation rate calculation unit 45 may calculate the confirmation rate F in consideration of the delay time.

Further, the delay record data D shown in FIG. 6 shows the delay time (time) for each operation rate indicating the ratio of the service providing servers 20 that are operating among the plurality of service providing servers 20 that exist based on the past delay results. ) And the fluctuation range of the fixed rate are defined. For example, when the operating rate α is “0.95 ≦ α <1”, the delay time is “0.8” time, and the fluctuation range of the confirmation rate is “−0.05”. When the confirmation rate calculation unit 45 determines that the operation rate α is “0.95 ≦ α <1” based on the service operation status report B, the confirmation rate calculation unit 45 subtracts the confirmation rate from 1 by 0.05 and determines the confirmation rate F. Is calculated. Further, the confirmation rate calculation unit 45 may calculate the confirmation rate F in consideration of the delay time.

Further, the delay actual data D shown in FIG. 7 includes data in which the fluctuation range of the confirmation rate is shown for each elapsed time (time) after the occurrence of the failure based on the past delay actual. For example, when the elapsed time t after the occurrence of a failure is “0 ≦ t <0.5”, the fluctuation range of the confirmation rate is “−0.10”. When the confirmation rate calculation unit 45 determines that the elapsed time t after the failure occurs is “0 ≦ t <0.5” based on the service operation status report B and the log collection operation status report C, the confirmation rate is set to 1. The confirmation rate F is calculated by subtracting 0.10 from. Further, when the confirmation rate calculation unit 45 determines that the elapsed time t after the failure occurs is “1 ≦ t”, the switching of the data transfer route is completed due to the redundant configuration (cluster configuration) of the device or the like, and the influence of the failure. Is presumed to have already been resolved, so the fluctuation range of the confirmation rate is set to "0". As a result, the value of the confirmation rate F returns to "1.0". That is, the confirmation rate calculation unit 45 calculates the confirmation rate F so that the confirmation rate F increases as the elapsed time after the occurrence of the failure increases.

Further, the confirmation rate calculation unit 45 may reduce the confirmation rate F by multiplying it by a predetermined coefficient based on the elapsed time. For example, the confirmation rate calculation unit 45 may linearly correct the data for which 0.8 to 2 hours have passed so that the confirmation rate F becomes 1. Further, the confirmation rate calculation unit 45 may set the confirmation rate to a fixed value (maximum value) of 1.0 according to the elapsed time (when the predetermined time is exceeded). Further, when it is assumed that an unnecessary access log (for example, a log based on the access of an aggressive user) exists in the log currently received, the confirmation rate calculation unit 45 performs this. The confirmation rate F may be determined in consideration of the removal of unnecessary access logs.

The storage unit 47 stores, for example, the delay actual data D, the aggregated value E aggregated by the aggregation unit 43, and the confirmation rate F calculated by the confirmation rate calculation unit 45.

Each component of the log collection server 30 and the stream processing server 40 is realized, for example, by executing a program (software) by a hardware processor such as a CPU (Central Processing Unit). In addition, some or all of these components are hardware (circuits) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), etc. It may be realized by the part; including circuitry), or it may be realized by the cooperation of software and hardware.

The storage unit 47 of the stream processing server 40 is, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), an HDD (Hard Disk Drive), a flash memory, or a hybrid storage device in which a plurality of these is combined. Is realized by. Further, a part or all of the storage unit 47 may be an external device such as NAS or an external storage server that can be accessed by the stream processing server 40.

The log collection server 30 and the stream processing server 40 may be configured by the same device.

[Report providing server]
For example, the report providing server 50 reads from the storage unit 47 of the stream processing server 40 in response to a reference request from a management terminal (not shown) used by the service provider of the service provided by the service providing server 20. The management screen displaying the value E and the confirmation rate F is returned to the management terminal. As a result, the service provider can refer to the management screen including the aggregated value E and the confirmation rate F displayed on the management terminal.

[motion]
Hereinafter, the operation of the system including the stream processing server 40 will be described. FIG. 8 is a sequence diagram showing the operation of the system including the stream processing server 40. First, the terminal device 10 transmits a request for a web page to the service providing server 20 based on the user's operation (S101). Next, the service providing server 20 outputs a log corresponding to the request received from the terminal device 10 to the access log A (S103). Further, the service providing server 20 transmits a web page in response to the request to the terminal device 10 (S105). Next, the terminal device 10 displays the web page received from the service providing server 20 by the browser installed in advance (S107).

The log collection server 30 outputs the access log A and the service operation status report B from the service providing server 20 at a predetermined cycle (for example, every second) asynchronously with the output processing of the access log by the service providing server 20. Collect (S109).

Next, the log collection server 30 sends the stream data A1, the service operation status report B, and the log collection operation status report C, which have not been transmitted to the stream processing server 40, to the stream processing server 40 among the data included in the access log A. Transmit (S111).

Next, the stream processing server 40 aggregates the stream data A1 according to a predetermined aggregation standard (S113). The stream processing server 40 stores the aggregated value E, which is the aggregated result, in the storage unit 47. FIG. 9 is a diagram showing an example of the aggregation result by the stream processing server 40. In the example shown in FIG. 9, the stream processing server 40 aggregates the number of accesses per hour for each content ID as the aggregated value E.

Next, the stream processing server 40 calculates the confirmation rate F of the aggregated value E (S115). For example, the confirmation rate calculation unit 45 predicts the delay status of the log based on the service operation status report B, the log collection operation status report C, and the delay actual data D, and calculates the confirmation rate F.

When calculating the confirmation rate F of the aggregated value with respect to the "ratio" such as the click rate, the stream processing server 40 calculates the confirmation rate F based on the variance of the confirmation degree F for each failure type obtained in advance. You may.

After that, the report providing server 50 reads the aggregated value E and the confirmation rate F from the storage unit 47 of the stream processing server 40 in response to a reference request from the management terminal used by the service provider, for example (S117). Then, the report providing server 50 generates a management screen for displaying the read aggregate value E and the confirmation rate F, and transmits the management screen to the management terminal (S119). As a result, the service provider can refer to the management screen including the aggregated value E and the confirmation rate F displayed on the management terminal.

FIG. 10 is a diagram showing an example of a management screen showing the aggregated value E and the confirmation rate F. In the management screen shown in FIG. 10, bars graphs A to I showing the aggregated values for each hour are shown, and the aggregated values with a lower confirmation rate are displayed so that the color becomes lighter. That is, the aggregated values A to F have the maximum definite value of "1.0", the aggregated value G has the definite value of "0.9", and the aggregated value H has the definite value of "0.7". The aggregated value I is displayed so that the final value is “0.5” and the color of the bar graph becomes lighter as the finalization rate decreases. The service provider can easily confirm the aggregated value E and the confirmation rate F by referring to this management screen. This completes the processing of this sequence diagram.

According to the embodiment described above, the acquisition unit for acquiring the stream data to be aggregated and the first operation status data indicating the operation status of the first device that outputs the stream data, and the acquisition unit acquire the data. An aggregation unit that aggregates the collected stream data, a confirmation rate calculation unit that calculates the confirmation rate of the aggregated value aggregated by the aggregation unit based on the first operation status data acquired by the acquisition unit, and a confirmation rate calculation unit. By providing the above, it is possible to provide the confirmation rate together while maintaining the promptness of the aggregated result. By indicating the confirmation rate (confirmation degree) of the aggregated value, even if the aggregated value is changed after the influence of the delay is eliminated, the service provider can anticipate this change in advance. This makes it possible to prevent confusion among service providers due to changes in aggregated values.

Although the embodiments for carrying out the present invention have been described above using the embodiments, the present invention is not limited to these embodiments, and various modifications and substitutions are made without departing from the gist of the present invention. Can be added.

10 ... Terminal device 20 ... Service providing server 30 ... Log collecting server 40 ... Stream processing server 50 ... Report providing server 31 ... Collecting unit 33 ... Monitoring unit 35 ... Output unit 41 ... Acquisition unit 43 ... Aggregation unit 45 ... Confirmation rate calculation unit 47 ... Memory

Claims (11)

An acquisition unit that acquires stream data to be aggregated and first operation status data indicating the operation status of the first device that outputs the stream data.
An aggregation unit that aggregates the stream data acquired by the acquisition unit, and an aggregation unit.
Based on the first operation status data acquired by the acquisition unit, a confirmation rate calculation unit that calculates the confirmation rate of the aggregated values aggregated by the aggregation unit, and a confirmation rate calculation unit.
Information processing device. The confirmation rate calculation unit calculates the confirmation rate of the aggregated value aggregated by the aggregation unit based on both the first operation status data and the actual data regarding the processing delay of the first apparatus.
The information processing device according to claim 1. The confirmation rate calculation unit estimates the delay status of the stream data output by the first device based on both the first operation status data and the actual data regarding the processing delay of the first device.
The information processing device according to claim 1. The acquisition unit further acquires the second operation status data indicating the operation status of the second device arranged between the first device and the information processing device.
The confirmation rate calculation unit calculates the confirmation rate of the aggregated value based on the first operation status data, the actual data, and the second operation status data.
The information processing device according to any one of claims 2 or 3. The actual data is data in which the fluctuation amount of the confirmation rate is defined for each type of failure of the first device.
The information processing device according to any one of claims 2 to 4. The actual data is data in which the fluctuation amount of the fixed rate is defined for each operating rate of the first device.
The information processing device according to any one of claims 2 to 4. The actual data is data in which the fluctuation amount of the confirmation rate is defined for each elapsed time after the failure of the first device occurs.
The information processing device according to any one of claims 2 to 4. The confirmation rate calculation unit calculates the confirmation rate so that the confirmation rate increases as the elapsed time after the failure of the first device increases.
The information processing device according to claim 7. The confirmation rate calculation unit sets the confirmation rate to the maximum value when the elapsed time after the failure of the first device exceeds a predetermined time.
The information processing device according to claim 8. The computer
Acquire the stream data to be aggregated and the first operation status data indicating the operation status of the first device that outputs the stream data.
The acquired stream data is aggregated and
Based on the acquired first operation status data, the confirmation rate of the aggregated value is calculated.
Information processing method. On the computer
The stream data to be aggregated and the first operating status data indicating the operating status of the first device that outputs the stream data are acquired.
The acquired stream data is aggregated and
Based on the acquired first operation status data, the determination rate of the aggregated value is calculated.
program.

Images