JP2017103621A - Stream data integration method and system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a stream data merging method and system for easily synchronizing and merging detection data and feature amount data.SOLUTION: A fingerprint signal generator 10 emits a fingerprint signal S which may be superposed on detection data Dd, to a target with a sequence pattern and at transmission intervals indicating observation conditions in observing the target T, and registers fingerprint information P indicating a content of the fingerprint signal S in a fingerprint information DB 14. A stream data merging device 15 receives main stream data Dm and sub stream data Ds from a main stream data distribution device 12 and a sub stream data distribution device 13 and, based on the fingerprint information P acquired from the fingerprint information DB14 and corresponding to the main stream data Dm and the sub stream data Ds and the fingerprint signal S extracted from the detection data Dd within the main stream data Dm and the feature amount data within the sub stream data Ds, merges the detection data Dd and the feature amount data while synchronizing them with each other.SELECTED DRAWING: Figure 1

Description

  The present invention provides main stream data obtained by observing detection data composed of video, audio, sensor output, etc. in time series, and substream data such as feature data generated by processing these main stream data. And a stream data synchronization technique for synchronizing these stream data.

  As one of the standards regarding the data compression of multimedia contents and the metadata marking method established by the ISO Moving Picture Experts Group, there is an international standard ISO / IEC 15938: MPEG-7 regarding metadata added to contents. In this MPEG-7, for example, a standard for embedding metadata (additional information) such as what scene is in the hour, minute, and second portion in the video is defined. The metadata may be image feature data calculated from video data, and is used for data collation such as search.

In such a technique, in order to ensure synchronization between content and metadata, the content and metadata are closely linked in the data structure. Therefore, there is a problem that the total amount of content is enlarged when various types of metadata are added to the content.
In general, feature data required as metadata varies depending on the contents of the collation process. If the number of metadata types is increased in order to cope with various collation processes, metadata that is less frequently used is also embedded in video data and transmitted, and network transmission resources are wasted.

MPEG-7, international standard ISO / IEC 15938, The Moving Picture Experts Group, http://mpeg.chiariglione.org/standards/mpeg-7

  In recent years, services that create new value by continuing to flow main stream data obtained by observing detection data consisting of video, audio, sensor output, etc. in time series on the network and processing the data are being studied. ing. For example, if it is possible to detect in real time abnormal behavior of unusual people from images installed in public spaces such as airports and stations, it becomes possible to plan early responses to abnormalities. It is said that it is useful from the viewpoint of crime prevention and disaster prevention.

  Conventionally, information processing such as abnormality detection has been performed inside individual sensor devices and monitoring devices, but in the future, abnormalities will be calculated by calculating feature vectors necessary for abnormality detection and collating with learning examples. Processing such as determination can be executed in any information processing device distributed on the network. That is, the main stream data generated by an information source such as a sensor device or a monitoring device is subjected to secondary processing as needed by any information processing device on the communication network to generate sub-stream data. A distributed data processing method is generally used in which the other information processing apparatus previously integrates the main / sub stream data and processes the information to obtain valuable information.

  In such a distributed data processing environment, it is not expected to adjust time synchronization and clock width fluctuation among a plurality of sensor devices and monitoring devices because work load and cost load increase. Therefore, in information processing equipment on the network, it is difficult to ensure synchronization such as time adjustment and time interval adjustment between main and sub stream data of different information sources, but detection included in these main and sub stream data When performing integrated analysis of data and feature quantity data, it is necessary to strictly match the timing of detection data and feature quantity data to be analyzed, and resynchronization of main and sub stream data becomes an issue.

  The present invention has been made to solve such problems, and it is an object of the present invention to provide a stream data integration technique capable of easily synchronizing and integrating detection data and feature amount data included in main and sub stream data. .

  In order to achieve such an object, the stream data integration method according to the present invention includes an observation step for outputting detection data obtained by observing a target in time series, and an observation condition for observing the target. A fingerprint signal generation step of emitting a fingerprint signal that can be superimposed on the detection data to the target with a sequence pattern and a transmission interval that are indicated, and registering fingerprint information indicating the content of the fingerprint signal in a fingerprint information DB; A main stream data distribution step of generating main stream data from the detection data and distributing it to the communication network, and the fingerprint signal is superimposed based on the detection data included in the main stream data received from the communication network To calculate sub-stream data from these feature data. Based on the substream data distribution step for distributing to the communication network and the input stream data integration instruction, the main stream data and the substream data are received from the communication network and acquired from the fingerprint information DB. Based on the fingerprint information corresponding to the stream data and the sub-stream data, and the detection data in the main stream data and the fingerprint signal extracted from the feature data in the sub-stream data, the detection data and the feature quantity A stream data integration step of integrating data in synchronization with each other.

  Also, in one configuration example of the stream data integration method according to the present invention, the stream data integration step is based on the emission date and time included in the sequence pattern of the fingerprint signal as the observation condition. The detection data located at a position and the feature value data located at a reference position in the substream data are time-synchronized, and the subsequent detection data and the feature value data, based on the transmission interval of the fingerprint signal, The time axis scale is adjusted.

  Further, the stream data integration system according to the present invention includes an observation device that outputs detection data obtained by observing a target in time series, a sequence pattern indicating an observation condition when observing the target, and a transmission interval. A fingerprint signal that can be superimposed on the detection data is emitted to the target, and fingerprint information indicating the contents of the fingerprint signal is registered in a fingerprint information DB, and main stream data is generated from the detection data. Main stream data distribution apparatus for distributing to a communication network, and feature data on which the fingerprint signal is superimposed in time series based on the detection data included in the main stream data received from the communication network Substream data is calculated, substream data is generated from the feature data, and distributed to the communication network. The main stream data and the substream data acquired from the fingerprint information DB by receiving the main stream data and the substream data from the communication network based on the stream data distribution device and the input stream data integration instruction And the detection data in the main stream data and the fingerprint signal extracted from the feature data in the sub-stream data, the detection data and the feature data are synchronized with each other. And a stream data integration device for integration.

According to the present invention, the detection data included in the main stream data and the feature data included in the substream data generated from the detection data are output corresponding to the same time. Therefore, the detection data obtained by observing the same target T and the feature data calculated from the detection data can be integrated and analyzed in synchronization with each other.
For this reason, even in a distributed data processing environment in which observation devices, main stream data distribution devices, and substream data distribution devices are distributed on a communication network, time synchronization and clock width fluctuations among these devices Therefore, the timings of the detection data and the feature amount data to be analyzed can be precisely matched. Therefore, it is possible to perform integrated analysis of detection data and feature amount data with high accuracy.

It is a block diagram which shows the structure of a stream data integration system. It is a structural example of a fingerprint signal. It is explanatory drawing which shows the example of detection data (video). It is explanatory drawing which shows the example of detection data (voice | voice). It is explanatory drawing which shows the example of detection data (vibration). It is explanatory drawing which shows feature-value data (image HOG feature-value). It is explanatory drawing which shows feature-value data (image SIFT feature-value). It is explanatory drawing which shows feature-value data (MFCC feature-value). It is explanatory drawing which shows feature-value data (time-resolved frequency feature-value). It is a flowchart which shows a stream data integration process.

Next, an embodiment of the present invention will be described with reference to the drawings.
[Stream data integration system]
First, a stream data integration system 1 according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing a configuration of a stream data integration system.

  The stream data integration system 1 in FIG. 1 is generated by processing main stream data obtained by observing detection data composed of video, audio, sensor output, etc. in time series, and detection data of these main stream data. And substream data such as feature amount data are integrated in synchronization with each other.

  In recent years, video cameras and various observation sensors are installed in observation devices such as drones that are remotely controlled or autonomous flying objects, monitoring the natural environment and disaster sites, monitoring traffic and human flow in urban areas, etc. Studies on various monitoring activities are in progress. In such monitoring activities, the main stream data recorded and distributed by each observation device and the substream data generated from this main stream data are analyzed in an integrated manner, More advanced and accurate knowledge can be found.

  In order to analyze these main / sub stream data in an integrated manner, it is necessary to observe the detection data consisting of video, audio, sensor output, etc. in time series and analyze the main / sub stream data in synchronization. However, in an actual recording environment, it is difficult to perform timing adjustment such as time adjustment of observation equipment.

  The present invention detects detection data in which a fingerprint signal is superimposed from a target to be monitored / observed by emitting a fingerprint signal (observation identification signal) indicating a specific observation condition such as time / place / device to the target. Main stream data and further substream data are generated, and based on the fingerprint signal superimposed on the main / substream data, the two are synchronized and integrated.

Next, the configuration of the stream data integration system 1 according to the present invention will be described in detail with reference to FIG.
The stream data integration system 1 includes a fingerprint signal generator 10, an observation device 11, a main stream data distribution device 12, a sub stream data distribution device 13, a fingerprint information DB 14, and a stream data integration device 15 as main functional units. Is provided.

  The fingerprint signal generator 10 includes a signal generation source such as a light source, a lighting device, a speaker, and a physical vibration generator as a whole. The fingerprint signal generator 10 emits a fingerprint signal S to a target T and fingerprint information P related to the fingerprint signal S. Is registered in the fingerprint information DB 14 via the communication network NW. The fingerprint signal generator 10 may be shared by a plurality of observation devices 11, may be provided for each observation device 11, and may be mounted in the observation device 11.

  FIG. 2 is a configuration example of a fingerprint signal. Here, a fingerprint signal (observation identification signal) consisting of a sequence pattern (pulse sequence) having a fixed transmission interval is shown. These sequence patterns and transmission intervals are the observation conditions such as the emission date and time, the emission location, and the emission equipment. The various attribute information which shows is shown. The sequence pattern may be transmitted with an interval corresponding to the time synchronization accuracy, and a simple spike signal may be transmitted at a constant transmission interval during the non-transmission period of the sequence pattern.

  Therefore, by comparing the sequence pattern of the fingerprint signal S extracted from the main / sub stream data Dm, Ds with the fingerprint information P registered in the fingerprint information DB 14, the emission date and time, the emission location, the emission device, etc. regarding the fingerprint signal S , That is, the observation conditions specific to the main / sub stream data Dm, Ds can be derived, and the time scale of the main / sub stream data Dm, Ds can be derived from the transmission interval of the sequence pattern.

  The observation device 11 is composed of various observation devices such as a drone, a surveillance camera installed in a public space such as an airport or a station as a whole, and a target on which the fingerprint signal S emitted from the fingerprint signal generator 10 is superimposed. The detection data Dd consisting of video, audio, sensor output, etc. relating to T is observed by the detector 11A and output to the main stream data distribution device 12 in time series. As the detection unit 11A, various detection devices such as a camera, a microphone, and a vibration sensor may be used according to the physical quantity to be observed.

  FIG. 3 is an explanatory diagram showing an example of detection data (video). Here, the fingerprint data S emitted from the fingerprint signal generator 10 by infrared light is included in the detection data Dd. Thereby, the fingerprint signal S can be made difficult to be perceived by humans at the time of observation, and the fingerprint signal S can be superimposed on the detection data Dd without degrading the observation content made of visible light.

  FIG. 4 is an explanatory diagram showing an example of detection data (voice). Here, the fingerprint signal S emitted from the fingerprint signal generator 10 by the low frequency sound lower than the audible frequency is included in the detection data Dd. As a result, the fingerprint signal S can be made difficult to be perceived by humans at the time of observation, and the fingerprint signal S can be superimposed without degrading the observation content made of audible sound.

  FIG. 5 is an explanatory diagram showing an example of detection data (vibration). Here, the detection data Dd includes the fingerprint signal S emitted from the fingerprint signal generator 10 by low frequency vibration. As a result, the fingerprint signal S can be made difficult to be perceived by humans at the time of observation, and the fingerprint signal S can be superimposed without degrading the observation content including the vibration amount.

  The main stream data distribution device 12 includes a control device 12A such as a PC that controls the observation device 11 as a whole. The main stream data distribution device 12 receives the detection data Dd output from the observation device 11, and based on a preset data configuration. It has a function of generating stream data Dm and distributing it to the communication network NW.

  The sub-stream data distribution device 13 is composed of information processing devices 13A, 13B, and 13C such as server devices as a whole, and receives the feature data Dc from the main stream data Dm received from the main stream data distribution device 12 via the communication network NW. It has a function of calculating in time series and a function of generating sub-stream data Ds from the time-series feature amount data Dc based on a preset data configuration and distributing it to the communication network NW.

  The substream data distribution device 13 may be provided separately for each feature amount data Dc to be calculated. In FIG. 1, an information processing device 13A that extracts image HOG (Histograms of Oriented Gradients) feature values, an information processing device 13B that extracts image SIFT (Scale-Invariant Feature Transform) feature values, and an MFCC (Mel-Frequency Cepstrum Coefficients) An information processing apparatus 13C that extracts a feature amount is provided as an example.

  FIG. 6 is an explanatory diagram showing feature data (image HOG feature). The image HOG feature value is a feature value roughly representing the shape of the target object obtained by histogramming the luminance gradient direction in the local region with respect to the target edge included in the image data. It is widely used for human area extraction, is strong against geometric transformations, and is robust against lighting fluctuations.

  FIG. 7 is an explanatory diagram showing feature amount data (image SIFT feature amount). The image SIFT feature value is a feature value representing the feature of the image obtained by extracting feature points from the image by local search in different scale spaces and histogramming the luminance gradient direction at the feature points. It is invariant to rotation and scale changes, and is robust to changes in lighting.

  FIG. 8 is an explanatory diagram showing feature amount data (MFCC feature amount). The MFCC feature value is a feature value indicating the spectral envelope of the speech obtained by emphasizing the high frequency component of the audio signal, then obtaining the mel frequency spectrum based on the mel scale, converting it to a cepstrum, and extracting the low-order component. It is. It is said that a component derived from a sound source and a component derived from the vocal tract can be expressed separately.

FIG. 9 is an explanatory diagram showing feature amount data (time-resolved frequency feature amount). The time-resolved frequency feature amount is obtained by arranging frequency analyzes related to detected data in the order of passage of time, and is a feature amount indicating characteristics of various vibrations including mechanical vibrations.
Note that the process of generating the substream data Ds from the main stream data Dm is not limited to the execution in the substream data distribution apparatus 13, and this substream data generation function is provided in the main stream data distribution apparatus 12. May be executed.

  The fingerprint information DB 14 consists of a database composed of server devices as a whole. The fingerprint information DB 14 receives the fingerprint information P notified from the fingerprint signal generator 10 via the communication network NW and registers it in the database, and the communication network NW. In response to a request from the stream data integration device 15, the designated fingerprint signal generator 10 has a function of returning fingerprint information P corresponding to the date, time, and location to the stream data integration device 15. The contents of the fingerprint information P registered in the database include various attribute information such as the emission date and time of the fingerprint signal S, the emission location, the emission device, and the transmission interval.

The stream data integration device 15 is composed of an information processing device such as a server device as a whole, and the main stream data Dm and the sub stream data Ds received from the main stream data distribution device 12 and the sub stream data distribution device 13 via the communication network NW. Have the function of synchronizing and integrating each other.
The stream data integration device 15 includes a data acquisition unit 15A, a fingerprint signal extraction unit 15B, and a data processing unit 15C as main processing units.

  The data acquisition unit 15A receives the main stream data Dm and the sub stream data Ds from the main stream data distribution device 12 and the sub stream data distribution device 13 via the communication network NW based on the input stream data integration instruction. And a function of acquiring the fingerprint information P corresponding to the main stream data Dm and the sub stream data Ds from the fingerprint information DB 14.

  The fingerprint signal extraction unit 15B extracts a fingerprint signal S (sequence pattern) superimposed on the detection data Dd in the main stream data Dm received by the data acquisition unit 15A, and feature amount data in the substream data Ds It has a function of extracting a fingerprint signal S (sequence pattern) superimposed on Dc.

  The data processing unit 15C receives the data acquisition unit 15A based on the fingerprint signal S of the main stream data Dm and the substream data Ds extracted by the fingerprint signal extraction unit 15B and the fingerprint information P acquired by the data acquisition unit 15A. The detection data Dd included in the main stream data Dm and the feature data Dc included in the substream data Ds are synchronized with each other, integrated, and output.

  At this time, specifically, based on the emission date and time included in the fingerprint information P, the detection data Dd located at the reference position in the main stream data Dm and the feature amount data Dc located at the reference position in the substream data Ds Are time synchronized. In this case, the reference position may be the time when the fingerprint signal S indicating the emission date and time is extracted from the detection data Dd and the feature data Dc.

In addition, based on the transmission interval of the fingerprint signal S included in the main stream data Dm and the sub stream data Ds, the subsequent detection data Dd included in the main stream data Dm and the subsequent feature data included in the sub stream data Ds. The time axis scale with Dc is adjusted. This time axis scale is generated due to a difference in data size between the detection data Dd and the feature data Dc.
Further, based on the emission location and the emission device included in the fingerprint information P, the target T and the observation conditions regarding the detection data Dd and the feature data Dc whose time synchronization and time axis scale are adjusted are specified.

[Operation of this embodiment]
Next, the stream data integration operation of the stream data integration system 1 according to the present embodiment will be described with reference to FIG. FIG. 10 is a flowchart showing stream data integration processing.

  First, at the start of observation, a fingerprint signal S is emitted from the fingerprint signal generator 10 to the target T (step 100), and fingerprint information P related to the fingerprint signal S is registered from the fingerprint signal generator 10 in the fingerprint information DB14. (Step 101).

  The observation device 11 detects detection data Dd including video, audio, sensor output, and the like from the target T by the detection unit 11A such as a camera, a microphone, and a vibration sensor, and outputs the detection data Dd to the main stream data distribution device 12 (step 102). ). As a result, the detection data Dd on which the fingerprint signal S is superimposed is output to the main stream data distribution device 12.

  Subsequently, the main stream data distribution device 12 receives the detection data Dd output from the observation device 11, generates the main stream data Dm based on a preset data configuration, and distributes it to the communication network NW ( Step 103). Thereby, the detection data Dd on which the fingerprint signal S is superimposed is distributed as the main stream data Dm.

  Thereafter, the sub-stream data distribution device 13 calculates the feature data Dc in time series from the detection data Dd of the main stream data Dm received from the main stream data distribution device 12 via the communication network NW (step 104). Then, the substream data Ds is generated from the time-series feature data Dc based on a preset data structure and distributed to the communication network NW (step 105). As a result, the fingerprint signal S superimposed on the detection data Dd is also superimposed on the feature amount data Dc and distributed as substream data Ds.

  Next, the stream data integration device 15 receives the main stream data Dm from the main stream data distribution device 12 and the sub stream data distribution device 13 via the communication network NW based on the integration instruction input by the data acquisition unit 15A. And the substream data Ds are received (step 106), and the fingerprint information P corresponding to the main stream data Dm and the substream data Ds is acquired from the fingerprint information DB 14 (step 107). A fingerprint signal S (sequence pattern) superimposed on the detection data Dd in the main stream data Dm and the feature data Dc in the sub stream data Ds is extracted (step 108).

  Thereafter, the stream data integration device 15 causes the data processing unit 15C to send the detection data Dd in the main stream data Dm and the feature data Dc in the sub stream data Ds to the time based on the emission date and time included in the fingerprint information P. At the same time, the time scale of the detection data Dd and the feature data Dc is adjusted based on the transmission interval of the fingerprint signal S (step 109). Further, the data processing unit 15C specifies the observation conditions regarding the detection data Dd and the feature amount data Dc based on the emission location and the emission device included in the fingerprint information P, and the series of stream data integration operation is finished.

  As a result, the detection data Dd included in the main stream data Dm and the feature data Dc included in the substream data Ds generated from the detection data Dd are output corresponding to the same time timing. Even when there are fluctuations due to individual differences in the clock generation elements mounted on the observation device 11, the main stream data distribution device 12, and the sub stream data distribution device 13, the time lag is corrected. .

[Effects of the present embodiment]
As described above, according to the present embodiment, the fingerprint signal generator 10 applies the fingerprint signal S that can be superimposed on the detection data Dd to the target T with the sequence pattern and the transmission interval indicating the observation conditions when observing the target T. In addition, the fingerprint information P indicating the content of the fingerprint signal S is registered in the fingerprint information DB 14, and the stream data integration 15 receives the main stream data Dm and the substream data from the main stream data distribution device 12 and the substream data distribution device 13. Fingerprint information P received from the stream data Ds and corresponding to the main stream data Dm and the substream data Ds acquired from the fingerprint information DB 14, and the detection data Dd in the main stream data Dm and the feature data Dc in the substream data Ds Based on the fingerprint signal S extracted from the detection data Dd and the feature amount data It is obtained so as to integrate each other to synchronize the data Dc.

  As a result, the detection data Dd included in the main stream data Dm and the feature data Dc included in the substream data Ds generated from the detection data Dd are output corresponding to each same time. . Therefore, the detection data Dd obtained by observing the same target T and the feature data Dc calculated from the detection data Dd can be integrated and analyzed in synchronization with each other.

  For this reason, even in a distributed data processing environment in which the observation device 11, the main stream data distribution device 12, and the sub stream data distribution device 13 are distributed on the communication network NW, time synchronization and clock width between these devices Therefore, the time timing of the detection data Dd and the feature data Dc to be analyzed can be precisely matched without requiring adjustment regarding fluctuation of the image data. Therefore, it is possible to perform integrated analysis of Dd and feature amount data Dc with high accuracy.

[Extended embodiment]
The present invention has been described above with reference to the embodiments, but the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In addition, each embodiment can be implemented in any combination within a consistent range.

FIG. 1 shows an example in which one fingerprint signal S is emitted for one target T, but the correspondence between the target T and the fingerprint signal S is limited to one-to-one. It may be one-to-many, many-to-one, or many-to-many.
Moreover, although the case where one target T is observed with a plurality of observation devices 11 is shown as an example, the correspondence relationship between the target T and the observation devices 11 is not limited to one-to-many, but one-to-one. , Many to one, or many to many.

  DESCRIPTION OF SYMBOLS 1 ... Stream data integrated system, 10 ... Fingerprint signal generator, 11 ... Observation apparatus, 11A ... Detection part, 12 ... Main stream data delivery apparatus, 13 ... Sub stream data delivery apparatus, 14 ... Fingerprint information DB, 15 ... Stream data Integrated device, 15A ... data acquisition unit, 15B ... fingerprint signal extraction unit, 15C ... data processing unit, NW ... communication network, T ... target, S ... fingerprint signal, P ... fingerprint information, Dd ... detection data, Dm ... main stream Data, Dc ... feature data, Ds ... substream data.

Claims (3)

An observation step of outputting the detection data obtained by observing the object in time series;
A fingerprint signal that can be superimposed on the detection data is emitted to the target with a sequence pattern and a transmission interval indicating an observation condition when observing the target, and fingerprint information indicating the content of the fingerprint signal is output to the fingerprint information DB. A fingerprint signal generation step to be registered in
A main stream data distribution step of generating main stream data from the detection data and distributing the data to a communication network;
Based on the detection data included in the main stream data received from the communication network, the feature amount data on which the fingerprint signal is superimposed is calculated in time series, and substream data is generated from the feature amount data. Substream data delivery step for delivering to the communication network;
Based on the input stream data integration instruction, the main stream data and the sub stream data are received from the communication network, and the main stream data and the fingerprint information corresponding to the sub stream data acquired from the fingerprint information DB; Stream data integration step of integrating the detection data and the feature data in synchronization with each other based on the detection data in the main stream data and the fingerprint signal extracted from the feature data in the sub-stream data And a stream data integration method comprising: The stream data integration method according to claim 1,
In the stream data integration step, the detection data located at the reference position in the main stream data and the reference position in the substream data based on the emission date and time included in the sequence pattern of the fingerprint signal as the observation condition A stream data integration method characterized in that time synchronization is performed with the feature quantity data that is located, and a time axis scale between subsequent detection data and feature quantity data is adjusted based on a transmission interval of the fingerprint signal. Observation equipment that outputs the detection data obtained by observing the target in time series,
A fingerprint signal that can be superimposed on the detection data is emitted to the target with a sequence pattern and a transmission interval indicating an observation condition when observing the target, and fingerprint information indicating the content of the fingerprint signal is output to the fingerprint information DB. A fingerprint signal generator to register with
A main stream data distribution device for generating main stream data from the detection data and distributing the data to a communication network;
Based on the detection data included in the main stream data received from the communication network, the feature amount data on which the fingerprint signal is superimposed is calculated in time series, and substream data is generated from the feature amount data. A secondary stream data delivery device for delivering to the communication network;
Based on the input stream data integration instruction, the main stream data and the sub stream data are received from the communication network, and the main stream data and the fingerprint information corresponding to the sub stream data acquired from the fingerprint information DB; A stream data integration device that integrates the detection data and the feature data in synchronization with each other based on the detection data in the main stream data and the fingerprint signal extracted from the feature data in the sub-stream data And a stream data integration system comprising: