CN103235287B - Sound source localization camera shooting tracking device - Google Patents

Abstract

A sound source positioning camera tracking device, which includes a pan-tilt carrying a camera, a single-chip microcomputer and four pickups, wherein the first pickup, the second pickup and the third pickup are arranged in sequence along a horizontal straight line, and the fourth pickup is located on the side of the second pickup. Directly above, the output ends of the four pickups are respectively connected to the four input ports of the single-chip microcomputer through four filters, and the horizontal angle motor and the elevation angle motor of the cloud platform are connected to the output ports of the single-chip microcomputer. According to the sound signals picked up by the four pickups, the present invention uses a positioning method based on time delay estimation to determine the position of the monitoring target, which not only realizes the dynamic tracking of the specific target, but also has the advantages of small amount of calculation, easy implementation, and high positioning accuracy. The invention can realize multi-angle coverage monitoring by using a single camera, which not only improves the monitoring ability, but also saves the monitoring cost.

Description

A sound source localization camera tracking device

technical field

The invention relates to a sound source positioning camera tracking device based on a generalized cross-correlation algorithm, which can enable a camera to automatically track a monitoring target, and belongs to the field of measurement technology.

Background technique

At present, domestic and foreign video monitoring devices can be roughly divided into two types, one is a static monitoring device, and the other is a dynamic monitoring device. The camera of the static monitoring device is fixed, and can only shoot a certain scene, and cannot realize the dynamic tracking of the monitoring target, and its monitoring range is small. There are many kinds of dynamic monitoring devices, and some use the method of rotating the camera at a uniform speed to realize multi-angle monitoring. This kind of monitoring device cannot realize the dynamic tracking of the monitoring target, and the monitoring effect is not ideal; although some monitoring devices can track a specific target, but Mainly based on image processing to capture dynamic targets, its disadvantages are that it is difficult to find dynamic targets and the cost of image data processing equipment is high.

The Chinese patent application number is 201010204772.9, and the name is "a method of sound source localization". Acoustic signal, denoising processing to obtain a clean observation signal, estimate the separation matrix of the measured sound source, obtain the frequency response matrix, adopt the overall direction of arrival estimation strategy based on peak detection, and obtain the arrival of all sound source signals at one time. Accurate estimation of the direction and the use of relevant knowledge of spatial geometry to calculate the spatial angle and finally realize the spatial positioning of the sound source signal. It overcomes the problems that the existing sound source-location method cannot effectively realize the three-dimensional space positioning of multiple aliasing sound sources, and there are problems such as inflexible use of positioning methods, unstable estimation results, low precision, and insufficient practicability, etc., but it is applied to Video surveillance devices are still inconvenient.

In short, the monitoring range of traditional video surveillance devices is very limited, and multi-angle coverage monitoring cannot be realized. To realize all-round monitoring, the number of monitoring equipment must be increased, which increases the cost of monitoring equipment.

Contents of the invention

The object of the present invention is to provide a sound source localization camera tracking device for the disadvantages of the prior art, which can reduce the manufacturing cost of the monitoring equipment as much as possible while realizing multi-angle coverage monitoring.

Problem described in the present invention is realized with following technical scheme:

A camera tracking device for sound source localization, comprising a camera-carrying platform, a single-chip microcomputer, and four pickups, wherein the first pickup, the second pickup, and the third pickup are arranged in sequence along a horizontal line, and the fourth pickup is positioned at the second pickup. Directly above, the output ends of the four pickups are respectively connected to the four input ports of the single-chip microcomputer through four filters, and the horizontal angle motor and the elevation angle motor of the pan/tilt are connected to the output ports of the single-chip microcomputer. The sound source positioning camera tracking device is as follows How it works:

A. Establish a three-dimensional rectangular coordinate system with the second pickup as the origin, the third pickup is on the positive semi-axis of the X axis, the fourth pickup is on the positive semi-axis of the Y axis, and the distance between the second pickup and the remaining three pickups is a , the camera pan/tilt is placed at the origin;

B. Use four pickups to collect the sound signal from the tracked object in real time, and use the time delay estimation algorithm to obtain the distance difference d12 from the sound source to the first pickup and the second pickup, and the distance from the sound source to the first pickup and the third pickup The distance difference d13, the distance difference d32 from the sound source to the third pickup and the second pickup, the distance d42 from the sound source to the fourth pickup and the second pickup;

C. The horizontal angle θ of the sound source relative to the origin and the X-axis is obtained according to the following formula:

θ=acos(d13/2a),

The elevation angle φ of the sound source relative to the origin and the Y axis is obtained by the following formula:

φ=acos(d24/a),

D. The single-chip microcomputer drives the pan-tilt's horizontal angle motor and elevation angle motor to run, and adjusts the horizontal angle of the camera to θ and the elevation angle to φ, so as to realize the dynamic tracking of the monitoring target.

The specific steps for obtaining the distance difference between the sound source and the two pickups by using the delay estimation algorithm in the above-mentioned sound source localization camera tracking device are as follows:

① Find the cross-correlation function between two signals

Set the output signal of the two pickups After Fourier transform, they are , then the cross power spectrum between the two signals for:

,

in To take the conjugate;

Taking Fourier transform on both sides of the above formula, we have:

,

After weighting the signal in the frequency domain, by ,Will Inverse transform to time domain to get generalized cross-correlation function between two signals :

,

in, is the weighting function;

② According to the cross-correlation function between the two signals Find the distance difference between the sound source and the two pickups

Find the cross-correlation function The maximum value of , to get the time delay between the two sets of signals, multiply the obtained time delay by the propagation speed of the sound, you can get the distance difference between the sound source and the two pickups.

The above-mentioned sound source localization camera tracking device, while using the sound source localization camera to track the target, also needs to improve the accuracy of the sound source location through the live body recognition technology of the camera. The specific steps are as follows:

1. Use B*0.11 + G*0.59 + R*0.3 to calculate the gray value, and supply the obtained gray value to the pixel with a smaller gray value in the image;

Among them, R represents red, G represents green, and B represents blue;

0.11, 0.59, and 0.3 are the weights of the three primary colors of blue, green, and red respectively;

2,

, , Rv1, Bv1, Gv1: respectively the three primary colors (red, blue, green) components of the source image pixels; Rv2, Bv2, Gv2: respectively the three primary colors (red, blue, green) components of the target image pixels; Rvd , Bvd, and Gvd take the average value, and then compare it with the allowable maximum difference value. If it is greater than the source image pixel, the target image pixel is updated, otherwise it is not updated, and finally a difference map is formed.

The above-mentioned sound source localization camera tracking device, the weighting function is: .

In the above-mentioned sound source localization camera tracking device, the sound pickup adopts a microphone.

According to the sound signals picked up by the four pickups, the present invention uses a positioning method based on time delay estimation to determine the position of the monitoring target, which not only realizes the dynamic tracking of the specific target, but also has the advantages of small amount of calculation, easy implementation, and high positioning accuracy. The invention can realize multi-angle coverage monitoring by using a single camera, which not only improves the monitoring ability, but also saves the monitoring cost.

The device can automatically find and observe the place where the sound is emitted, making the security more secure. In a video conference or a large meeting place, this device can realize intelligent video collection, reflecting the humanization of the camera function.

Description of drawings

The present invention will be further described below in conjunction with accompanying drawing.

Figure 1 is an electrical schematic diagram of the camera servo servo system;

Fig. 2 is the installation position schematic diagram of four pickups;

Figure 3 is a block diagram of an image recognition algorithm for illumination compensation differences.

The list of labels in the figure is: MIC1～MIC 4, the first pickup to the fourth pickup, U1～U4, the first filter to the fourth filter, U5, single-chip microcomputer, M1, horizontal angle motor, M2, elevation angle motor.

The list of symbols in the text is: d12, the distance difference from the sound source to the first pickup and the second pickup, d13, the distance difference from the sound source to the first pickup and the third pickup, d32, the distance difference from the sound source to the third pickup and the second pickup distance difference, d42, the distance difference from the sound source to the fourth pickup and the second pickup, r, the distance from the sound source to the origin, θ, the horizontal angle of the sound source to the origin and the X axis, φ, the sound source to the origin and Y The elevation angle of the axis, a, the distance between the second pickup and the remaining three pickups, , the output signals of the two pickups, , The signal after Fourier transform, , the cross-power spectrum between the two signals, , the generalized cross-correlation function between the two signals, , Weighting function.

Detailed ways

This system uses a pickup (microphone) array to pick up voice signals, and judges the sound direction according to the sound frequency domain and time domain, and locates the sound source. After that, the camera automatically turns to the direction of the sound to monitor the position of the sound source, and then the accuracy of the sound source position is improved through the living body recognition technology of the camera. It realizes the all-round monitoring of a single device, improves the monitoring ability and saves the cost.

In the present invention, the model used for the camera cloud platform is Putianshi PTS-303; wherein, the horizontal angle motor M1 that drives the camera to rotate up and down on the single-chip microcomputer PB1 and PB2 ends controls the cloud platform; Elevation motor M2.

The layout of the microphone is shown in Figure 2. The second pickup MIC2 is at the origin, and the distance between it and other pickups is a, and the distance difference between the sound source and each pickup pair is d12, d13, d32, d42 (distance difference Equal to the time delay multiplied by the speed of sound); establish the coordinate system shown in the figure, and according to the geometric approximation, the horizontal angle θ of the sound source relative to the origin and the X axis can be obtained as:

θ=acos(d13/2a);

The elevation angle φ of the sound source relative to the origin and m2-m4 is:

φ=acos(d24/a);

Let the coordinates of the sound source be (x, y, z), and the distance r of the sound source relative to the origin is deduced from the following formula:

sqrt((x+a)*(x+a)+(y*y)+(z*z))-sqrt((x*x)+(y*y)+(z*z))=d12;

sqrt((x-a)*(x+a)+(y*y)+(z*z))-sqrt((x*x)+(y*y)+(z*z))=d32;

r=sqrt((x*x)+(y*y)+(z*z));

sqrt() is the square root.

Arranging and simplifying can be obtained: r=((a*a)-((d12*d12)+(d32*d32)))/(d12+d32);

In the actual positioning, just put the camera at the origin, first turn to the horizontal angle θ, and then turn to the elevation angle φ.

In terms of sound source localization, considering the real-time performance of the system, this system adopts the sound source localization technology based on time delay estimation. Since the sound source signal is a non-stationary signal, which is easily disturbed by noise and reverberation, but has short-term stability, it can be regarded as a periodic signal within 30ms, so the FFT algorithm can be used for frequency domain analysis. In the delay estimation algorithm, the generalized cross-correlation function (GCC) method with certain anti-noise and reverberation ability is adopted. This method obtains the cross-correlation function between two groups of signals by calculating the cross-power spectrum between the two signals, then giving different weighting operations in the frequency domain, and finally inversely transforming it to the time domain. Finding the maximum value of the cross-correlation function is the time delay between the two groups of signals. The distance difference between the sound source and the two microphones is calculated according to the time delay, and the position of the sound source can be determined by using the distance difference to the four points. We use four pickups to collect sound signals to determine the direction of the sound source.

The basic principle of the generalized cross-correlation (GCC) method to obtain the time delay indirectly is: to obtain the cross-power spectrum between the two signals, and then give different weighting operations in the frequency domain, and finally inversely transform to the time domain to obtain the two Cross-correlation function between groups of signals. Finding the maximum value of the cross-correlation function is the time delay between the two groups of signals.

two sets of signals After the Fourier transform of , the cross power spectrum between the two signals as follows, where For conjugation:

,

Since the cross-correlation function of the two signals and the cross-power spectrum between the signals are Fourier transform pairs, taking Fourier transform on both sides of the above formula, we have:

,

Depend on ,Will Inverse transformation to the time domain can be obtained:

,

In practice, due to the presence of reverberation, contains the cross-power spectrum component of the reflected sound, which leads to the peak shift of the cross-correlation function; in addition, in the case of low signal-to-noise ratio, the peak of the mutual light function will be weakened, resulting in difficulty in peak detection. In order to suppress the influence of noise and reverberation, the generalized cross-correlation function weights the signal in the frequency domain, so as to whiten the signal and noise, thereby enhancing the frequency components with high signal-to-noise ratio in the signal, suppressing the influence of noise, and finally Inversely transformed to the time domain, the generalized cross-correlation function between the two signals is obtained, namely:

,

is the weighting function. We have tested when = works best. So the weighting function we use is .

live recognition

Effect of illumination compensation difference image recognition algorithm:

Select two successively adjacent images to perform difference algorithm operations to form a difference map. Find the point that is really different from the background in the difference map, update it to the cumulative difference map, and accumulate the remaining points. And remove the difference part that has been removed in the cumulative difference graph. Judging whether there is a target object moving according to the size of the difference part.

Algorithm implementation:

1. Accumulate the number of non-difference points in the difference map. If the number of times to update the background is reached, use the points in the old map to update the background. If it is a new point and has been stationary for 2 times, set the value of the corresponding point on the cumulative difference graph to a higher value.

2. Use the "illumination compensation difference algorithm" to find the difference between the difference points in the difference map between the new map and the background map, and choose according to the difference:

2.1. If there is no difference, refresh the corresponding pixels of the "accumulated number of stationary times" and "accumulated difference".

2.2 If there is a difference, refresh the corresponding pixel of the "cumulative image of static times" and set the corresponding pixel of the cumulative image of the difference to a higher value. The number of pixels with differences accumulated in the corresponding records.

2.3. Find the pixel that was a new difference point in the "difference cumulative map", use the "light compensation difference algorithm" and the corresponding background pixel to find the difference in the corresponding pixel of the new map, if there is a difference, refresh the "still times cumulative map" and The "difference cumulative map" corresponds to pixels.

2.4. Estimate the movement of the target object based on the number of difference pixels calculated above. Select the fixed pixel matrix surface to calculate the percentage of different pixels in the matrix surface. When it is greater than the preset value, it will be judged that there is a target object moving, and the center of the pixel matrix surface is positioned to the concentrated area of different pixel points to achieve the effect of tracking and positioning.

In terms of the camera, we use the living body recognition technology of the camera on the basis of collecting images, and use the "light compensation difference algorithm". The high-resolution image data is first subjected to illumination compensation, and then to the difference algorithm. Light compensation method: use B*0.11 + G*0.59 + R*0.3 to calculate the gray value, and supply the obtained gray value to the pixel with a smaller gray value. Difference Algorithm: v take r, g, b respectively (subscripts 1 and 2 represent the source pixel and target pixel respectively), and finally take the average value of the three vd, and then return the comparison result with the allowable maximum difference value. Specialization and optimization in algorithm implementation: No compensation will be performed when the gray level difference is greater than a certain value. For optimization, the "illumination compensation method" and "difference algorithm" are fused together, instead of finding the difference after supplementing. Finally, the live dynamic recognition effect on the image is realized.

In terms of data transmission, 3G wireless network is used to transmit image data to the server. It realizes more automation and intelligentization of video monitoring and image acquisition.

The invention realizes automatic monitoring of a specific target, can reduce the work intensity of monitoring personnel, improve monitoring efficiency and save manpower. In the on-site monitoring, the specific location can be observed according to the location of the sound, making the security more secure. In video conferences or large meeting places, this function can be used to realize more intelligent video collection, reflecting the humanization of the camera function.

Claims (4)

1. A sound source localization camera tracking method is characterized in that it utilizes a sound source localization camera tracking device, and said sound source localization camera tracking device comprises a pan platform carrying a video camera, a single-chip microcomputer (U5) and four pickups, wherein , the first pickup (MIC1), the second pickup (MIC2) and the third pickup (MIC3) are arranged in sequence along a horizontal line, the fourth pickup (MIC4) is located directly above the second pickup (MIC2), and the output terminals of the four pickups The four input ports of the single-chip microcomputer (U5) are respectively connected through four filters, the horizontal angle motor (M1) and the elevation angle motor (M2) of the pan/tilt are connected to the output port of the single-chip microcomputer (U5), and the sound source positioning camera tracking The device works as follows: A. Establish a three-dimensional rectangular coordinate system with the second pickup (MIC2) as the origin, the third pickup (MIC3) is on the positive semi-axis of the X axis, the fourth pickup (MIC4) is on the positive semi-axis of the Y axis, and the second pickup (MIC2) The distance from the other three pickups is a, and the camera pan/tilt is placed at the origin; B. Use four pickups to collect the sound signal from the tracked object in real time, and use the delay estimation algorithm to obtain the distance difference d12 from the sound source to the first pickup (MIC1) and the second pickup (MIC2), and the distance difference from the sound source to the first pickup The distance difference d13 between the pickup (MIC1) and the third pickup (MIC3), the distance difference d32 between the sound source and the third pickup (MIC3) and the second pickup (MIC2), the distance difference between the sound source and the fourth pickup (MIC4) and the second pickup (MIC2) distance difference d42; C. The horizontal angle θ of the sound source relative to the origin and the X-axis is obtained according to the following formula: θ=acos(d13/2a), The elevation angle φ of the sound source relative to the origin and the Y axis is obtained by the following formula: φ=acos(d24/a), D. The single-chip microcomputer (U5) drives the pan-tilt’s horizontal angle motor (M1) and elevation angle motor (M2) to run, and adjusts the camera’s horizontal angle to θ, and the elevation angle to φ, so as to realize the dynamic tracking of the monitoring target; The specific steps to obtain the distance difference between the sound source and the two pickups using the time delay estimation algorithm are as follows: ① Find the cross-correlation function between two signals Set the output signal of the two pickups After Fourier transform, they are , then the cross power spectrum between the two signals for: , in To take the conjugate; Taking Fourier transform on both sides of the above formula, we have: , After weighting the signal in the frequency domain, by ,Will Inverse transform to time domain to get generalized cross-correlation function between two signals : , , in, is the weighting function; ② According to the cross-correlation function between the two signals Find the distance difference between the sound source and the two pickups Find the cross-correlation function The maximum value of , to get the time delay between the two sets of signals, multiply the obtained time delay by the propagation speed of the sound, you can get the distance difference between the sound source and the two pickups. 2. a kind of sound source localization camera tracking method according to claim 1, it is characterized in that, when adopting sound source localization camera to track target, also will improve the accuracy of sound source position by the living body recognition technology of camera, its Specific steps are as follows: Use B*0.11 + G*0.59 + R*0.3 to calculate the grayscale value, and supply the obtained grayscale value to the pixels with smaller grayscale in the image; Among them, R represents red, G represents green, and B represents blue; 0.11, 0.59, and 0.3 are the weights of the three primary colors of blue, green, and red respectively; , , , Rv1, Bv1, Gv1: respectively the three primary colors (red, blue, green) components of the source image pixels; Rv2, Bv2, Gv2: respectively the three primary colors (red, blue, green) components of the target image pixels; Rvd , Bvd, and Gvd take the average value, and then compare it with the allowable maximum difference value. If it is greater than the source image pixel, the target image pixel is updated, otherwise it is not updated, and finally a difference map is formed. 3. A kind of sound source location camera tracking method according to claim 2, is characterized in that, described weighting function is: . 4. A sound source localization camera tracking method according to claim 3, wherein the sound pickup adopts a microphone.