CN114155462A - Method, device and fisheye camera for acquiring passenger flow status - Google Patents

Abstract

The embodiments of the present application provide a method, a device and a fisheye camera for acquiring a passenger flow state, which specifically include: collecting video information in public places through the fisheye camera; determining a multi-frame image for inputting a deep neural network model according to the video information ; Perform heat map prediction analysis on multi-frame images through deep learning neural network to determine local thermal values; determine the current passenger flow state of public places according to local thermal values. Through the solution of the present disclosure, the fisheye camera is used to collect video information, and the video information is predicted and analyzed by the deep neural network model integrated with the fisheye camera, so as to obtain the local thermal value of the public place, and further determine the current value based on the local thermal value. Passenger flow status, improve camera coverage, reduce costs, automatically report passenger flow congestion status, and improve the efficiency of early warning processing of emergencies.

Description

Method, device and fisheye camera for acquiring passenger flow status

technical field

The present application relates to the technical field of computer vision, and in particular, to a method and device for acquiring a passenger flow state, and a fisheye camera.

Background technique

With the increasing number of public activities, there are more and more public activities that need to be participated in public places such as rail transit, hospitals, shopping malls, and airports. The general passenger flow in public places changes. In some periods, the local passenger flow increases, which is easy to cause congestion. In severe cases, it may lead to stampede incidents. Therefore, maintaining appropriate social distance, waiting distance, and queuing distance are basic needs. Therefore, in public places, real-time monitoring of the crowdedness of people in the scene is of great significance to the risk of environmental congestion and maintaining social order.

Most of the existing public places are equipped with guns, domes or dome cameras as monitoring equipment to manage and monitor public areas. The aforementioned types of cameras are divided into long focal length, short focal length or zoom cameras, but the field of view is often smaller than 140°, the coverage is not comprehensive enough. In addition, the analysis of the degree of crowding mostly relies on manual judgment, and the flow of crowded people is dredged by the diverter. In the prior art, the coverage of the camera is insufficient. If full coverage is required, additional installation sites are required, which increases the cost; manual inspection and dredging increase the operating cost.

SUMMARY OF THE INVENTION

In view of this, the embodiments of the present disclosure provide a method, a device, and a fisheye camera for acquiring a passenger flow state, so as to solve the problems existing in the prior art.

In a first aspect, an embodiment of the present disclosure provides a method for acquiring a passenger flow state, which is applied to a fisheye camera, including:

Collect video information in public places through fisheye cameras;

Determine a multi-frame image for inputting the deep neural network model according to the video information;

Perform heat map prediction analysis on multiple frames of the images through the deep learning neural network to determine local heat values;

The current passenger flow state of the public place is determined according to the local thermal value.

Optionally, the determining according to the video information is used to input the multi-frame image of the deep neural network model, including:

Obtain the passenger flow status of the public place at the last moment;

Determine the image capture frame rate according to the passenger flow state at the last moment;

Corresponding multi-frame images are determined from the video information according to the image acquisition frame rate.

Optionally, the method further includes:

Determine whether the current passenger flow state is in a crowded state;

If so, acquire the current image of the public place, and send the current image and the congestion warning information to the server.

Optionally, the method further includes:

Performing human body part labeling on the original image data collected by the fisheye camera to obtain the first training data;

Post the image of the person on the base map of the unmanned scene, and obtain the second training data;

The initial deep learning neural network is trained according to the first training data and the second training data to obtain the deep learning neural network.

In a second aspect, an embodiment of the present disclosure provides a device for acquiring a passenger flow state, which is applied to a fisheye camera, including:

The acquisition module is used to collect video information in public places through the fisheye camera;

a first determining module, configured to determine, according to the video information, a multi-frame image for inputting a deep neural network model;

an analysis module, configured to perform thermal map prediction analysis on multiple frames of the images through the deep learning neural network to determine a local thermal value;

The second determining module is configured to determine the current passenger flow state of the public place according to the local thermal value.

Optionally, the first determining module includes:

Get the sub-module, which is used to obtain the passenger flow status of the public place at the last moment;

a first determination submodule, configured to determine an image capture frame rate according to the passenger flow state at the last moment;

The second determination submodule is configured to determine corresponding multi-frame images from the video information according to the image acquisition frame rate.

Optionally, the device further includes:

a judging module for judging whether the current passenger flow state is in a crowded state;

If so, acquire the current image of the public place, and send the current image and the congestion warning information to the server.

Optionally, the device further includes:

The training module is used for labeling the human body parts on the original image data collected by the fisheye camera to obtain the first training data;

Post the image of the person on the base map of the unmanned scene, and obtain the second training data;

The initial deep learning neural network is trained according to the first training data and the second training data to obtain the deep learning neural network.

In a third aspect, an embodiment of the present disclosure further provides a fisheye camera, where the fisheye camera includes:

at least one processor; and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the passenger flow of the aforementioned first aspect or any implementation of the first aspect How to get the status.

In a fourth aspect, embodiments of the present disclosure further provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and the computer program is used to cause the computer to execute the foregoing first aspect or any implementation of the first aspect The acquisition method of the passenger flow status in the method.

In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, the computer program product includes a computer program stored on a computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by a computer, The computer is made to execute the method for acquiring the passenger flow state in the first aspect or any implementation manner of the first aspect.

The method, device and fisheye camera for acquiring a passenger flow state in the embodiments of the present disclosure collect video information in public places through the fisheye camera; determine a multi-frame image for inputting a deep neural network model according to the video information; The deep learning neural network performs heat map prediction and analysis on the multiple frames of the images to determine local thermal values; the current passenger flow state of the public place is determined according to the local thermal values. Through the solution of the present disclosure, the fisheye camera is used to collect video information, and the video information is predicted and analyzed by the deep neural network model integrated with the fisheye camera, so as to obtain the local thermal value of the public place, and further determine the current value based on the local thermal value. Passenger flow status, improve camera coverage, reduce costs, automatically report passenger flow congestion status, and improve the efficiency of early warning processing of emergencies.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

1 is a schematic flowchart of a method for acquiring a passenger flow state according to an embodiment of the present invention;

FIG. 2 is a schematic flowchart of step S102 in the method for obtaining a passenger flow state according to an embodiment of the present invention;

3 is a schematic structural diagram of an apparatus for acquiring a passenger flow state provided by an embodiment of the present invention;

4 is another schematic structural diagram of an apparatus for acquiring a passenger flow state provided by an embodiment of the present invention;

5 is another schematic structural diagram of an apparatus for acquiring a passenger flow state provided by an embodiment of the present invention;

FIG. 6 is another schematic structural diagram of an apparatus for acquiring a passenger flow state according to an embodiment of the present invention.

Detailed ways

The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

The embodiments of the present application are described below through specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. The present application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments may be combined with each other under the condition of no conflict. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

To illustrate, various aspects of embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms and that any specific structure and/or function described herein is illustrative only. Based on this application, one skilled in the art should appreciate that an aspect described herein may be implemented independently of any other aspects and that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented and/or a method may be practiced using any number of the aspects set forth herein. Additionally, such an apparatus may be implemented and/or such a method may be practiced using other structure and/or functionality in addition to one or more of the aspects set forth herein.

It should also be noted that the drawings provided in the following embodiments only illustrate the basic concept of the present application in a schematic way, and the drawings only show the components related to the present application rather than the number, shape and the number of components in actual implementation. For dimension drawing, the type, quantity and proportion of each component can be changed at will in actual implementation, and the component layout may also be more complicated.

Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, one skilled in the art will understand that the described aspects may be practiced without these specific details.

The embodiment of the present application provides a method for acquiring a passenger flow state.

Referring to FIG. 1 , a method for acquiring a passenger flow state provided by an embodiment of the present disclosure, applied to a fisheye camera, includes:

In step S101, video information of a public place is collected through a fisheye camera.

In this embodiment, the fisheye camera is used to monitor the public place in an all-round manner without dead angle, so as to obtain video information with relatively complete coverage. In this way, the integrity of the video information monitored in the public place can be improved.

Step S102: Determine, according to the video information, a multi-frame image for inputting the deep neural network model.

In this embodiment, the fisheye camera integrates a chip equipped with a deep neural network model, receives multiple frames of images in the video information through the deep neural network model, analyzes the multiple frames of images, and then determines the number of passengers in the images in the public place.

In this embodiment, referring to FIG. 2 , step S102 may include the following steps:

Step S1021, acquiring the passenger flow status of the public place at the last moment.

Step S1022: Determine an image capture frame rate according to the passenger flow state at the last moment.

Step S1023: Determine corresponding multi-frame images from the video information according to the image acquisition frame rate.

In this embodiment, the fisheye camera determines the number of people, the heat map, and the crowding degree of the video information in the public place in real time. Since the number of people in the image of video information often has a process from sparse to crowded, or from crowded to sparse, a set of power consumption reduction logic can be provided, that is, the acquisition frame rate sent to the deep neural network model varies with the number of people. increase and decrease with the decrease in the number of personnel. It is understandable that when there are no people or few people in the image, the frame rate sent to the deep neural network model is reduced; when there are many people in the picture, the frame rate sent to the deep neural network model is increased. This can reduce the consumption of computing resources of the device and prolong the life of the device.

In this embodiment, the method for acquiring the passenger flow state may further include the following steps:

Performing human body part labeling on the original image data collected by the fisheye camera to obtain the first training data;

Post the image of the person on the base map of the unmanned scene, and obtain the second training data;

The initial deep learning neural network is trained according to the first training data and the second training data to obtain the deep learning neural network.

In this embodiment, images captured by a large number of fisheye cameras are used as the original image data, and the head and shoulders of the personnel in the original image data are marked to obtain the first training data. At the same time, the unmanned scene is used as the base map. Post a person image to get the second training data, the first training data and the second training data can enrich the training data, obtain the deep learning training data and test data, combine the first training data and the second training data into the heat map prediction analysis, Design a heatmap analysis loss function, train the original deep neural network model, perform batch testing on the trained deep neural network model, and select the optimal model on the test set as the deep neural network model.

In this way, the deep neural network model can be obtained by pre-training the deep neural network model, and the accuracy of subsequent heat map prediction analysis based on the deep neural network model can be improved.

Step S103, performing heat map prediction analysis on the multiple frames of the images through the deep learning neural network to determine a local heat value.

In this embodiment, the fisheye camera uses a deep neural network model to perform preprocessing and heat map analysis on the collected multi-frame images, and analyzes the thermal distribution of passenger flow and the degree of passenger flow congestion, and obtains the thermal map of passenger flow in the current image. Among them, the deep neural network model can be integrated to obtain the corresponding computer vision algorithm, and the heat map corresponding to the image can be obtained through the computer vision algorithm.

Step S104, determining the current passenger flow state of the public place according to the local thermal value.

In this embodiment, the passenger flow state may include a crowded state, a sparse state, and an unmanned state. The corresponding thermal value ranges can be set for the crowded state, the sparse state, and the unmanned state. For example, the crowded thermal value range, the sparse thermal value range, and the unmanned thermal value range can be set. If the local thermal value belongs to the range of the crowded thermal value, it is determined that the current passenger flow state is a crowded state. If the local thermal value belongs to the sparse thermal value range, it is determined that the current passenger flow state is a sparse state. If the local thermal value belongs to the unmanned thermal value range, it is determined that the current passenger flow state is the unmanned state.

In this embodiment, the method for acquiring the passenger flow state may further include the following steps:

Determine whether the current passenger flow state is in a crowded state;

If so, acquire the current image of the public place, and send the current image and the congestion warning information to the server.

In this embodiment, it is determined whether the local thermal value exceeds the preset threshold, and if the local thermal value exceeds the preset threshold, an alarm of excessive congestion is triggered. At the same time, the captured images are sent to the monitoring and management platform to assist the management personnel in the flow of passengers. It is added that the monitoring and management platform may be a back-end server. When the congestion level is too high alarm is triggered, the fisheye camera sends the congestion warning information and the current screen snapshot image to the back-end server. It can also support the processing of removing repeated alarm events to avoid continuous reporting of the same congestion event. The staff checks the congestion warning information of the back-end server, and processes the warning events. After the processing is completed, click the processing completion button, and send the command to the fisheye camera, so that it has the warning function again.

In the method for acquiring the passenger flow state in the embodiment of the present disclosure, the video information of the public place is collected by the fisheye camera; the multi-frame images used for inputting the deep neural network model are determined according to the video information; Frame the image to perform heat map prediction analysis to determine a local heat value; determine the current passenger flow state of the public place according to the local heat value. Through the solution of the present disclosure, the fisheye camera is used to collect video information, and the video information is predicted and analyzed by the deep neural network model integrated with the fisheye camera, so as to obtain the local thermal value of the public place, and further determine the current value based on the local thermal value. Passenger flow status, improve camera coverage, reduce costs, automatically report passenger flow congestion status, and improve the efficiency of early warning processing of emergencies.

Corresponding to the above method embodiments, referring to FIG. 3 , an embodiment of the present disclosure further provides an apparatus 300 for acquiring a passenger flow state, including:

The collection module 301 is used to collect video information in public places through a fisheye camera;

a first determining module 302, configured to determine, according to the video information, a multi-frame image for inputting a deep neural network model;

An analysis module 303, configured to perform a thermal map prediction analysis on the multiple frames of the images through the deep learning neural network to determine a local thermal value;

The second determining module 304 is configured to determine the current passenger flow state of the public place according to the local thermal value.

In this embodiment, referring to FIG. 4 , the first determining module 302 includes:

The acquisition sub-module 3021 is used to acquire the passenger flow status of the public place at the last moment;

The first determination submodule 3022 is used to determine the frame rate of image acquisition according to the passenger flow state at the last moment;

The second determination sub-module 3023 is configured to determine the corresponding multi-frame images from the video information according to the image acquisition frame rate.

In this embodiment, referring to FIG. 5 , the device 300 for acquiring the passenger flow state further includes:

Judging module 305, for judging whether the current passenger flow state is in a crowded state;

If so, acquire the current image of the public place, and send the current image and the congestion warning information to the server.

In this embodiment, referring to FIG. 6 , the device 300 for acquiring the passenger flow state further includes:

A training module 306, configured to perform human body part labeling on the original image data collected by the fisheye camera to obtain first training data;

Post the image of the person on the base map of the unmanned scene, and obtain the second training data;

The initial deep learning neural network is trained according to the first training data and the second training data to obtain the deep learning neural network.

The apparatus 300 for obtaining the passenger flow state provided in this embodiment can correspondingly execute the content in the foregoing method embodiments. For the parts not described in detail in this embodiment, reference is made to the content recorded in the foregoing method embodiments, which will not be repeated here.

The device for acquiring the passenger flow state provided by this embodiment collects video information of public places through a fisheye camera; determines a multi-frame image for inputting a deep neural network model according to the video information; The image is subjected to a thermal map predictive analysis to determine a local thermal value; the current passenger flow state of the public place is determined according to the local thermal value. Through the solution of the present disclosure, the fisheye camera collects video information, and performs heat map prediction and analysis on the video information through the deep neural network model integrated with the fisheye camera, so as to obtain the local heat value of the public place, and further determine the current passenger flow based on the local heat value. Status, improve camera coverage, reduce costs, automatically report passenger congestion status, and improve the efficiency of early warning and processing of emergencies.

This embodiment also provides a fisheye camera, the fisheye camera includes:

at least one processor; and,

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute the method for acquiring the passenger flow state in the foregoing method embodiments.

Embodiments of the present disclosure further provide a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and the computer program is used to cause the computer to execute the method for obtaining the passenger flow state in the foregoing method embodiments.

Embodiments of the present disclosure also provide a computer program product, the computer program product includes a computer program stored on a computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by a computer, cause the computer to execute The acquisition method of the passenger flow state in the foregoing method embodiments.

In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network, or from a storage device, or from a read-only memory. When the computer program is executed by the processing apparatus, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed.

It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, electrical wire, optical fiber cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

The above-mentioned computer-readable medium may be included in the above-mentioned fisheye camera; or may exist alone without being assembled into the fisheye camera.

The computer-readable medium carries one or more programs, and when the one or more programs are executed by the processor of the fisheye camera, the fisheye camera implements the above-mentioned method for obtaining the passenger flow state.

Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including object-oriented programming languages—such as Java, Smalltalk, C++, but also conventional Procedural programming language - such as the "C" language or similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through Internet connection).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented with a combination of dedicated hardware and computer programs.

The units involved in the embodiments of the present disclosure may be implemented in a software manner, and may also be implemented in a hardware manner. Among them, the name of the unit does not constitute a limitation of the unit itself under certain circumstances.

It should be understood that portions of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof.

The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited to this. Any person skilled in the art who is familiar with the technical scope of the present disclosure can easily think of changes or substitutions. All should be included within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims (10)

1. a kind of acquisition method of passenger flow state, is characterized in that, is applied to fisheye camera, and described method comprises: Collect video information in public places through fisheye cameras; Determine a multi-frame image for inputting the deep neural network model according to the video information; Perform heat map prediction analysis on multiple frames of the images through the deep learning neural network to determine local heat values; The current passenger flow state of the public place is determined according to the local thermal value. 2. The method for obtaining a passenger flow state according to claim 1, wherein the determining according to the video information is used to input a multi-frame image of a deep neural network model, comprising: Obtain the passenger flow status of the public place at the last moment; Determine the image capture frame rate according to the passenger flow state at the last moment; Corresponding multi-frame images are determined from the video information according to the image acquisition frame rate. 3. The acquisition method of passenger flow state according to claim 1, is characterized in that, described method also comprises: Determine whether the current passenger flow state is in a crowded state; If so, acquire the current image of the public place, and send the current image and the congestion warning information to the server. 4. The acquisition method of passenger flow state according to claim 1, is characterized in that, described method also comprises: Performing human body part labeling on the original image data collected by the fisheye camera to obtain the first training data; Post the image of the person on the base map of the unmanned scene, and obtain the second training data; The initial deep learning neural network is trained according to the first training data and the second training data to obtain the deep learning neural network. 5. A device for acquiring a passenger flow state, wherein the device comprises: The acquisition module is used to collect video information in public places through the fisheye camera; a first determining module, configured to determine, according to the video information, a multi-frame image for inputting a deep neural network model; an analysis module, configured to perform heat map prediction and analysis on multiple frames of the images through the deep learning neural network to determine a local heat value; The second determining module is configured to determine the current passenger flow state of the public place according to the local thermal value. 6. The device for obtaining a passenger flow state according to claim 5, wherein the first determining module comprises: Get the sub-module, which is used to obtain the passenger flow status of the public place at the last moment; a first determination submodule, configured to determine an image capture frame rate according to the passenger flow state at the last moment; The second determination submodule is configured to determine corresponding multi-frame images from the video information according to the image acquisition frame rate. 7. The device for obtaining a passenger flow state according to claim 5, wherein the device further comprises: a judging module for judging whether the current passenger flow state is in a crowded state; If so, acquire the current image of the public place, and send the current image and the congestion warning information to the server. 8. The device for obtaining a passenger flow state according to claim 5, wherein the device further comprises: The training module is used for labeling the human body parts on the original image data collected by the fisheye camera to obtain the first training data; Post the image of the person on the base map of the unmanned scene, and obtain the second training data; The initial deep learning neural network is trained according to the first training data and the second training data to obtain the deep learning neural network. 9. A fisheye camera, characterized in that the fisheye camera comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the preceding claims 1-4. The method for obtaining the passenger flow status described above. 10 . A computer-readable storage medium, characterized in that it stores a computer program, and when the computer program runs on a processor, the computer program executes the method for obtaining a passenger flow state according to any one of claims 1 to 4 .

Images