CN113450566A

CN113450566A - Urban traffic flow prediction method

Info

Publication number: CN113450566A
Application number: CN202110690799.1A
Authority: CN
Inventors: 左洪年
Original assignee: Zhongke Shuguang Nanjing Computing Technology Co ltd
Current assignee: Zhongke Shuguang Nanjing Computing Technology Co ltd
Priority date: 2021-06-22
Filing date: 2021-06-22
Publication date: 2021-09-28

Abstract

The invention discloses an urban traffic flow prediction method, which comprises the steps of collecting real-time traffic data through devices such as a sensor and the like and preprocessing the real-time traffic data to form a track data stream, converting the traffic data into a plurality of pieces in batches through Spark Streaming to generate a data set, reading the data in the data set in a Direct mode, then establishing a support vector machine model and solving the support vector machine model by using a random gradient descent method to obtain traffic condition trend prediction data; the urban traffic flow prediction method has small memory pressure on the chip, high prediction precision and high calculation speed, is suitable for the domestic Loongson 3B3000 chip, and expands the software ecology of the domestic chip.

Description

Urban traffic flow prediction method

Technical Field

The invention relates to an urban traffic flow prediction method.

Background

An effective way for relieving traffic jam is to establish an intelligent traffic system on the infrastructure of the existing road, design and realize an accurate and real-time traffic flow prediction algorithm which is the prerequisite for the operation of the intelligent traffic system, and how to ensure that massive urban traffic flow data is mined and analyzed in as short a time as possible and accurately predict the urban short-time traffic flow state in real time is the problem to be solved urgently at present; in addition, because the domestic chips cannot reach the level of the international latest chips at present, the current traffic flow prediction method has high requirements on resource utilization rate, memory utilization rate and the like, and cannot be adapted to most domestic chips.

Disclosure of Invention

The purpose of the invention is as follows: the invention aims to provide a city traffic flow prediction method which has small pressure on a chip memory, high prediction precision and high calculation speed.

The technical scheme is as follows: the urban traffic flow prediction method comprises the following steps: (1) collecting real-time traffic data and preprocessing the real-time traffic data; (2) converting the data preprocessed in the step (1) into a plurality of batch fragments and generating a data set; (3) establishing a support vector machine model and solving by using a random gradient descent method to obtain traffic condition trend prediction data; (4) the model is validated and updated based on the traffic data and traffic situation trend prediction data.

The method comprises the following steps that (1) real-time traffic data are collected through a sensor or a satellite and uploaded to a server, the real-time traffic data are preprocessed to form a track data stream, and the track data stream is transmitted into a Kafka message queue. The preprocessing is to fuse redundant data in the trace data stream and filter false data.

Further, in the step (2), data in the data set is read in a Direct manner, and the traffic data is converted into batch fragments through Spark Streaming.

Has the advantages that: compared with the prior art, the invention has the following remarkable advantages: (1) the resource utilization rate and the memory utilization rate are low, data consumption is carried out based on a Direct mode, parallel reading is simplified, and the resource utilization rate and the memory utilization rate of application are reduced; (2) the prediction real-time performance is strong, the prediction precision is high, a Support Vector Machine (SVM) model based on a random gradient descent (SGD) method is constructed, and the optimal solution can be effectively and quickly solved; (3) the software ecology of a domestic chip is expanded, the SVMWithSGD model is used for predicting the rail traffic flow, the method is suitable for a domestic Loongson 3B3000 chip, and the requirements of high safety, strong real-time performance and high prediction precision of a domestic chip platform under the current big data background are met.

Drawings

FIG. 1 is a block diagram of an urban traffic flow forecast overview of the present invention;

FIG. 2 is a diagram of a Spark Streaming combination Kafka real-time processing system according to the present invention;

FIG. 3 is a flowchart of the SVMWithSGD algorithm training of the present invention.

Detailed Description

The technical scheme of the invention is further explained by combining the attached drawings.

As shown in FIG. 1, the urban traffic flow prediction method provided by the invention utilizes Spark Streaming to perform real-time Streaming processing on the preprocessed track data, simultaneously performs data compression to retain original data so as to facilitate model adjustment and reduce storage space, and finally constructs a gradient optimization SVM model to realize a track prediction function, so that the method can meet the requirements of high safety, strong real-time performance and high prediction precision of a domestic chip platform under the current big data background.

(1) Collecting and pre-processing traffic flow data

The method comprises the steps of collecting real-time traffic data by using devices such as sensors and satellites, transmitting the real-time traffic data into a server of a big data analysis center for data preprocessing, fusing redundant data, filtering false data, transmitting the data into a Kafka message queue, compressing the preprocessed urban traffic flow data to form a track data stream, reducing storage space, and storing the data for model verification and updating.

(2) Track data stream conversion batch (batch) slicing and generating data sets

As shown in fig. 2, a continuously input track-forming data stream is converted into a plurality of batch fragments through Spark Streaming, and data is read based on a Direct method that does not require a special Receiver any more, that is, data is read when calculation is required. When the batch task is triggered, the data is read by the executive and participates in the data calculation process of other executors, and when the batch task is triggered next time, the Kafka data is read by the executive and calculated. The requirement of data consumption based on the Direct mode on the memory is not high, only the memory required by batch calculation needs to be considered, meanwhile, the data accumulation cannot be influenced when the batch task is accumulated, and a data set generated after data stream processing is used as a training data set T of the random gradient optimization SVM model.

(3) Building a SVMWithSGD model

The learning algorithm of the SVM is an optimization algorithm for solving convex quadratic programming, and the learning strategy of the SVM is interval maximization, can be formalized into a problem for solving the convex quadratic programming, and is also equivalent to a minimization problem of a regularized hinge loss function. The basic principle of the SVM algorithm is to solve a separation hyperplane, which is infinite for a linearly separable data set, but the separation hyperplane which can correctly divide a training data set and has the largest geometrical interval is unique. And wx + b is 0, namely the separating hyperplane. The gradient descent method is the most common method for solving the unconstrained optimization problem, and is an iterative method, and the main operation of each step is to solve the gradient vector of an objective function, and take the negative gradient direction of the current position as the search direction. The gradient descent method is characterized in that: the closer to the target value, the smaller the step size, the slower the descent speed. The random gradient descent randomly selects a sub-gradient approximate integral gradient of a training sample to calculate when calculating the direction of the fastest descent instead of scanning all training data sets, and simultaneously updates the required separation hyperplane method vector w through multiple iterations, so that the optimal solution of w can be effectively and quickly calculated, and the iteration speed is accelerated.

The solution of the SVM optimal problem can be divided into two directions, one of which can be expressed as the dual of the solution of the original problem, and the other is to directly optimize the original problem. The SGD randomly selects a sub-gradient of a training sample to approximate the whole gradient to calculate when calculating the direction of the fastest descending, instead of scanning all training data sets, and meanwhile, the optimal solution of w can be effectively and quickly solved by updating the solved separated hyperplane normal vector w through multiple iterations, so that the iteration speed is accelerated.

As shown in fig. 3, which is a SVMWithSGD algorithm training flow chart, firstly, the SVM model parameters are determined by using the processed feature vectors, and then, a gradient optimization method is used for solving; training data set T { (x)₁，y₁)，(x₂，y₂)，...，(x_N，y_N) In which x_i∈Rⁿ，y_i∈{+1，-1}，i＝1，2，...，N，x_iIs the i-th feature vector, y_iIs a class label, which is positive when it equals +1 and negative when it equals-1. Then, let the training data set be linearly separable, λ be the optimization parameter, λ ≧ 0. The quadratic programming model is as follows:

the gradient of w is:

st.y_iw^Tx_i＜1，i＝1，2，...，n

the batch gradient descent method needs to substitute all samples into a model for calculation and solution, when the n-dimensional data quantity with the number of the samples being m is subjected to cyclic iteration to solve the descent gradient, the calculation complexity is O (mn), and when the processed data quantity is the same as the batch data of the urban traffic, the iteration frequency needs to be large, the program operation time is slow, and the requirement of traffic flow prediction cannot be met. Therefore, the SGD is selected to solve the SVM model to obtain traffic condition trend prediction data, such as the average speed pre-data of the vehicles at the entrance, the prediction data of the jam condition at the entrance and the like, and the updating process at each time is as follows.

Wherein w_tIn the descending direction, g (w)_t) Is a gradient representation in the falling direction, eta_tStep size, i.e. the step size is searched in the descending direction, t ═ 1, 2.

(4) Timely updating SVMWithSGD model

And adjusting the prediction result in real time according to the continuously input urban traffic data stream, carrying out batch processing on the compressed preprocessed data to verify the accuracy of the model and updating the model.

Claims

1. The urban traffic flow prediction method is characterized by comprising the following steps:

(1) collecting real-time traffic data and preprocessing the real-time traffic data;

(2) converting the data preprocessed in the step (1) into a plurality of batch fragments and generating a data set;

(3) establishing a support vector machine model and solving by using a random gradient descent method to obtain traffic condition trend prediction data;

(4) the model is validated and updated based on the traffic data and traffic situation trend prediction data.

2. The urban traffic flow prediction method according to claim 1, wherein the support vector machine model in step (3) is:

wherein (x)_i,y_i) And (3) regarding the data in the data set in the step (2), wherein w is a separation hyperplane normal vector, n is a sample dimension in the data set, and lambda is an optimization parameter.

3. The urban traffic flow prediction method according to claim 1, wherein the iterative formula for solving the support vector machine model by using the stochastic gradient descent method in step (3) is as follows:

wherein w_tIn the descending direction, η_tFor the search step size, g (w)_t) For the gradient in the descending direction, t is 1, 2.

4. The urban traffic flow prediction method according to claim 1, wherein the step (1) is: real-time traffic data are collected through a sensor or a satellite and uploaded to a server, and are preprocessed to form a track data stream which is transmitted into a Kafka message queue.

5. The urban traffic flow prediction method according to claim 4, wherein the preprocessing is to fuse redundant data in the trajectory data stream and filter spurious data.

6. The urban traffic flow prediction method according to claim 1, characterized in that in step (2), data in the data set is read in a Direct manner.

7. The urban traffic flow prediction method according to claim 1, wherein the traffic data is converted into batch segments through Spark Streaming in step (2).