CN106022522A - Method and system for predicting stocks based on big data published by internet - Google Patents

Abstract

The invention discloses a method and system for predicting stocks based on big data published on the Internet. Firstly, relevant information about stocks before the trading day is crawled; Predictive model training, the evaluation standard of the model is the rate of return for a period of time according to the operation method of selling the stocks bought in the previous trading day at the opening of the market every day and buying the stocks recommended in the current trading day; Construct a new test set from the data, and use the prediction model trained in the previous step to make predictions to get the final recommended stocks. The present invention provides a new useful and reliable source of information for quantitative stock selection or stock forecasting. The addition of these information combined with traditional information can better reflect the market. On this basis, the stock forecasting model obtained by using machine learning technology can better capture the market. The internal operating mechanism can effectively improve the returns of investors.

Description

A method and system for predicting stocks based on Internet-disclosed big data

technical field

The present invention relates to a big data stock forecasting method, and in particular to a big data stock forecasting method and system based on Internet-based stockholder operations, analyst forecasts, stockholder comments, news, announcements, historical stock prices, capital flows, and fundamentals.

Background technique

Before the 1970s, stock investment was a qualitative analysis without data application, but a subjective art. With the popularization of computers, many people began to study the laws driving stock price changes, and replaced the traditional fundamental research methods with models. The concepts of price-earnings ratio and price-to-book ratio were born, and quantitative investment emerged.

From subjective judgment to quantitative investment is a process from art to science. Before the 1970s, a fundamental researcher could only pay attention to 20 to 50 stocks, and the coverage was very limited. With quantitative models, all stocks can be covered, which is a big leap. In addition, with the development of computer processing power, the amount of information used has also changed dramatically. In the past, it was enough to look at three indicators, but now more and more indicators are looked at, and the predictions made are becoming more and more accurate.

With the advent of the 21st century, quantitative investment has encountered a new bottleneck, which is homogeneous competition. The quantitative models of various institutions are increasingly converging, resulting in investment results rising and falling at the same time. "Can you use larger data to find patterns before seeing the report data?" This is the problem that big data strategy entrepreneurs are trying to solve.

The investment model designed by Robert Shiller, the 2013 Nobel Prize winner in economics, is still praised by the industry today. In his model, three variables are mainly referred to: the planned cash flow of the investment project, the estimated cost of capital of the company, and the reaction of the stock market to the investment (market sentiment). He believes that the market itself has subjective judgment factors, investor sentiment will affect investment behavior, and investment behavior directly affects asset prices. Computers extract useful information by analyzing news, research reports, social information, search behavior, etc., with the help of natural language processing methods; and with the help of machine learning intelligent analysis, quantitative investment in the past could only cover dozens of strategies, while big data investment can cover Thousands of strategies.

Based on this, the traditional stock forecast is based on the historical trend of stock prices, capital flows, and the market value of each stock, price-earnings ratio and other information for stock analysis and forecasting. In the current situation where the Internet has deeply affected many traditional industries, compared to decades ago before the Internet was invented, or even before the Internet was not so popular, in addition to the traditional stock data, there are many stocks on the Internet. Data, including the actual operation of stockholders with public data, analysts' forecasts, stockholders' comments, news, announcements and other information. To a certain extent, this information is the response to the current stock market, and it will also show the response to the expectations of the future stock market. The present invention attempts to utilize these new useful data and traditional data to utilize techniques such as natural language processing, machine learning to create a kind of big data stock prediction model.

Invention content:

Purpose of the invention: Aiming at the problems existing in the prior art, the present invention proposes a big data quantitative stock selection method and system based on the operating behaviors of shareholders and analysts published on the Internet, and provides investment reference for shareholders and fund companies.

Technical solution: The present invention proposes a method for predicting stocks based on Internet-disclosed big data, including the following steps:

1) Crawl the relevant information of the stock before the trading day;

The specific crawling method is: first crawl some proxy IPs, then use the Scrapy framework to crawl the data of related websites, convert the data into json format and store it in the Mongodb database;

The specific information crawled includes stock operations, analysts' forecasts, stockholder comments, news, announcements, and the stock price of each stock on websites such as Xueqiu.com, Golden Compass, Stock Bar, Phoenix Finance, and Sina Finance. Historical price data, market capitalization, return on equity, return on assets, earnings per share growth rate, current liability ratio, enterprise value multiples, year-on-year growth rate of net profit, equity concentration, free float market value, and stock price returns in the last month rate and volatility.

2) Use the data crawled in step 1 to perform feature extraction, construct a training data set, and use Group Lasso for prediction model training;

Constructed training data set: It is composed of the data of 5 trading days in the previous week of the current trading day. For each trading day of these 5 trading days, each stock is composed of features and categories, where the features are based on relevant information The processed vector indicates whether the stock price will rise in the next trading day, if it rises, it will be 1, otherwise it will be 0, so as to obtain the initial training matrix; due to the redundancy of data, this step will first filter out the lack of information The specific filtering criteria are: filter out the samples with less than 10 operations on stocks by stockholders in the crawled data on that day.

The extraction method of the vector characterizing the characteristics of the stock is as follows: for the stockholders’ operation data, the stockholders are divided into 10 groups according to the stockholders’ last month’s return rate, and for each level group, the group’s previous 1 day, 3 Features such as the number of buys, number of sells, position volume, position changes, and the average rate of return of the group at each time stamp in each time stamp of 1 day, 7 days, 15 days, and 30 days ;

For the analyst's forecast data, extract the analyst's characteristics such as the number of purchases and the number of sales in each timestamp of the previous 1 day, 3 days, 7 days, 15 days, 30 days, etc.;

For the shareholder comment data, extract the number of analysts’ comments on the stock in each timestamp of the previous 1 day, 3 days, 7 days, 15 days, 30 days, etc., and the average value of the sentiment value of each comment, Variance and other characteristics;

For the news data, extract the number of news of the stock in each timestamp of the analyst's previous 1 day, 3 days, 7 days, 15 days, 30 days, etc., and the average value of the sentiment value of each news, Variance and other characteristics;

For the announcement data, extract the number of announcements made by analysts on the stock in each timestamp of the previous 1 day, 3 days, 7 days, 15 days, 30 days, etc., and the corresponding announcement keywords in each announcement Features such as the sum of the number of occurrences of words in the library;

For historical stock price data, extract the opening price, closing price, highest price, and lowest price of the stock in each timestamp of the analyst's previous 1 day, 3 days, 7 days, 15 days, 30 days, etc. , the ratio to the price of the previous 30 days, the slope of the 3-day line, the slope of the 7-day line, the slope of the 10-day line, the slope of the 15-day line, the slope of the 30-day line, etc.;

For the capital flow data, extract the ratio of the inflow and outflow of the stock's main funds in each time stamp of the previous 1 day, 3 days, 7 days, 15 days, 30 days and other time stamps of the stock, etc. feature;

For other information data, extract the stock's current market value, return on equity, return on assets, growth rate of earnings per share, current liability ratio, enterprise value multiple, year-on-year growth rate of net profit, equity concentration, free float market value, and recent Features such as one-month stock price return and volatility;

For text data such as stockholder comments, news, announcements, etc., first use natural language processing technology to segment the text based on the two lexicons of financial sentiment and announcement keywords, and then calculate each stockholder comment, The emotional value of news, etc., and the number of times the corresponding keywords appear in the announcement. The financial emotional lexicon lists some stock emotional keywords and the corresponding emotional scores of the keywords. The announcement keyword library lists some announcement-related keywords. Key words, these two thesauruses are manually marked by crowdsourcing.

Since in the feature extraction, the shareholder operation data is divided into 10 groups according to the shareholder's last month's rate of return, each group here is equivalent to a group (Group), and the characteristics of each group are relatively Strong correlation, but the correlation between features between different groups is not so strong. When training the model, it is hoped that the features in the same group can be considered as a whole. On this basis, use machine learning. The Group Lasso algorithm can better take these factors into consideration, so the Group Lasso algorithm is selected.

The Group Lasso algorithm is expressed as follows:

${\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}}_{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; min</span>}}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; Y</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; x</span>}\mathrm{<span\; class="notranslate">\; \&beta;</span>}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}\underset{\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; G</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; g</span>}}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>$

in, is the model training result, X is the training sample matrix, Y is the category vector of the sample, I _g represents the feature index belonging to the gth Group, where g=1,...,G, Indicates the value of the weight result of the model training corresponding to the feature index belonging to the gth Group.

In the process of model training, use the cross-validation method to select the stock with the highest prediction probability in descending order according to the predicted probability for each round of test set, and then sell the stocks bought on the previous trading day according to the opening of each day, and buy the current transaction. Adjust the parameters of the model based on the rate of return of the total return of the stocks recommended on a daily basis for two weeks.

3) Crawl the data of the trading day to construct a new test set, and use the prediction model trained in step 2 to make predictions to obtain the final recommended stocks.

The present invention also proposes a system for predicting stocks based on Internet-disclosed big data, including a data crawling storage module, a forecasting model training module, and a stock prediction module; wherein, the data crawling storage module is used to crawl and store stock related information ;The forecasting model training module uses the data crawled before the trading day to construct a training data set, and uses GroupLasso to train the forecasting model; the stock forecasting module uses the data crawled on the trading day to construct a new test set, and uses the trained forecasting model to predict Ultimate recommended stock.

The big data stock prediction system based on Internet public data also includes a display module, which is used to display the stock prediction results to customers.

Beneficial effects: the present invention provides a new useful and reliable source of information for quantitative stock selection or stock forecasting, such as stockholders' operations, analysts' predictions, news, announcements, research reports and other data relative to traditional historical prices such as stocks Data such as data and capital flow are new sources of data. To a certain extent, this information is a response to the current stock market, and it will also show a response to the expectations of the future stock market. Because there is a large amount of text data, the difficulty of real-time crawling and analysis of these data is more difficult than the crawling and processing of traditional stock data. The present invention uses technologies such as Scrapy frame crawler and natural language processing to perform real-time crawling for these types of data And processing, as well as the combination with traditional data such as historical stock prices, capital flows, etc., can better reflect the market. Because some parts of the extracted features of the present invention are divided into multiple groups according to the stockholders' last month's rate of return, the features inside each group have a strong correlation, while the features between different groups The correlation is not so strong. During model training, we hope to have an overall consideration of the features in the same group. On this basis, using the Group Lasso algorithm in machine learning can better take these factors into account, and get The stock forecasting model is more able to capture the internal operating mechanism of the market, which greatly improves the returns to investors.

Description of drawings

Fig. 1 is the overall architecture diagram of the stock prediction system of the present invention;

Fig. 2 is the architectural diagram of the data crawling storage module of the present invention;

Fig. 3 is the architecture diagram of the predictive model training module of the present invention;

Fig. 4 is a structure diagram of the stock prediction prediction module of the present invention.

detailed description

Below in conjunction with specific embodiment, further illustrate the present invention, should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention, after having read the present invention, those skilled in the art will understand various equivalent forms of the present invention All modifications fall within the scope defined by the appended claims of the present application.

Fig. 1 is the overall framework of the stock prediction system of the present invention, including four modules, a data crawling storage module, a stock prediction model training module, a stock prediction module and a display module. The language of the present invention uses Python, and the database uses Mongodb.

The data crawling storage module is shown in Figure 2. The crawler uses the Scrapy framework. Scrapy is a fast, high-level Web information crawling system developed based on Python. It is mainly used to automatically access related Web sites and extract structured information from pages. data. Scrapy uses the efficient Twisted asynchronous network library to handle network communication. The overall architecture of Scrapy is shown in Figure 3.

In the crawler, in order to solve the anti-crawling problem of websites such as Xueqiu.com, first crawl some proxy IPs, and then use the Scrapy framework to crawl websites such as Xueqiu.com, Golden Compass, Stock Bar, Phoenix Finance, Sina Finance, and Juchao Information. The data is converted into json format and stored in the Mongodb database. Among them, Xueqiu.com can crawl some stockholder operation data, stockholder comments, news, announcements and other data, Golden Compass can crawl to analysts’ forecasts and other data, Stock Bar can crawl to stockholder comments and other data, Phoenix Finance And Sina Finance can crawl to news, historical stock prices, capital flows, fundamentals and other data, and Juchao Information can crawl to announcements and other data.

The stock forecasting model training module is shown in Figure 4. First, construct the training data set for machine learning. The training data set consists of the data of 5 trading days one week before the current trading day. For each trading day of these 5 trading days, each stock of 2780 A shares consists of features and categories, where the features are represented by a vector, which has about 700 dimensions, and the category is whether the stock price on the next trading day is Increase, if it rises, it will be 1, otherwise it will be 0, so that a matrix of about 5*2780*701 can be obtained. This is the initial training set.

Table 1 Composition of feature vectors with about 700 dimensions

Since some stocks do not crawl a lot of data on a certain day, the original 700-dimensional vector description may be distorted, so the stock prediction model training module will filter out data with insufficient information, and the specific filtering criteria can be adjusted according to the evaluation criteria , at this stage, the present invention filters out the samples in which the stockholders operate on stocks less than 10 times in the crawled data on that day. In this way, the filtered training set can be obtained.

Then use the Group Lasso algorithm in machine learning for model training, and the same type of statistics is a Group. This is different from traditional machine learning problems. The evaluation criteria for the quality of the model here is not accuracy rate, F1, etc., but recommends 8 stocks every day according to the model, and sells the stocks bought on the previous trading day according to the daily opening. The rate of return during this period of time is based on the operation method of stocks recommended in the current trading day. This is used to adjust the parameters of the model. The Group Lasso algorithm is expressed as follows:

${\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}}_{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; min</span>}}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; Y</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; x</span>}\mathrm{<span\; class="notranslate">\; \&beta;</span>}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}\underset{\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; G</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; g</span>}}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>$

in, is the model training result, X is the training sample matrix, Y is the category vector of the sample, I _g represents the feature index belonging to the gth Group, where g=1,...,G, Indicates the value of the weight result of the model training corresponding to the feature index belonging to the gth Group.

In this way, a big data stock forecasting model is obtained, and the present invention trains the model of the day about 10 hours before the opening of each trading day.

The prediction module of the big data stock prediction model is shown in Figure 4. The test data set is obtained according to the extracted features of the data crawled on the same day, so that 2780 samples of 2780 A-share stocks can be obtained. According to the training data and filtering method, samples with less information are removed to obtain a filtered test set. Finally, use the trained big data stock prediction model to predict the filtered test set, and select the 8 stocks with the highest output probability as the recommended stocks for the next trading day.

Claims (9)

1. A method for predicting stocks based on big data disclosed by the Internet comprises the following steps:

1) crawling related data information of stocks on the trading day before;

2) performing feature extraction by using the data crawled in the step 1, constructing a training set, and performing training of a big data stock prediction model by using a Group Lasso algorithm;

3) and (4) crawling data on the current day of the trading day to construct a new test set, and predicting by using the prediction model trained in the step (2) to obtain the finally recommended stocks.

2. The method for predicting stocks based on internet published big data as claimed in claim 1, wherein the method for extracting stock information in step 1 is: crawling some agent IPs, then crawling data of related websites by using a Scapy framework, converting the data into a Json format, and storing the Json format into a Mongolb database.

3. The method for predicting stocks based on internet published big data as claimed in claim 1, wherein the specific information crawled in step 1 includes stock operations of stocks related to stocks on websites such as snowweb, gold compass, stock bar, phoenix, new wave, etc., predictions of analysts, stocks reviews, news, announcements, and reports of price history data, market value, net asset profitability, income growth rate per stock, mobile liability ratio, business value multiple, net profit unity growth rate, share right concentration, free circulation market value, and stock price profitability and volatility in the last month.

4. The method for predicting stocks based on internet published big data as claimed in claim 1, wherein said step 2 filters out data with insufficient information, and the specific filtering criteria are: and filtering out samples of the crawled data, wherein the number of operations of stocks by the current shareholder is less than 10 times.

5. The method of claim 1, wherein the training data set constructed in step 2 is composed of data of 5 trading days one week before the current trading day, and each stock is composed of a feature and a category for each of the 5 trading days, wherein the feature is represented by a vector processed according to the related information, and the category is whether the stock price increases for the next trading day, and if the increase is 1, the category is 0, so that an initial training matrix is obtained.

6. The method for predicting stocks based on internet published big data as claimed in claim 5, wherein the vector extraction method for characterizing the stocks is:

for the operation data of the stocks, dividing the stocks into 10 groups according to the previous monthly profitability of the stocks, and extracting the characteristics of the group, such as the number of bought items, the number of sold items, the amount of taken positions, the amount of position change, the average profitability of the group in each time stamp in the time stamp of the group, such as the first 1 day, the last 3 days, the last 7 days, the last 15 days, the last 30 days and the like of the stocks;

for the analyst prediction data, extracting characteristics such as the number of bought items and the number of sold items in each timestamp in timestamps such as the first 1 day, the first 3 days, the first 7 days, the first 15 days, the first 30 days and the like of the stock by the analyst;

for the stock comment data, extracting the comment number of the stock in each time stamp in the time stamps of 1 day, 3 days, 7 days, 15 days, 30 days and the like of the stock, and characteristics such as the mean value, the variance and the like of the emotion value of each comment by an analyst;

for news data, extracting the characteristics of the number of news of the stock, the mean value and the variance of the emotion value of each news and the like in each timestamp of the timestamps of 1 day, 3 days, 7 days, 15 days, 30 days and the like of the stock by an analyst;

for the announcement data, extracting the number of announcements of the stock in each timestamp in timestamps of 1 day, 3 days, 7 days, 15 days, 30 days and the like of the stock by an analyst, and the characteristics of the sum of the occurrence times of words in an announcement keyword library corresponding to each announcement;

for historical stock price data, extracting the characteristics of the analyst such as the opening price, closing price, highest price, lowest price, ratio to the price of the stock of the previous 30 days, slope of a 3-day line, slope of a 7-day line, slope of a 10-day line, slope of a 15-day line, slope of a 30-day line and the like in each timestamp of timestamps such as the previous 1 day, 3 days, 7 days, 15 days, 30 days and the like of the stock;

for the capital flow data, extracting characteristics such as the ratio of the inflow amount and the outflow amount of capital funds of the stocks in each time stamp of the time stamps of the previous 1 day, 3 days, 7 days, 15 days, 30 days and the like of the stocks by an analyst;

for other information data, extracting the current market value, net asset profitability, income increase rate of each stock, mobile liability ratio, enterprise value multiple, net profit same-ratio increase rate, equity concentration ratio, free circulation market value, stock price profitability and fluctuation rate of the last month and other characteristics of the stock;

for text data such as stock comments, news, bulletins and the like, firstly, segmenting a text by adopting a natural language processing technology based on two word banks of a financial emotion word bank and a bulletin key word bank, then, calculating the emotion value of each stock comment, news and the like according to financial emotion words appearing in the text and the frequency of the appearance of corresponding keywords in the bulletin, wherein some stock emotion keywords and emotion scores corresponding to the keywords are listed in the financial emotion word bank, some keywords related to the bulletin are listed in the bulletin key word bank, and the two word banks are obtained by manual labeling in a crowdsourcing mode.

7. The method for predicting stocks based on internet published big data as claimed in claim 6, wherein in the feature extraction, for the operation data of the stocks, the stocks are divided into 10 groups according to the previous month profitability of the stocks, each Group is equivalent to a Group, the features in each Group are strongly related, and the relationship between the features in different groups is not so strong, in order to take the features in the same Group into consideration as a whole, the prediction model training is performed by using the Group Lasso algorithm in machine learning, and the Group Lasso algorithm is expressed as follows:

${\widehat{\mathrm{\&beta;}}}_{\mathrm{\&lambda;}}=\underset{\mathrm{\&beta;}}{\arg \min }\left(\right||Y-X\mathrm{\&beta;}|{|}_2^2+\mathrm{\&lambda;}\underset{g=1}{\overset{G}{\&Sigma;}}\left|\right|{\mathrm{\&beta;}}_{I_g}\left|{|}_2\right)$

wherein,for the model training results, X is the training sample matrix, Y is the class vector of the sample, I_gDenotes a feature index belonging to the G-th Group, where G1., G,representing the value of the weight result trained by the model corresponding to the characteristic index belonging to the g Group;

in the process of model training, a cross-checking method is utilized, stocks with the highest prediction probability are selected according to the predicted probability descending order of each round of test sets, and then the yield of the total income in two weeks is adjusted according to the operation mode of selling the stocks bought on the last trading day and buying the stocks recommended on the current trading day every day.

8. A system for predicting stocks based on big data disclosed by the Internet comprises a data crawling storage module, a prediction model training module and a stock prediction module; the data crawling and storing module is used for crawling and storing relevant information of the stocks; the prediction model training module constructs a training data set by using data crawled before the transaction day and trains a prediction model by using Group Lasso; and the stock forecasting module is used for constructing a new test set by utilizing data crawled on the same day of the trading day and forecasting the finally recommended stocks by using a trained forecasting model.

9. The system for forecasting stocks based on internet published big data as claimed in claim 8, further comprising a presentation module for presenting the stock forecasting result to the client.