RU2828611C1 - System and method of classifying electronic mail - Google Patents

Abstract

FIELD: data processing.

SUBSTANCE: invention relates to systems and methods for detecting spam emails based on analysis of statistical data on received emails from clients. Result is achieved due to stages, at which: obtaining statistical data of messages of clients and result of classification of messages on the side of clients; excluding from further analysis data of messages, which relate to at least one of: messages, which are automatic replies; messages relating to internal correspondence of the organization; generating features based on statistical data for non-excluded messages; combining features into a single vector for each message; clustering of messages is carried out using a machine learning algorithm based on vectors of signs of messages; messages in each cluster are classified according to the following rules: if the cluster contains more than a predetermined threshold of messages classified on the client side as messages containing spam, classifying all messages in the cluster as messages containing spam; if the cluster contains not more than a predetermined threshold of messages classified as spam on the client side, the messages which were not classified as spam on the client side are classified using a machine learning model.

EFFECT: high accuracy of detecting spam emails.

17 cl, 4 dwg

Description

Field of technology

The present invention relates to the field of information technology, and more specifically to systems and methods for detecting spam emails based on an analysis of statistical data on emails received from customers.

State of the art

Spam is a type of mailing with the purpose of advertising (often unwanted) a particular product or service, as well as with the purpose of committing fraudulent actions. All kinds of technical means are currently actively used to send spam, including proxy servers, free email servers that allow automated sending of emails, and botnets. In addition to messages sent by email, spam also includes messages sent using instant messaging protocols in social networks, blogs, dating sites, forums, as well as SMS and MMS messages.

Spam emails are a serious problem in the modern world, as they already account for 70-90% of the total volume of email traffic. Such a quantity of spam sent over computer networks causes significant inconvenience to email users. In particular, spam causes a deterioration in network capacity, waste of email server resources, and an increase in the time it takes for users to process emails. Thus, it is necessary to continuously combat spam.

Various mechanisms have been developed to combat spam, such as blacklists of senders, greylists that require repeated mail server requests for sending, and contextual spam filtering methods. For example, in publication RU 2541123 C1 describes a method of message analysis that is carried out using whitelists and blacklists to identify legitimate and unwanted messages. Spam filtering typically focuses on analyzing the content of an email message, which may require, firstly, significant analysis time and, secondly, openness of the analyzed information, which is unacceptable in the case of transmitting information without the consent of users. There are also cases where emails may contain confidential information. Filtering-based methods typically require an exact match to detect and distinguish emails containing spam from legitimate emails. In addition, mass spam distribution further exacerbates the filtering problem, since the greater the volume of emails that need to be filtered, the more computing resources must be allocated for filtering.

It is possible to improve the effectiveness of spam filtering by avoiding extensive content analysis of each email message or by introducing pre-analysis. However, current anti-spam technologies do not provide an effective mechanism for rapid and statistically accurate analysis of email headers and the characters that make up the headers. Therefore, it is necessary to have an effective mechanism for performing rapid analysis of email headers, either alongside standard email filtering mechanisms, or as a pre-analysis for full content analysis during filtering, or even as a stand-alone method for analyzing the summary of some new forms of spam messages.

Another problem with modern anti-spam tools is that there is a possibility of a type I error, i.e. a false positive. A false positive in this case is a situation where a legitimate email is identified as spam. In this case, depending on the settings of the protection tools, a legitimate email identified as spam is quarantined or even deleted automatically without notifying the sender or recipient, which leads to the loss of legitimate emails.

In addition, senders of spam emails are also becoming increasingly inventive and trying to take into account the capabilities of spam filtering solutions. Therefore, in some cases, in order to avoid automatic detection, they disguise spam by adding random text to the content of emails or data similar to that contained in legitimate emails.

Therefore, it is important to use statistical data analysis of received customer emails to actually direct spam concealment techniques by spam senders towards spam detection. At the same time, in order to take into account users' interests in the privacy of messages/information contained in emails, it is necessary to identify emails containing spam without analyzing the text contained in the body of the emails.

Therefore, there is a need to create a solution that would eliminate the above-mentioned shortcomings and would allow for more accurate spam detection. The present invention allows for solving the above-mentioned shortcomings and problems by clustering emails and then classifying the cluster emails.

Disclosure of the essence of the invention

The objective of the proposed solution is to classify emails based on statistical data to identify emails that contain spam.

The technical result of the present invention consists in increasing the accuracy of detecting emails containing spam, and is achieved using the proposed solution designed to classify emails based on statistical data of emails.

As one embodiment of the present invention, a method is proposed for classifying emails for detecting spam based on statistical data of the emails, comprising the steps of: obtaining statistical data of client emails and a result of email classification on the client side; excluding from further analysis email data that relate to at least one of: emails that are automatic responses; emails that relate to the internal correspondence of the organization; generating features based on statistical data for the non-excluded emails; combining the features into a single vector for each email; clustering the emails using a machine learning algorithm based on the feature vectors of the emails; classifying the emails in each cluster according to the following rules: if the cluster contains more than a predetermined threshold of emails classified on the client side as containing spam, classifying all the emails in the cluster as containing spam; if the cluster contains no more than a predetermined threshold of emails classified on the client side as containing spam, classifying the emails that were not classified on the client side as containing spam using the machine learning model.

In another embodiment of the method, the statistical data of the letters contain at least one of: data on the rules that have been triggered for the letter; the letter category; the sequence of the letter part formats; the type of letter attachments; the size of the attachments in the letter; the type of divisions between the parts of the message; the letter format; domains from the links in the letter; the sequence of technical headers in the letter; the sender's domain from the "From" header; the checksum of the user name from the "From" header; the IP address of the sender of the letter; the size of the letter body; the domain from "message-id"; the "message-id" header type; the sender's domain from the SMTP session; links from the letter; the number of links in the letter; the "User-Agent" header; the "X-Mailer" header.

In another embodiment of the method, the size of the letter attachments is represented as a vector of attachment sizes.

In another embodiment of the method, the size of the letter attachments is presented as the sum of the sizes of all attachments.

In another embodiment of the method, all statistical data are divided into two types: numerical and categorical.

In another embodiment of the method, each feature is formed using one of the following methods: OneHotEncoding; min-max scaling; hashing followed by the application of min-max scaling; CountVectorizer.

In another embodiment of the method, the MurmurHash3 method is used for hashing.

In another embodiment of the method, one of the following methods is used as a machine learning algorithm for clustering letters: k-means; k-medians; DBSCAN.

In another embodiment of the method, clustering of letters is performed each time after the end of a specified period of time.

In another embodiment of the method, two feature vectors are passed to the input of the machine learning model for classifying writing.

In another embodiment of the method, the two feature vectors are: a feature vector of a letter in a cluster (the first vector), which was classified on the client side as a letter that does not contain spam; a feature vector of a letter that was classified on the client side as a letter that contains spam, and the cosine distance to the first vector must be minimal.

In another embodiment of the method, the two feature vectors are the feature vector of a letter in a cluster that was classified on the client side as a letter that does not contain spam, and the feature vector of the cluster centroid.

As another embodiment of the present invention, a system for classifying email messages based on statistical data of messages is proposed, consisting of at least one computer system and containing: a data processing means interacting with a data collection and storage device and a clustering means, and intended for: receiving statistical data of messages and a result of classifying messages on the client side from the data collection and storage device; filtering messages based on statistical data of messages; generating message attributes based on statistical data of messages; a clustering means interacting with the data processing means and the cluster analysis means, and intended for: clustering messages based on the generated message attributes received from the data processing means; transmitting message clusters to the cluster analysis means; a cluster analysis means intended for classifying messages in each received cluster of messages according to the following rules: if the cluster contains more than a predetermined threshold of messages classified on the client side as messages containing spam, classifying all messages in the cluster as messages containing spam; If the cluster contains no more than a pre-defined threshold of emails classified as spam on the client side, classify the emails that were not classified as spam on the client side using a machine learning model.

In another embodiment, the data processing engine generates each feature using one of the following methods: OneHotEncoding; min-max scaling; hashing followed by min-max scaling; CountVectorizer.

In another embodiment, as a machine learning algorithm for clustering, the clustering engine implements one of the following methods: k-means; k-medians; DBSCAN.

In another embodiment, the clustering tool clusters emails each time after a specified period of time has elapsed.

Brief description of the drawings

Additional objects, features and advantages of the present invention will be apparent from reading the following description of the embodiment of the invention with reference to the accompanying drawings, in which:

Fig. 1 shows a diagram of the letter classification system.

Fig. 2 shows a diagram of the dependence of the choice of the letter classification option in the cluster.

Fig. 3 shows the method of classifying letters.

Fig. 4 illustrates an example of a computer system suitable for implementing non-limiting embodiments of the present invention.

Although the invention may have various modifications and alternative forms, the characteristic features shown by way of example in the drawings will be described in detail. It should be understood, however, that the purpose of the description is not to limit the invention to a specific embodiment. On the contrary, the purpose of the description is to cover all changes, modifications, included within the scope of the invention, as defined in the appended claims.

Implementation of the invention

The objects and features of the present invention, the methods for achieving these objects and features will become apparent by reference to exemplary embodiments. However, the present invention is not limited to the exemplary embodiments disclosed below, it can be embodied in various forms. The description provided is intended to help a person skilled in the art to fully understand the invention, which is defined only within the scope of the appended claims.

An email (hereinafter also referred to as a letter) has a certain structure. Therefore, special programs that support the email structure are used to write a letter. The structure of a letter consists of two parts: the header and the body of the letter. The header, in turn, contains a number of fields (fields) intended for filling, and each field is defined by its name (field header) and value. The field value is understood as some type of information. For example, for a field that is intended to indicate the sender of the letter, the name (field header) is "from", and the value will be in the form of the sender's email address, for example, username@kaspersky.com. Examples of other fields in the email header are:

"Sender" - the sender of the letter, usually similar to the "From" field;

"To" - the recipient's name and email address. This is a required field;

"Subject" - the subject of the letter; is an optional but desirable field to fill in;

"CC" (Copy, from the English "CarbonCopy") - addresses of other subscribers receiving copies of the message;

"Date" - date and time of sending the message;

"Reply-to" - email addresses to which the response is sent (they may differ from the sender's address);

"Received" - various identifiers of the Internet servers that forwarded the message from the sender to the recipient;

"Content-type" - the format of the message being sent and the encoding used to create the letter;

"Content-Transfer-Encoding" - a method of data transfer (for example, 7-bit, 8-bit message, etc.);

"Message-ID" - unique message identifier;

"X-mailer" is a program for sending messages via e-mail.

Fig. 1 shows a diagram of a letter classification system (hereinafter referred to as the system) 100. It should be noted that the system 100 is shown only to demonstrate at least one embodiment of the present invention. Other embodiments are possible, and if in some cases they are not shown (i.e., examples of embodiments are missing), this does not mean that they are impossible and/or that the description contains the only embodiment of one or another element of the present invention. In addition, it should be understood that the system 100 in some cases may represent a simplified embodiment of the present invention, and that such embodiments are shown to facilitate a better understanding thereof. Those skilled in the art should understand that other embodiments of the present invention may be significantly more complex.

In general, the system 100 comprises a data processing means 110, a clustering means 120 and a cluster analysis means 130. The system 100 detects spam emails based on the processing and analysis of statistical data of emails 165 received from the data collection and storage device 160. It is worth noting that in one embodiment, the system 100 can be implemented using the computer system (device) shown in Fig. 4.

The data collection and storage device 160 in one embodiment is a "cloud" server that collects and stores statistical data of emails 165 of clients 170A, 170B...170N (hereinafter referred to as clients 170) and the result of email classification on the side of clients 170. For example, a "cloud" server is understood to be a technical means containing the Kaspersky Security Network (KSN) system of Kaspersky Lab. Clients 170 are understood to be at least email clients (for example, Microsoft Outlook, Google Mail, and others) of users installed on users' computers, or email servers. Each client 170 contains an agent (not shown in Fig. 1) designed to classify emails of each client 170 based on the emails of each client 170, apply a set of rules to the emails (for example, characters of different alphabets are used in the "From" header), and also depersonalize information of emails of each client 170.

The agent anonymizes the following data: information from the body of the letter, for example, text messages from users; information from the header of the letter, for example, the email address. Anonymization means data transformation, for example, using hash functions. The agent of each client 170 transmits to the device 160 anonymized statistical data of the letters 165 of each client 170 and the result of classification on the side of each client 170.

The data collection and storage device 160 transmits the result of the classification of letters on the client side 170 and the statistical data of the letters 165 of the clients 170 to the system 100, namely to the data processing means 110. The statistical data of the letters 165 of the clients 170 at least contain one of:

• data on the rules that worked for the letter;

• category of the letter, for example, “casino” or “dating site”;

• type of email attachments, for example, a document with the docx extension or an image with the png extension;

• letter format, for example, html or plain-text;

• sequence of formats of parts of the letter;

• size of investments, for example, in MB;

• the type of separation between parts of the message (boundary), for example outlook_boundary or apple_boundary;

• the sequence of technical headers in the letter, for example, from/to/subject/date/message-id/content-type;

• sender domain from the “From” header;

• checksum of the user name from the "From" header;

• IP address of the letter sender;

• size of the letter body, for example, in KB;

• domain from “message-id”;

• header type "message-id";

• sender domain from the SMTP session (Simple Mail Transfer Protocol);

• links from the body of the letter;

• number of links in the letter;

• header “User-Agent”;

• "X-Mailer" header.

Data on the rules that worked for the letter are presented as a list of rules that worked for the letter of client 170.

"Message-id" is a unique identifier of a digital message used in email.

The User-Agent header is a string containing characteristics that allow servers and network nodes to determine the application type, operating system, manufacturer, and/or version of the user agent.

The sequence of letter parts formats is understood as, for example, text/plain-text/html. This entry means that the first part of the letter is written in text, the second in plain-text format, and the third in html.

The data processing tool 110 is designed to filter letters based on the statistical data of letters 165 and to form characteristics of letters based on the statistical data of letters 165 for further clustering of letters.

In a preferred embodiment, the size of the email attachments is represented as a vector of attachment sizes.

In a particular implementation, the size of the letter attachments is presented as the sum of the sizes of all attachments.

Data processing tool 110 filters the email data to exclude from further data analysis emails that are definitely not spam. Examples of such emails are emails that fall into the following types of emails:

• service letters that are an automatic response (auto-reply);

• letters related to internal correspondence within the organization.

In a particular embodiment, the data processing facility 110 excludes emails from further analysis if the client 170 has outdated rule bases. Outdated rule bases are defined as a rule base that has not been updated for a certain period of time, such as 30 days.

After filtering the letters, the data processing tool 110 forms the letter features based on the statistical data of the letters 165. All data is divided into two types, namely, numerical and categorical. The data processing tool 110 forms the features based on the statistical data of the letters 165.

To convert the numerical data of letters, the processing tool 110 implements the min-max scaling method - a method of normalizing data within a given range, for example, between 0 and 1.

To transform categorical data, the data processing means 110 implements various methods.

OneHotEncoding is a method of encoding categorical data that transforms each unique category into a binary vector in which only one bit, representing the presence of the category, is set to one, and the remaining bits are set to 0. This method is used, for example, to transform the attachment type, the type of separation between message parts, the type of the "message-id" header, and the "User-Agent" header.

Some statistical data of letters 165 are converted by the processing tool 110 into features by means of hashing and subsequent application of the min-max scaling method. This method is used, for example, for the sender domain from the "From" header or the sender domain from the SMTP session.

In a private implementation, the MurmurHash3 method is used to hash statistical data of 165 letters (Mmh3 project description // The Python Package Index (PyPI) repository of software for the Python programming language [Electronic resource]. Access mode: https://pypi.org/project/mmh3 (date of access 03/09/2024/).

CountVectorizer is a categorical data encoding method that transforms a list of categories into a binary vector, where 1 means the category is in the list, and 0 means the category is not in the list. The vector dimension is equal to the number of unique categories. This method is used, for example, for rules triggered for a letter, a letter category, or a sequence of letter part formats.

After the features have been generated, the processing tool 110 combines all the features into a single vector for each letter and passes the specified vectors to the clustering tool 120.

Clustering tool 120 is designed to cluster letters based on the feature vectors received from processing tool 110. Clustering tool 120 performs clustering using a machine learning algorithm, namely the k-means method. This method allows for quick clustering of letters, but its implementation requires specifying the number of clusters. The number of clusters depends on the estimated number of homogeneous mailings, which in turn depends on the network load. For example, to cluster letters that arrive to clients 170 at night, one number of clusters is used, for example 800, and to cluster letters that arrive to clients 170 during the day, another number of clusters is used, for example 1200.

Homogeneous mailing list is a set of letters with similar meaning, where the similarity of meaning is determined by the similarity of statistical data of letters 165 from the specified set.

In a particular embodiment, clustering of letters is performed every hour, and the number of clusters may vary depending on the time of receipt of letters by client 170. For example, for clustering letters that arrive to clients 170 from 11:00 to 12:00, clustering tool 120 uses one number of clusters, for example 1100, and for clustering letters that arrive to clients 170 from 13:00 to 14:00, clustering tool 120 uses another number of clusters, for example 1200.

In another implementation, email clustering is implemented using a different machine learning algorithm, which requires specifying the number of clusters, such as k-medoids.

In another implementation option, email clustering is implemented using a machine learning algorithm that does not require specifying the number of clusters, for example, the Density-based spatial clustering of applications with noise (DBSCAN) method (A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise // Institute for Computer Science, University of Munich Oettingenstr [electronic resource]. Access mode: https://cdn.aaai.org/KDD/1996/KDD96-037.pdf (accessed 03/09/2024)).

After clustering the letters, the clustering tool 120 passes the clusters of letters to the cluster analysis tool 130.

The cluster analysis tool 130 is designed to classify emails containing spam that were not classified on the client side 170 as emails containing spam.

In addition, the cluster analysis tool 130 additionally allows identifying emails that were incorrectly classified on the client side 170.

If a cluster contains more than a predetermined threshold of messages classified on the client side 170 as messages containing spam, the cluster analysis engine 130 classifies all messages in this cluster as messages containing spam. For example, the predetermined threshold is 95 percent. In this case, if the messages classified on the client side 170 as messages containing spam occupy 95 percent or more in this cluster, the cluster analysis engine 130 classifies all messages in the specified cluster as messages containing spam.

If the cluster contains less than a predetermined threshold of emails classified on the client side 170 as emails containing spam, the cluster analysis tool 130 classifies the emails that were not classified on the client side 170 as emails containing spam using a machine learning model that was previously trained to solve the email similarity problem. The machine learning model is based on the decision tree boosting method. Two feature vectors are sequentially passed to the input of the trained machine learning model: a feature vector of the centroid of the specified cluster and a feature vector of the email in the cluster classified as an email containing spam. At the output, the machine learning model produces a similarity coefficient from 0 to 1. If the similarity coefficient is greater than a predetermined number, the cluster analysis tool 130 classifies the analyzed email as an email containing spam.

In a particular implementation option, the machine learning model for solving the classification problem is based on one of the following methods: logistic regression method, support vector machine (SVM), k-nearest neighbors algorithm (k-NN).

In a particular embodiment, the cluster analysis tool 130 classifies emails that were not classified on the client side 170 as emails containing spam using a machine learning model that was previously trained to solve the problem of email similarity, wherein two feature vectors are sequentially transmitted to the input of the trained machine learning model, namely: a feature vector of a email in a cluster (the first vector) that was classified on the client side 170 as an email that does not contain spam, and a feature vector of a email that was classified on the client side 170 as an email containing spam, wherein the cosine distance to the first vector must be minimal.

In a particular embodiment, using the cluster analysis tool 130, letters are identified that do not contain spam, but were classified on the client side 170 as letters containing spam.

In another particular embodiment, if a cluster contains less than a predetermined percentage of messages classified as spam, the cluster analysis tool 130 classifies all messages in the cluster as non-spam. For example, the predetermined percentage is 5. In such a case, if the messages classified on the client side 170 as spam occupy 5 percent or less in the specified cluster, the cluster analysis tool 130 classifies the remaining messages as non-spam.

In a particular embodiment, the cluster analysis tool 130 classifies the emails that were classified on the client side 170 as emails containing spam using a machine learning model that was previously trained to solve the email similarity problem. Two feature vectors are passed to the input of the machine learning model, one of which corresponds to the email in the cluster that was classified on the client side 170 as an email containing spam, and the second vector corresponds to the email that is the centroid of the specified cluster. At the output, the machine learning model produces a similarity percentage. If the similarity percentage is higher than a predetermined number, the cluster analysis tool 130 classifies the analyzed email as an email containing spam.

In a particular embodiment, the cluster analysis tool 130 classifies the letters that were classified on the client side 170 as letters containing spam, using a machine learning model that was previously trained to solve the letter similarity problem, wherein two feature vectors are sequentially transmitted to the input of the trained machine learning model, namely: a feature vector of a letter in a cluster that was classified on the client side 170 as a letter containing spam, and a feature vector of a letter that was not classified on the client side 170 as a letter containing spam, wherein the cosine distance to the first vector must be minimal.

In this manner, the system 100 detects e-mails containing spam that were not detected on the client side 170, thereby increasing the accuracy of detecting e-mails containing spam.

In another particular embodiment, the cluster analysis tool 130 classifies all messages in a cluster using a combination of the methods described above. As shown in Fig. 2, the method for classifying messages in a cluster depends on the percentage of messages in the cluster classified on the client side 170 as messages containing spam. The classification method is selected according to the following rules:

• if the percentage of emails classified on the client side 170 as containing spam is between 0 and the first boundary inclusive, all emails in the cluster are classified as not containing spam;

• if the percentage of emails classified on the client side 170 as emails containing spam is in the range from the first boundary to the second boundary inclusive, all emails in the cluster classified on the client side 170 as emails containing spam are classified using the machine learning model;

• if the percentage of emails classified on the client side 170 as emails containing spam is between the second boundary and the third boundary, all emails in the cluster are classified in accordance with the classification on the client side 170;

• if the percentage of emails classified on the client side as spam is between the third boundary and the fourth boundary, all emails in the cluster classified on the client side as non-spam are classified using the machine learning model;

• if the percentage of emails classified on the client side 170 as containing spam is between the fourth boundary inclusive and 100 percent, all emails in the cluster are classified as containing spam.

In another particular embodiment, if the percentage of emails classified on the client side 170 as containing spam is equal to a value corresponding to the first boundary, all emails in the cluster are classified using a machine learning model.

In another particular embodiment, if the percentage of emails classified on the client side 170 as emails containing spam is equal to a value corresponding to the second or third boundary, all emails in the cluster are classified in accordance with the classification on the client side 170.

In another particular embodiment, if the percentage of emails classified on the client side 170 as containing spam is equal to a value corresponding to the fourth boundary, all emails in the cluster are classified using a machine learning model.

The first, second, third and fourth boundaries correspond to a certain percentage of letters in the cluster classified on the client side 170 as containing spam, and are ordered among themselves as follows - , where I, II, III, IV are the first, second, third and fourth boundaries respectively.

In a particular embodiment, cluster analysis tool 130 sets boundary values using a machine learning model designed to validate boundary values by finding optimal boundary values to minimize false positives.

In another particular embodiment, the cluster analysis tool 130 sets the boundary values depending on the time of receipt of the letters by the client 170. For example, when classifying the letters received by the client 170 from 11:00 to 12:00, the boundaries are equal to 5%, 10%, 90% and 95%, respectively, and when classifying the letters received by the client 170 from 22:00 to 23:00, the boundaries are equal to 10%, 35%, 65% and 90%, respectively.

Fig. 3 shows a method for classifying letters. In some embodiments of the present invention, the method may be performed using the means of the system 100 (see Fig. 1). The various steps of the method are described in more detail below.

At step 310, statistical data of letters 165 of clients 170 and the result of letter classification on the side of clients 170 are obtained by means of data collection and storage device 160. Examples of the obtained statistical data of letters 165 are one or a combination of: data on the rule that was triggered for the letter; letter category; sequence of letter part formats; letter attachment type; size of attachments in the letter; type of divisions between parts of the message; letter format; domains from letter links; sequence of technical headers in the letter; sender domain from the “From” header; user name checksum from the “From” header; letter sender IP; letter body size; domain from “message-id”; “message-id” header type; sender domain from the SMTP session; links from the letter; number of links in the letter; “User-Agent” header; “X-Mailer” header.

In one implementation option, the 165 letter statistics are divided into two types: numeric and categorical.

At step 320, data from letters that relate to at least one of the letters is excluded from further analysis:

• a letter that is an automatic response;

• a letter related to the internal correspondence of the organization.

A specific implementation also excludes emails from further analysis if client 170 has outdated rule bases.

At step 330, features are formed for each non-excluded letter based on the statistical data of letters 165. As described above, features are formed using various methods. For numerical data of letters, the min-max scaling method is used. For categorical data of letters, hashing is used followed by the application of the min-max scaling method, OneHotEncoding or CountVectorizer.

At step 340, the generated features are combined into a single vector for each letter.

At step 350, the letters are clustered using a vector-based machine learning algorithm.

In one implementation option, the k-means method is used as a machine learning algorithm for clustering emails.

At step 360, the letters in each cluster are classified.

In one embodiment, the emails that were not classified as spam on the client side are classified in each cluster according to the following rules: if the cluster contains more than a predetermined threshold of emails classified as spam on the client side, then all remaining emails in the cluster are classified as spam; if the cluster contains no more than a predetermined threshold of emails classified as spam on the client side, then the emails that were not classified as spam on the client side are classified using a machine learning model. Two feature vectors are sequentially passed to the input of the machine learning model for classifying the email. The two feature vectors are:

a) the feature vector of a letter in the cluster (the first vector) that was classified on the client side as a letter that does not contain spam;

b) a vector of features of a letter that was classified as a letter containing spam, where the cosine distance to the first vector must be minimal.

In one implementation option, the decision tree boosting method is used as a machine learning algorithm for classifying letters.

In a particular implementation variant, the two feature vectors are the feature vector of a letter in a cluster that was classified on the client side as a non-spam letter, and the feature vector of the cluster centroid.

In another embodiment, step 360 classifies all emails in each cluster based on the percentage of emails classified on the client side as containing spam, according to the following rules:

• if the percentage of emails classified on the client side 170 as containing spam is between 0 and the first boundary inclusive, all emails in the cluster are classified as not containing spam;

• if the percentage of emails classified on the client side 170 as emails containing spam is in the range from the first boundary to the second boundary inclusive, all emails in the cluster classified on the client side 170 as emails containing spam are classified using the machine learning model;

• if the percentage of emails classified on the client side 170 as emails containing spam is between the second boundary and the third boundary, all emails in the cluster are classified in accordance with the classification on the client side 170;

• if the percentage of emails classified on the client side as spam is between the third boundary and the fourth boundary, all emails in the cluster classified on the client side as non-spam are classified using the machine learning model;

• if the percentage of emails classified on the client side 170 as containing spam is between the fourth boundary inclusive and 100 percent, all emails in the cluster are classified as containing spam.

In a particular implementation, if the percentage of emails classified on the client side 170 as spam is equal to the value corresponding to the first boundary, all emails in the cluster are classified using a machine learning model.

In a particular embodiment, if the percentage of emails classified on the client side 170 as emails containing spam is equal to a value corresponding to the second or third boundary, all emails in the cluster are classified in accordance with the classification on the client side 170.

In a particular implementation, if the percentage of emails classified on the client side 170 as spam is equal to the value corresponding to the fourth boundary, all emails in the cluster are classified using a machine learning model.

In another particular embodiment, the values of the boundaries are set depending on the time of receipt of letters by client 170. For example, when classifying letters received by client 170 from 11:00 to 12:00, the boundaries are equal to 5%, 10%, 90% and 95%, and when classifying letters received by client 170 from 22:00 to 23:00, the boundaries are equal to 10%, 35%, 65% and 90%.

Fig. 4 shows a computer system on which various embodiments of the systems and methods disclosed in this document can be implemented. The computer system 20 can be a system configured to implement the present invention and can be in the form of a single computing device or in the form of multiple computing devices, such as a desktop computer, a portable computer, a notebook, a mobile computing device, a smartphone, a tablet computer, a server, a mainframe, an embedded device, and other forms of computing devices.

As shown in Fig. 4, the computer system 20 includes: a central processor 21, a system memory 22 and a system bus 23 that connects various system components, including memory associated with the central processor 21. The system bus 23 is implemented as any bus structure known in the art, which in turn contains a bus memory or a bus memory controller, a peripheral bus and a local bus capable of interacting with any other bus architecture. Examples of buses are: PCI, ISA, PCI-Express, HyperTransport™, InfiniBand™, Serial ATA, I2C and other suitable connections between components of the computer system 20. The central processor 21 includes one or more processors having one or more cores. The central processor 21 executes one or more sets of machine-readable instructions that implement the methods presented in this document. The system memory 22 may be any memory for storing data and/or computer programs executed by the central processor 21. The system memory may contain both a read-only memory (ROM) 24 and a random access memory (RAM) 25. The main input/output system (BIOS) 26 contains the main procedures that ensure the transfer of information between elements of the computer system 20, for example, at the time of loading the operating system using the ROM 24.

The computer system 20 includes one or more data storage devices, such as one or more removable storage devices 27, one or more non-removable storage devices 28, or combinations of removable and non-removable devices. One or more removable storage devices 27 and/or non-removable storage devices 28 are connected to the system bus 23 via the interface 32. In one embodiment, the removable storage devices 27 and the corresponding computer-readable storage media are non-volatile modules for storing computer instructions, data structures, program modules and other data of the computer system 20. The system memory 22, the removable storage devices 27 and the non-removable storage devices 28 can use various computer-readable storage media. Examples of computer-readable storage media include computer memory such as cache memory, SRAM, DRAM, Z-RAM, T-RAM, eDRAM, EDO RAM, DDR RAM, EEPROM, NRAM, RRAM, SONOS, PRAM; flash memory or other memory technologies such as solid-state drives (SSD) or flash drives; magnetic cassettes, magnetic tapes, and magnetic disks such as hard disks or floppy disks; optical media such as compact discs (CD-ROM) or digital versatile discs (DVD); and any other media that can be used to store the desired data and that can be accessed by the computer system 20.

The system memory 22, the removable storage devices 27 and the non-removable storage devices 28 contained in the computer system 20 are used to store the operating system 35, applications 37, other program modules 38 and program data 39. The computer system 20 includes a peripheral interface 46 for transmitting data from input devices 40, such as a keyboard, mouse, stylus, game controller, voice input device, touch input device, or other peripheral devices, such as a printer or scanner through one or more input/output ports, such as a serial port, a parallel port, a universal serial bus (USB) or another peripheral interface. A display device 47, such as one or more monitors, projectors or built-in displays, is also connected to the system bus 23 through an output interface 48, such as a video adapter. In addition to the display devices 47, the computer system 20 is equipped with other peripheral output devices (not shown in Fig. 4), such as speakers and other audiovisual devices.

The computer system 20 may operate in a network environment using a network connection with one or more remote computers 49. The remote computer (or computers) 49 is a working personal computer or a server that includes most or all of the components mentioned earlier in describing the nature of the computer system 20 shown in Fig. 7. Other devices may also be present in the network environment, such as routers, network stations or other network nodes. The computer system 20 may include one or more network interfaces 51 or network adapters for communicating with the remote computers 49 via one or more networks such as a local area network (LAN) 50, a wide area network (WAN), an intranet and the Internet. Examples of a network interface 51 are an Ethernet interface, a Frame Relay interface, a SONET interface and wireless interfaces.

Embodiments of the present disclosure may be a system, a method, or a computer-readable storage medium (or carrier).

A computer-readable storage medium is a tangible device that stores and stores program code in the form of computer-readable instructions or data structures that are accessible to the central processor 21 of the computer system 20. The computer-readable storage medium may be an electronic, magnetic, optical, electromagnetic, semiconductor memory device, or any suitable combination thereof. As an example, such a computer-readable storage medium may include random access memory (RAM), read-only memory (ROM), EEPROM, a portable compact disc with read-only memory (CD-ROM), a digital versatile disk (DVD), flash memory, a hard disk, a portable computer diskette, a memory card, a floppy disk, or even a mechanically encoded device such as punched cards or relief structures with instructions recorded thereon.

The system and method of the present invention can be considered in terms of means. The term "means" as used herein refers to an actual device, component or group of components implemented by hardware, such as an application-specific integrated circuit (ASIC) or FPGA, or by a combination of hardware and software, such as a microprocessor system and a set of machine-readable instructions for implementing the functionality of the means, which (when executed) transform the microprocessor system into a special-purpose device. The means can also be implemented as a combination of these two components, where some functions can be implemented only by hardware, and other functions - by a combination of hardware and software. In some embodiments, at least a portion, and in some cases all of the means, can be executed on the central processor 21 of the computer system 20. Accordingly, each means can be implemented in various suitable configurations and should not be limited to any particular embodiment given herein.

Finally, it should be noted that the information provided in the description is examples that do not limit the scope of the present invention defined by the claims. It will be clear to one skilled in the art that in developing any actual embodiment of the present invention, many decisions specific to a particular embodiment must be made to achieve specific goals, and these specific goals will be different for different embodiments. It is understood that such development efforts can be complex and time-consuming, but nevertheless, they will be a routine engineering task for those of ordinary skill in the art, using the present disclosure.

Claims (93)

1. A method for classifying email messages to detect spam based on email statistics, comprising the steps of: receive statistical data of customer letters and the result of letter classification on the client side; exclude from further analysis data from letters that relate to at least one of: letters that are automatic responses; letters related to the internal correspondence of the organization; generate features based on statistical data for non-excluded letters; combine features into a single vector for each letter; carry out clustering of letters using a machine learning algorithm based on the letter feature vectors; classify letters in each cluster according to the following rules: if a cluster contains more than a pre-defined threshold of emails classified on the client side as spam, classify all emails in the cluster as spam; If the cluster contains no more than a pre-defined threshold of emails classified as spam on the client side, classify the emails that were not classified as spam on the client side using a machine learning model. 2. The method according to claim 1, wherein the statistical data of letters contains at least one of: data on the rules that worked for writing; writing category; sequence of letter part formats; type of letter attachments; size of attachments in the letter; type of divisions between parts of the message; letter format; domains from email links; sequence of technical headings in the letter; sender domain from the "From" header; checksum of the username from the "From" header; IP address of the letter sender; letter body size; domain from "message-id"; header type "message-id"; sender domain from SMTP session; links from the letter; number of links in the letter; header "User-Agent"; "X-Mailer" header. 3. The method according to paragraph 2, wherein the size of the letter attachments is represented as a vector of attachment sizes. 4. The method according to paragraph 2, in which the size of the letter attachments is presented as the sum of the sizes of all attachments. 5. The method according to paragraph 1, in which all statistical data are divided into two types: numerical and categorical. 6. The method according to item 1, in which each feature is formed using one of the following methods: OneHotEncoding; min-max scaling; hashing followed by min-max scaling; CountVectorizer. 7. The method according to item 6, in which the MurmurHash3 method is used for hashing. 8. The method according to item 1, wherein one of the following methods is used as a machine learning algorithm for clustering letters: k-means; k-median; DBSCAN. 9. The method according to item 1, in which the clustering of letters is performed each time after the end of a specified period of time. 10. The method according to claim 1, wherein two feature vectors are transmitted to the input of the machine learning model for classifying writing. 11. The method according to item 10, wherein the two feature vectors are: vector of features of a letter in the cluster (the first vector) that was classified on the client side as a non-spam letter; a vector of features of a letter that was classified on the client side as a letter containing spam, where the cosine distance to the first vector must be minimal. 12. The method according to item 10, wherein the two feature vectors are a feature vector of a letter in a cluster that was classified on the client side as a letter that does not contain spam, and a feature vector of the cluster centroid. 13. A system for classifying e-mail messages for detecting spam based on statistical data of messages, consisting of at least one computer system and containing: a data processing facility that interacts with a data collection and storage device and a clustering facility and is designed to: obtaining statistical data of letters and the result of letter classification on the client side from the data collection and storage device; filtering emails based on email statistics; formation of letter characteristics based on statistical data of letters; a clustering tool that interacts with a data processing tool and a cluster analysis tool and is designed to: clustering of letters based on the generated characteristics of letters received from the data processing tool; transferring email clusters to a cluster analysis tool; a cluster analysis tool designed to classify the messages in each received message cluster according to the following rules: if a cluster contains more than a pre-defined threshold of emails classified on the client side as spam, classify all emails in the cluster as spam; If the cluster contains no more than a pre-defined threshold of emails classified as spam on the client side, classify the emails that were not classified as spam on the client side using a machine learning model. 14. The system of claim 13, wherein the statistical data of letters comprises at least one of: data on the rules that worked for writing; writing category; sequence of letter part formats; type of letter attachments; size of attachments in the letter; type of divisions between parts of the message; letter format; domains from email links; sequence of technical headings in the letter; sender domain from the "From" header; checksum of the username from the "From" header; IP address of the letter sender; letter body size; domain from "message-id"; header type "message-id"; sender domain from SMTP session; links from the letter; number of links in the letter; header "User-Agent"; "X-Mailer" header. 15. The system of claim 13, wherein the data processing means generates each feature using one of the following methods: OneHotEncoding; min-max scaling; hashing followed by min-max scaling; CountVectorizer. 16. The system according to claim 13, wherein, as a machine learning algorithm for clustering, the clustering tool implements one of the following methods: k-means; k-median; DBSCAN. 17. The system of claim 13, wherein the clustering means clusters the letters each time after a specified period of time has elapsed.