RU2775591C2 - Method and system for detecting abnormal crowdsourcing label - Google Patents

Abstract

FIELD: abnormal crowdsourcing label detection systems.

SUBSTANCE: invention relates to the field of defining an abnormal crowdsourcing label, in particular to methods and systems for detecting the source of an abnormal crowdsourcing label. The effect of the proposed technical solution is achieved by the fact that the claimed method provides for the possibility of analyzing part of the set of crowdsourcing labels; identifying an anomalous subset of crowdsourced labels; getting browsing history; dividing browsing history into groups; forming a set of web resources based on the analysis of groups and matching the web resource with the source of the external triggering event.

EFFECT: optimization of network traffic and reduction of load on network servers.

23 cl, 16 dwg, 2 tbl

Description

The technical field to which the invention belongs

The present technology relates generally to methods and systems for detecting an anomalous crowdsourcing tag, and more particularly to methods and systems for detecting the source of an anomalous crowdsourcing tag.

State of the art

The simplification of the use of the Internet has provided users with access to various web application services using digital platforms. Such web application services may be related to publications (for example, music, books, films, etc.), services (for example, airlines, hotels, restaurants, etc.), goods (for example, electronic equipment, automobiles, kitchen utensils, etc.), etc. Some web application services provide users with the ability to post content corresponding to those web application services in the form of ratings, tags, upvotes, downvotes, comments, votes, and the like. For example, in a video content application web service, users can generate "labels" for the various movies available for download or streaming from the service.

Service providers in such web application services may rank digital items at least in part based on labels generated by users. In some cases, some of the labels generated by users may contain malicious, false, misleading, or erroneous labels. Some of the malicious labels can be generated by bots. Other labels may be generated by users hired for such purposes (ie, rewarded).

However, some labels may not be malicious, but may still be "anomalous" or "erroneous". In general, the appearance of such labels is triggered by an external event. For example, it's possible that a blogger posted a bad review about a movie that went viral, and users tend to tag the movie based on that bad review without actually watching the movie.

There are several computerized ways to detect and filter malicious labels. For example, US Pat. No. 9,479,516 B2 describes various ways to distinguish between malicious data submission (eg, ratings, comments, reviews, etc.) and legitimate data submission, eg, messages from real users of an application. These methods can be used to generate intermediate signals that may indicate maliciously sending data. One or more intermediate signals can be automatically combined to form a conclusion about the detection of malicious intent in the sent data. Various options are described (for example, automatically or manually) after malicious intent is detected in the submitted data, for example, maliciously submitted data may be ignored, or penalties may be applied to the person or account associated with the maliciously submitted data. In this case, the descriptions should be interpreted broadly, covering other services that accept ratings and / or user comments.

US Pat. No. 1,0089,660 B2 describes multiple review sources for the same product or service (eg, hotels, restaurants, clinics, hairdressers, etc.) used to generate a valid rating. This rating allows you to clearly identify hotels with signs of review manipulation, silence, and forgery, and provides the user with a complete understanding of product or organization reviews. When calculating the score, three types of information are used: spatial, temporal and network (graph-based). The information is mixed to form a representative set of features that ensures the reliable formation of a reliable estimate. The invention self-adapts to new reviews and websites. To ensure the reliability and validity of the assessment, the invention also provides a crowdsourced verification mechanism by generating falsified reviews.

Disclosure of invention

The developed embodiments of the present technology are based on the understanding by the developers of at least one technical problem associated with known solutions.

For example, although known solutions have certain advantages and contribute to the detection and filtering of malicious labels corresponding to various application web services, they still have some disadvantages. For example, while malicious label detection and filtering approaches are well-known, anomalous label detection and filtering and real-time detection of a web source that encourages users to generate false labels are still relevant.

This technology was developed after the developers determined through observation that a web source can generate a flash mob that causes the appearance of erroneous labels initiated or influenced by this web source. From the point of view of a server that has to process a lot of erroneous markup operations, such a flash mob can lead to a significant waste of resources.

When developing this technology, the developers paid attention to the fact that more and more application web services available using a browser application are served by web analytics services (such as Google Analytics ™, Yandex.Metrica ™, etc.). In short, these web analytics services are capable of collecting and storing data related to a browser application.

Without wishing to be bound by any particular theory, embodiments of the present technology are developed on the assumption that scoring the browsing/navigation history of tag-forming users more reliably determines when a tag is or will be in error.

According to the first aspect of the present technology, a method for determining an abnormal crowdsourcing label for a digital item available on a digital platform is implemented. The abnormal crowdsourced tag is a tag from a plurality of crowdsourced tags assigned to a digital item by multiple users. A set of crowdsourced tags is associated with a typical distribution of tags within it, and an anomalous crowdsourced tag is triggered by an external triggering event that references a digital element, but is not associated with this digital element. The method is performed by a server storing a web browsing log associated with at least some users of the plurality of users, and includes: parsing a portion of the plurality of crowdsourced tags collected over a period of time; determining an anomalous subset of crowdsourced tags potentially triggered by the occurrence of an external trigger event during that time period; obtaining from the web browsing log a browsing history associated with a subset of the plurality of users forming part of the plurality of crowdsourced tags; dividing a browsing history associated with a subset of the plurality of users into a first browsing history group associated with the abnormal subset of crowdsourced tags and a second browsing history group associated with the remaining tags of a portion of the crowdsourced tags set; generating a delta set of web resources based on the analysis of the first group of browsing history and the second group of browsing history for differences in web resources visited by the corresponding subset of the set of users, while the delta set contains at least one web resource from the first set search history associated with the anomalous nature of visits; and associating the at least one web resource with the source of the external trigger.

In some embodiments of the invention, the method further includes determining the users of the plurality of users accessing the at least one web resource during said time period.

In some embodiments of the invention, the method further includes deleting at least some of the labels from the plurality of labels generated by users accessing the at least one web resource during said period of time.

In some embodiments of the invention, the method further includes assigning a decreasing weight to at least some of the labels generated by users accessing the at least one web resource during said period of time.

In some embodiments of the invention, the method in the future time period after the considered time period further includes: determining a subset of users from the set of future label-sending users in the future time period who accessed the at least one web resource before the future time period; and removing labels generated by a subset of users from the set of future users or assigning a decreasing weight to labels generated by this subset of users from the set of future users.

In some embodiments of the method, determining the anomalous subset includes analyzing peak trends among all neighboring labels in the actual label distribution in a portion of the set of crowdsourced labels based on peak trends that do not match the typical label distribution defining the anomalous subset.

In some method embodiments, peak trends not corresponding to a typical label distribution are based on one set of two adjacent labels associated with an abnormal peak and/or two sets of consecutive labels associated with an abnormal peak.

In some embodiments of the method, one set of two adjacent labels contains a low level label or a high level label.

In some embodiments of the method, two sets of successive adjacent labels contain low-level labels or high-level labels.

In some embodiments of the method, it further includes determining the anomalous nature of the visits.

In some embodiments of the method, determining the anomalous nature of visits includes: calculating a first proportion of users associated with a first group of browsing history who accessed at least one web resource; calculating a second proportion of users associated with the second browsing history group and accessing the at least one web resource; determining that the at least one web resource is associated with abnormal visit patterns if the first fraction is greater than the second fraction.

In some embodiments, the method further includes comparing the first beat and the second beat using a Z-statistic.

In some embodiments of the method, if the Z-statistic indicates a statistically significant difference, then that at least one web resource is determined to be the source of the external trigger.

In some embodiments of the method, it further includes ordering the at least one web resource in a ranked list using the proportion of the plurality of users who have viewed a particular web resource from among the at least one web resource.

In some embodiments, the method further includes selecting an external trigger source subset from the at least one web resource, using the absolute number of users in the set of users who have viewed the web resource as a cutoff.

In some embodiments of the method, at least one web resource contains a plurality of web resources, each of which contains a corresponding version of the external trigger.

In some embodiments of the method, the digital element contains content, and the external trigger is a feedback digital element containing feedback about the content of that digital element.

In some embodiments of the method, the digital feedback element is generated by an entity other than the source of the digital element.

In some embodiments of the invention, the method at some future point in time further includes analyzing the user's browsing history for the presence of an external trigger source and taking corrective action if such an external trigger source is present.

In some embodiments, the method further includes, if there is an external trigger source, parsing the crowdsourced label generated by the user to verify that the external trigger source is the actual source of the external trigger.

In some embodiments of the method, the external trigger source is a plurality of external trigger sources, and the method further includes selecting the N sources with the highest rank from the plurality of sources and training a Machine Learning Algorithm (MLA) using the N sources with the highest rank to predict the source of the external trigger with the highest rank.

In some embodiments of the method, the anomalous pattern of visits corresponds to at least one resource not present in the second browsing history.

According to the second aspect of the present technology, a system is implemented for determining an abnormal crowdsourcing label for a digital item available on a digital platform. The abnormal crowdsourced tag is a tag from a plurality of crowdsourced tags assigned to this digital item by multiple users. A set of crowdsourced tags is associated with a typical distribution of tags within it, and an anomalous crowdsourced tag is triggered by an external triggering event that references a digital element, but is not associated with this digital element. The system comprises: a server storing a web browsing log associated with at least some users of a plurality of users, a machine-readable physical storage medium containing instructions, and a processor that, when executing the instructions, is capable of: parsing a portion of a plurality of crowdsourced tags collected over a period of time; determine an anomalous subset of crowdsourced tags potentially triggered by the occurrence of an external trigger event during that time period; obtain from the web browsing history a browsing history associated with a subset of the plurality of users forming part of the plurality of crowdsourced tags; separating a browsing history associated with a subset of the plurality of users into a first browsing history group associated with the abnormal subset of the crowdsourced tags and a second browsing history group associated with the remaining tags of a portion of the crowdsourced tags set; generate a delta set of web resources based on the analysis of the first group of browsing history and the second group of browsing history for differences in web resources visited by the corresponding subset of the set of users, while the delta set contains at least one web resource from the first set search history associated with the anomalous nature of visits; and map that at least one web resource to the source of the external trigger.

As used herein, the term "server" means a computer program executing in appropriate hardware and capable of receiving requests (eg, from electronic devices) over a network and executing or initiating those requests. The hardware may be one physical computer or one computer system, which is not essential to the present technology. In the present context, the expression "at least one server" does not mean that every task (for example, a received command or request) or some specific task is received, executed or started by the same server (i.e. the same software and / or hardware). This expression means that any number of software or hardware can receive, send, execute or initiate the execution of any task or request or the results of any tasks or requests. All of these software and hardware may be a single server or multiple servers, both of which are meant by the expression "at least one server".

In the context of the present description, the term "client device" means any computer hardware capable of executing programs suitable for solving the task. In the context of the present description, the term "client device" is generally associated with the user of the client device. Thus, some (non-limiting) examples of client devices include personal computers (desktops, laptops, netbooks, and the like), smartphones and tablets, and network equipment such as routers, switches, and gateways. It should be noted that in this context, a device functioning as a client device may also function as a server for other client devices. The use of the term "client device" does not preclude the use of multiple client devices to receive, send, perform or initiate the execution of any task or request, or the results of any tasks or requests or steps of any method described herein.

In the context of the present description, unless explicitly stated otherwise, the numerals "first", "second", "third", etc. are used only to indicate the difference between the nouns they refer to, but not to describe any specific relationship between these nouns. For example, it should be clear that the use of the terms "first server" and "third server" does not imply any particular order, type, chronology, hierarchy, or classification of, in this case, servers, and that their use (by itself) does not imply the mandatory presence of a "second server" in every situation. In addition, as occurs herein in another context, reference to a "first" element and a "second" element does not exclude that the two elements may in fact be the same element. Thus, for example, in some cases the "first" server and the "second" server may be the same software and/or hardware, and in other cases° different software and/or hardware.

In the context of the present description, the expression "information" includes information of any kind or kind that can be stored in a database. Thus, information includes audiovisual works (images, films, sound recordings, presentations, etc.), data (location data, numerical data, etc.), text (opinions, comments, questions, messages, etc.). .d.), but not limited to documents, spreadsheets, etc.

In the context of the present description, the expression "document" should be interpreted broadly to include any machine-readable or machine-storable work product. A document may contain an email message, a website, a file, a combination of files, one or more files with embedded links to other files, a newsgroup post, a blog, an Internet advertisement, and so on. In the context of the Internet, a common document is a web page. Web pages often contain textual information and may contain embedded information (such as metadata, images, hyperlinks, etc.) and/or embedded commands (such as Javascript commands, etc.). A page can correspond to a document or part of a document. Thus, the words "page" and "document" in some cases can be used interchangeably. In other cases, a page may correspond to a part of a document, such as a subdocument. A page can also correspond to multiple documents.

In the context of this description, unless expressly stated otherwise, the term "database" means any structured collection of data, regardless of its specific structure, database management software or computer hardware for storing this data, using it or making it available to others. way. The database may reside on the same hardware as the process for storing or using the information stored in the database, or the database may reside on separate hardware such as a dedicated server or multiple servers.

Each embodiment of the present technology relates to at least one of the above objectives and/or aspects, but not necessarily all of them. It should be understood that some aspects of the present technology, associated with an attempt to achieve the above goal, may not meet this goal and/or may meet other goals not explicitly mentioned here.

Additional and/or alternative features, aspects and advantages of embodiments of the present technology are contained in the following description, in the accompanying drawings and in the claims.

Brief description of the drawings

The following description is provided for a better understanding of the present technology, as well as other aspects and their features, and should be used in conjunction with the attached drawings.

In FIG. 1 depicts various components and features of a client device implemented in accordance with various non-limiting embodiments of the present technology.

In FIG. 2 is a diagram of a system implemented in accordance with various non-limiting embodiments of the present technology.

In FIG. 3 shows web application services provided by a service provider and implemented in the system shown in FIG. 2 according to various non-limiting embodiments of the present technology.

In FIG. 4 is a block diagram containing components, modules, connections, and interactions for an example application service according to various non-limiting embodiments of the present technology.

In FIG. 5 shows an example of a typical label distribution corresponding to crowdsourced labels and multiple users who sent those crowdsourced labels, according to various non-limiting embodiments of the present technology.

In FIG. 6-7 show examples of actual label distribution where crowdsourced labels contain abnormal crowdsourced labels according to various non-limiting embodiments of the present technology.

In FIG. 8 is a detailed high-level functional block diagram of an abnormal crowdsourcing tag processor according to various non-limiting embodiments of the present technology.

In FIG. 9 is a detailed high-level functional block diagram of an initial label parser according to various non-limiting embodiments of the present technology.

In FIG. 10 is a detailed high-level functional block diagram of a browsing history analyzer according to various non-limiting embodiments of the present technology.

In FIG. 11 shows the structure of browsing history data stored in a tracking server according to various non-limiting embodiments of the present technology.

In FIG. 12 shows a first browsing history group associated with users generating abnormal crowdsourced tags according to various non-limiting embodiments of the present technology.

In FIG. 13 shows a second browsing history group associated with remaining users corresponding to a portion of a plurality of crowdsourced tags, according to various non-limiting embodiments of the present technology.

In FIG. 14 is a delta set containing at least one web resource from a first set of browsing history associated with anomalous browsing patterns, according to various non-limiting embodiments of the present technology.

In FIG. 15 is a ranked list generated by the anomalous crowdsourcing label processor and containing a set of Uniform Resource Locators (URLs) according to various non-limiting embodiments of the present technology.

In FIG. 16 is a flowchart of a method for detecting an external trigger according to various non-limiting embodiments of the present technology.

Implementation of the invention

The examples and conventions presented here are intended to provide a better understanding of the principles of the present technology, and not to limit its scope to such specifically given examples and conditions. It is obvious that specialists in the art are able to develop various methods and devices that are not explicitly described or shown, but implement the principles of the present technology within its essence and scope.

In addition, to facilitate a better understanding, the following description may include simplified implementations of the present technology. Those skilled in the art will appreciate that various embodiments of the present technology can be significantly more complex.

In some cases, useful examples of modifications to the present technology are provided. They contribute to understanding, but also do not define the scope or boundaries of the present technology. The presented list of modifications is not exhaustive and the person skilled in the art can develop other modifications within the scope of this technology. In addition, if modifications are not described in some cases, this does not mean that they are impossible and / or that the description contains the only possible implementation of one or another element of the present technology.

Moreover, the description of principles, aspects, and embodiments of the present technology, as well as specific examples thereof, is intended to cover their structural and functional equivalents, whether they are currently known or will be developed in the future. For example, it should be apparent to those skilled in the art that any structural diagrams described herein correspond to conceptual representations of illustrative circuit diagrams that implement the principles of the present technology. It should also be obvious that any flowcharts, process diagrams, state transition diagrams, pseudocodes, etc. correspond to various processes that may be represented on a computer-readable physical storage medium and may be executed by a computer or processor, whether such computer or processor is explicitly shown or not.

The functions of the various elements shown in the drawings, including any functional unit labeled "processor" or "graphics processing unit", may be implemented using dedicated hardware as well as using hardware capable of executing the associated software. If a processor is used, these functions may be performed by a single dedicated processor, a single shared processor, or multiple individual processors, some of which may be shared. In some embodiments of the present technology, the processor may be a general purpose processor such as a central processing unit (CPU) or a specialized processor such as a graphics processing unit (GPU). In addition, explicit use of the term "processor" or "controller" should not be construed as referring solely to the hardware capable of executing the software and may refer to, but is not limited to, digital signal processor (DSP) hardware, network processor, ASIC (ASIC), Field Programmable Gate Array (FPGA), Read Only Memory (ROM) for storing software, Random Access Memory (RAM) and Non-Volatile Memory (ROM). Other general purpose and/or custom hardware may also be contemplated.

Software modules, or simply modules that are intended to be implemented in software, may be represented here as any combination of flowchart elements or other elements indicating the steps of a process and/or containing a textual description. Such modules may be implemented in hardware, as shown or implied.

In the following, in light of the principles set forth above, some non-limiting examples are provided to illustrate various embodiments of aspects of the present technology.

In FIG. 1 depicts various components and features of a client device 100 implemented in accordance with various non-limiting embodiments of the present technology. As shown, the client device 100 uses a processor 102, a graphics processor 104, a solid state drive 106, a memory 108, a display 110, and a network module 112. It should be understood that other elements may be present not shown for simplicity and ease of interpretation.

Communication between the various components of client device 100 may be via one or more internal and/or external buses (not shown), such as a PCI bus, Universal Serial Bus, IEEE 1394 FireWire bus, SCSI bus, Serial-ATA bus, etc. . to which various hardware components are connected electronically.

In accordance with embodiments of the present technology, the solid state drive 106 stores software instructions suitable for loading into memory 108 and execution by processor 102 and/or graphics processor 104. The software instructions may, for example, be included in a library or application. For example, memory 108 may be implemented as a computer-readable storage medium, including ROM, hard disk drives (HDDs), solid state drives (SSDs), and flash memory cards.

Client device 100 may be a server, desktop computer, tablet, smartphone, personal digital assistant, or any device that can be configured to implement the present technology, as one of skill in the art would understand.

In FIG. 2 depicts system 200 implemented in accordance with various non-limiting embodiments of the present technology. As shown, system 200 includes a plurality of 202 users associated with a plurality of 210 client devices. In FIG. 2 shows a first user 204, a second user 206, and a third user 208, respectively associated with the first client device 214, the second client device 216, and the third client device 218. In some embodiments of the present technology, the first client device 214 may be implemented similar to the client device 100, the second client device 216 may be implemented as a laptop, and the third client device 218 may be implemented as a smartphone. It should be noted that the association of the plurality of users 202 with the plurality of client devices 210 is not intended to suggest or imply any mode of operation, such as logging in, registering, or the like.

A plurality of 210 client devices are connected to the communication network 220 by respective communication lines 222 . It should be understood that communication network 220 may be implemented using any suitable technology such as the Internet, wide area network, local area network, private network, and the like.

The implementation of link 222 is not particularly limited, but depends on the implementation of the first client device 214, the second client device 216, and the third client device 218. By way of non-limiting example, in those embodiments of the present technology where the first client device 214 and/or second client device 216 and/or third client device 218 is implemented as a wireless communication device (such as a smartphone), communication link 222 may be implemented as a wireless communication link (such as 3G network link, network link 4G, Wireless Fidelity or WiFi ^® , Bluetooth ^® for short, etc.). In those instances where the first client device 214 and/or the second client device 216 and/or the third client device 218 is implemented as a laptop, the communication link 222 can be either wireless (such as Wireless Fidelity or ^WiFi® , Bluetooth® for ^short , etc.) etc.), or wired (such as an Ethernet-based connection).

It should also be understood that the implementations of the first client device 214, the second client device 216, the third client device 218, the communication network 220, and the communication line 222 are for illustration only. Other specific implementation details of the first client device 214, the second client device 216, the third client device 218, the communications network 220, and the communications link 222 should be apparent to those skilled in the art. The above examples do not limit the scope of the present technology in any way.

The communications network 220 is also connected to a plurality of 224 servers. In FIG. 2 shows a first application server 226, a second application server 228, a third application server 230, and a tracking server 232. The first application server 226, the second application server 228, the third application server 230, and the tracking server 232 may be implemented as conventional computer servers. In some embodiments of the present technology, the first application server 226, the second application server 228, the third application server 230, and the tracking server 232 may be implemented as a Dell™ PowerEdge™ server running a Microsoft™ Windows Server™ operating system. For example, a plurality of servers may include a computer-readable physical storage medium, including ROM and/or RAM, and one or more persistent storage devices such as hard disk drives (HDDs), solid state drives (SSDs), and flash memory cards.

Obviously, the first application server 226, the second application server 228, the third application server 230, and the tracking server 232 may be implemented using any other suitable hardware and/or application software and/or firmware, or combinations thereof. In addition, the first application server 226, the second application server 228, the third application server 230, and the tracking server 232 may be implemented in various hardware or software.

In the illustrated non-limiting embodiment of the present technology, the first application server 226, the second application server 228, the third application server 230, and the tracking server 232 are separate servers. In alternative non-limiting embodiments of the present technology, the functions of each server among the first application server 226, the second application server 228, the third application server 230, and the tracking server 232 may be implemented in a single server or may be distributed and implemented in multiple servers (not shown).

In FIG. 3 shows the web application services provided by the web application service provider 304 and implemented in the system shown in FIG. 2 according to various non-limiting embodiments of the present technology. The web application service provider 304 may provide a variety of services, referred to as web application services or application services, to a plurality 202 of users on the Internet. Examples of service providers include online services Yandex™, Google™, Yahoo™, etc.

In general, a user, such as user 204, may register a user account 302 with a web application service provider 304 using their first client device 214 and access a plurality of services, such as social media application services 306 hosted on the first server 226 applications, multimedia application services 308 hosted on the second application server 228, and other application services 310 hosted on the third application server 230 using a browser application (by accessing a website associated with the service) or a standalone application that can be executed in the first client device 214.

It is contemplated that the social media application services 306 may include various social media platforms such as the online services Facebook™, Twitter™, Instagram™, and the like.

The multimedia application services 308 may include various multimedia platforms such as the Youtube™ online service, and the other application services 310 may include any other available web application services such as Amazon™, IMDb™, Playstore™ online services. , App Store™, etc.

Web application service provider 304 as well as social media application services 306, multimedia application services 308, and other application services 310 may be associated with the tracking server 232. In other embodiments of the invention, the first user 204 may be registered with an email service (not shown), which may be associated with a web application service provider 304 capable of automatically creating a user account 302 and associating it with social media application services 306, multimedia application services 308 and other application services 310. In other non-limiting embodiments of the invention, the first user 204 may be registered with an email service and their accounts may be manually associated with social media application services 306, multimedia application services 308, and other application services. services 310 (which may be associated with another service provider), and the tracking server 232 may be allowed to track its activities. A single user account 302 is typically used with a single sign-on application such as Yandex.Passport™ and the like.

The web application service provider 304 may manage a tracking server 232 that is capable of tracking the activities of each user (eg, of a plurality of 202 users) associated with a plurality of services. However, in other embodiments of the invention, the functions of the tracking server 232 may be implemented directly on at least one server of the plurality of servers 224, such as the first application server 226, the second application server 228, or the third application server 230, or directly on each device. from a plurality of 210 client devices, such as a first client device 214, a second client device 216, and a third client device 218.

Thus, the tracking server 232 may receive each user action performed by a user, such as user 204, on an application service, such as social media application services 306, multimedia application services 308, and other application services 310, on its first client device 214 (or on another client device that the first user 204 can use) and can log user activities. There are no restrictions on the manner in which the tracking server 232 maintains a user activity log, which is well known in the art. For example, each user action or user event received by the tracking server 232 directly from the client device 204, either from the first application server 226, from the second application server 228, or from the third application server 230, can be associated with an IP address, device ID, timestamp A that contains the date, time, and time zone, the state, the content, and the score associated with the state. In at least some non-limiting embodiments of the present technology, the tracking server 232 may anonymize the user ID to protect the identity of the respective user, such as the first user 204.

According to various non-limiting embodiments of the present technology, a plurality of users 202 may access various application services, such as social media application services 306, multimedia application services 308, and other application services 310, and in at least one application service may leave reviews or post content corresponding to digital element 402 (described below) associated with at least one application service in the form of ratings, tags, upvotes, downvotes, comments, votes, and the like.

It is contemplated that digital element 402 (described below) may correspond to any digital document or resource, such as music, books, movies, airline reservations, hotel reservations, restaurants, software, etc., available on a digital platform, for example. such as Facebook™, Twitter™, Instagram™, Youtube™, Amazon™, IMDb™, Playstore™, App Store™, etc. online services. In a non-limiting example corresponding to a video content application web service, users can generate "labels" for the various movies available for download or streaming from the service.

At least one server of the plurality of 224 servers may be able to rank the numeric element 402 (described below) based at least in part on the labels corresponding to the numeric element 402 (described below) made by the plurality 202 of users. In some cases, a portion of the labels made by multiple 202 users contain "abnormal" or "erroneous" labels. In general, according to various non-limiting embodiments of the present technology, such labels are triggered by an external event. For example, it could be that a blogger posted a bad review about a movie that went viral, and users tend to tag the movie based on that bad review without actually watching the movie.

Therefore, it is important for the efficient operation of the system 200 to detect the source of the external trigger and the marks made under the influence of this external trigger. In FIG. 4 is a block diagram of example components, modules, connections, and interactions for an example application service according to various non-limiting embodiments of the present technology. In this example, there are a plurality of 202 users associated with a plurality of 210 client devices accessing a digital element 402 of other application services 310.

A plurality of 202 users may access the digital element 402 using their respective client devices 210 via an interface associated with other application services 310. The interface may, for example, be a web page that the user may view using a web browser (eg, a browser for desktop or mobile browser). In another example, the interface may be a mobile application that runs on at least one of the plurality of 210 client devices by default. It should be understood that the information associated with the digital element 402 may be stored on the third application server 230.

Multiple users 202 may also submit reviews, testimonials, ratings, comments, tags, and the like for the corresponding digital item 402. In some non-limiting examples illustrating various cases, a plurality of 202 users may generate ratings or comments corresponding to movies, television programs, home videos, video games, streaming content, etc. in the IMDB™ online service, or generate ratings or comments related to various products (such as electronic equipment, software, video games, clothing, furniture, food, toys, jewelry, etc.) on the Amazon™ online service, etc.

It should be understood that in some embodiments of the invention, information associated with the digital element 402 may be stored on the third application server 230. Such information may include digital content (eg, video, document, music, etc.) associated with digital element 402. In addition, any reviews, reviews, ratings, comments, tags, etc. generated by multiple 202 users for the corresponding digital element 402 may also be stored on the second application server 228.

In FIG. 5 is a non-limiting example of a typical label distribution 500 corresponding to crowdsourced labels 502, corresponding to various non-limiting embodiments of the present technology, and an indication of the number 504 of crowdsourced labels 502 submitted by at least some of the user set 202. It is assumed that the typical label distribution 500 is just an example representing a distribution without any variation in the label distribution. In other words, a typical distribution of 500 tags contains no or very few tags generated by "users associated with the flash mob".

The crowdsourced tags 502 may also be some numbers selected from a predetermined range. For example, a user rating may correspond to a value from one to five stars out of five stars, or the first number (integer or decimal) in a range delimited by the second maximum integer. For example, each step (eg, one integer) of a user rating may be represented by an icon, such as a star. In some non-limiting embodiments of the present technology, a subset of user set 202 may crowdsource tags 502 for digital element 402, and based on the number 504 of crowdsourced tags 502, a rating corresponding to digital element 402 may be compiled (e.g., by averaging) to form an overall user rating. For example, a number (integer or decimal) may be selected within a predetermined range (eg, 3.5 stars out of 5 stars).

In FIG. 6-7 are non-limiting examples of actual tag distributions 600 and 700 based on the assumption that during a flash mob, the actual tag distribution may differ from the typical tag distribution, either by a single crowdsourced tag having an anomalously high value, i. a single peak, or two adjacent crowdsourced labels that have abnormally large values, i.e. group peak. In FIG. 6 shows the first case where the crowdsourced tags 502 contain one anomalous crowdsourced tag 606 and their number 604 indicates that the crowdsourced tags 502 are submitted by at least some of the user set 202. In FIG. 7 shows a second case where the crowdsourced tags 502 contain a set of two anomalous crowdsourced tags 706, and their number 704 indicates that the crowdsourced tags 502 are submitted by at least some of the user set 202.

It is hypothesized that the anomalous crowdsourcing tags 606 and 706 may have been affected by the source of an external trigger. For example, a situation is possible when a certain blogger published a scandalous review about a movie that became viral. At least some users out of the plurality of 202 users become inclined to tag a movie based on such a scandalous review without actually seeing the movie. Likewise, based on a video corresponding to a product or service (e.g., electronic equipment, hotel, restaurants, etc.) that has gone viral on a social media platform, at least some of the user set of 202 begin to generate tags without actually using the product, or service. Such anomalous or erroneous crowdsourced tagging can generate flash mobs. As shown in FIG. 4, other application services 310 may also use the abnormal crowdsourcing label processor 404. The abnormal crowdsourced tag processor 404 may receive, parse, and process the crowdsourced tags 502 generated by the plurality of 202 users. Thus, the anomalous crowdsourced tag processor 404 determines an anomalous subset among the crowdsourced tags, specifically the anomalous crowdsourced tags 606 or 706. 608 or 708) or all of the crowdsourced tags 502, the abnormal crowdsourced tag processor 404 generates a delta set 1400 (described below) containing at least one web resource that is the source of the external trigger.

According to various non-limiting embodiments of the present technology, the delta set 1400 contains at least one web resource associated with abnormal browsing patterns (ie, abnormal browsing patterns).

In some non-limiting embodiments of the present technology, the anomalous pattern of visits may indicate at least one resource not present in the second browsing history. Alternatively, an anomalous pattern of visits may indicate that the number of visits to at least one resource present in the first browsing history is disproportionately greater than the number of visits in the second browsing history. It should be noted that the exact value for "out of proportion" may be determined for a particular implementation of non-limiting embodiments of the present technology and may be set based on experimental results of non-limiting embodiments of the present technology. For example, a disproportionate excess may correspond to a tenfold excess. It should be noted that the smaller this value, the smaller the threshold should be considered for a possible source of an external triggering event.

In FIG. 8 is a detailed high-level functional block diagram of an abnormal crowdsourcing tag processor 404 according to various non-limiting embodiments of the present technology. As shown, the anomalous crowdsourcing label processor 404 uses an initial label parser 802, a browsing history parser 804, and an end label parser 806. It should be understood that other elements not shown for simplicity may be present.

According to at least one non-limiting embodiment of the present technology, the abnormal crowdsourced label processor 404 receives crowdsourced labels 502 assigned to the digital item 402 by at least some of the user set 202. In some embodiments, the crowdsourced tags 502 may be stored on the third application server 230 and the abnormal crowdsourced tag processor 404 receives the crowdsourced tags 502 from the third application server 230. In some non-limiting embodiments of the present technology, other application services 310 may provide the crowdsourced tags 502 directly to the abnormal crowdsourced tag processor 404 prior to storing them on the third application server 230. The abnormal crowdsourced label processor 404 then provides the crowdsourced labels 502 to the initial label parser 802 .

Initial label analyzer 802 may analyze peak trends among crowdsourced labels 502 to select an anomalous subset among crowdsourced labels, such as abnormal crowdsourced labels 606 or 706. FIG. 9 is a detailed high-level functional block diagram of an initial label analyzer 802 according to various non-limiting embodiments of the present technology. As shown, the initial label analyzer 802 uses a peak trend analyzer 902 and an outlier subset selector 904. It should be understood that other elements not shown for simplicity may be present.

The peak trend analyzer 902 receives the crowdsourced labels 502 and may analyze a portion of the set of crowdsourced labels 502 (eg, labels 608 or 708) in the actual label distribution (eg, in the actual label distribution 600 or 700) or all of the crowdsourced labels 502 over a period of time to detection of peaks in the concentration of labels in the actual distribution of labels in relation to neighboring labels. In some embodiments, a portion of the crowdsourced labels 502 (eg, labels 608 or 708) in an actual label allocation (such as actual label allocations 600 or 700) may be associated with low level labels or with high level labels.

Accordingly, peak trend analyzer 902 may apply a currently ending sliding time window to labels sent during a predetermined time period T. The predetermined time period T may be based on saturation of the concentration of labels sent during that time period. As a non-limiting example, consider the cases where 10 tags and 10,000 tags are sent during the day. In the first case, the value of the predetermined time period T may be given in days, which provides a larger sliding time window compared to the sliding window in the second case, where the value of the predetermined time period T may be given in hours.

In some non-limiting embodiments of the present technology to detect such peaks, the peak trend analyzer 902 may be implemented as a classifier, such as a binary classifier, capable of generating a yes or no prediction. Peak trend analyzer 902 can be trained using two targets, a single peak and a group peak. In particular, two metrics are associated with these two goals:

- concentration in one label, such as the abnormal crowdsourcing label 606, with a peak character on both sides;

- concentration in multiple tags, such as anomalous 706 crowdsourcing tags, with a peak character on both sides.

In some embodiments of the invention, in order to calculate these two features (single peak score and group peak score), peak trend analyzer 902 may calculate a difference in values (DIV, Difference In Values) of label concentrations for two adjacent labels from among crowdsourced labels 502. Then, peak trend analyzer 902 trends normalizes the DIV differences and ranks the normalized DIV differences. Peak trend analyzer 902 then selects the largest normalized difference DIV as the first feature, i. e. single peak score or (if available) selects the two largest normalized DIVs associated with successive adjacent labels as the second feature, i.e. group peak index.

Table 1 below can be seen as a non-limiting example of label distribution in the range 1-5 without the flash mob.

Table 1

Label one 2 3 four 5 Share of each label fifteen% twenty% 25% 22% fourteen% DIV difference 5% 5% 3% eight% Normalized difference DIV 27.77% 27.77% 16.67% 44.44%

Table 1 contains the labels, the proportion of each label (in percent), the DIV differences (in percent), and the normalized DIV differences (in percent). Peak trend analyzer 902 may calculate DIV differences by determining the difference for each adjacent label. Peak trend analyzer 902 then calculates normalized DIV differences over the 100% range. On the basis that the normalized DIV differences are "normal" with respect to each other, and that a particular normalized DIV difference is "abnormally large" with respect to other normalized DIV differences, the peak trend analyzer 902 is able to determine that the label distribution is a typical distribution. tags or the actual distribution of tags with a "flash mob". In this case, the distribution is defined as a typical distribution of labels (i.e. without the "flash mob").

Table 2 below can be seen as another non-limiting example of the distribution of tags in the range 1-5 with a "flash mob".

table 2

Label one 2 3 four 5 Share of each label 60% thirty% 5% 3% 2% DIV difference thirty% 25% 2% one% Normalized difference DIV 51.72% 43.10% 3.48% 1.72%

Table 2 contains the labels, the proportion of each label (in percent), the DIV differences (in percent), and the normalized DIV differences (in percent). Peak trend analyzer 902 may calculate DIV differences by determining the difference for each adjacent label. Peak trend analyzer 902 then calculates normalized DIV differences over the 100% range. In this example, there is a pair of normalized DIV differences that are anomalously larger than other normalized DIV differences. Therefore, peak trend analyzer 902 may also select the largest normalized DIV difference as the first feature (e.g., 51.72% in this case) and the two largest normalized DIV differences as the second feature (e.g., 51.72% and 43.10% in this case).

It should be noted that the exact value for "abnormal excess" can be determined for a particular implementation of non-restrictive embodiments of the present technology, and it can be set based on experimental results of non-restrictive embodiments of the present technology. For example, an abnormal excess may correspond to a tenfold excess.

Based on the peak trends analyzed by the peak trend analyzer 902 for a set of crowdsourced tags 502 (e.g., among tags 608 or 708) or among all crowdsourced tags 502, an abnormal subset selector 904 determines subsets of crowdsourced tags that do not correspond to a typical distribution of tags (e.g., a typical distribution of 500 tags).

Depending on the implementation, the mismatch of crowdsourced label subsets to a typical label distribution may be based on one set of two adjacent labels associated with an abnormal label concentration peak (single peak index) or two sets of consecutive adjacent labels associated with an abnormal concentration peak. labels (group peak index). In addition, a single peak score or a group peak score is typically associated with a low-level mark or a high-level mark, and two sets of two adjacent marks contain low-level marks or high-level marks. If the range 1-10 is used as an example, then the low label may correspond to 1 or 2, and the high label may correspond to 9 or 10. However, the exact definition of the low and high labels can be chosen by the operator of the peak trend analyzer 902 . In general, the choice of low-level or high-level labels is based on the assumption that flashmob-related labels are usually either highly positive or highly negative.

The anomalous subset of crowdsourced tags 502 determined by the anomalous subset selector 904 is provided to the browsing history analyzer 804 for further processing (see FIG. 8). Browsing history analyzer 804 may parse browsing history data 1100 (described below) associated with a subset of users 202 forming part of a set of crowdsourced tags 502 (e.g., tags 608 or 708) or all crowdsourced tags 502 and can generate a delta set 1400 (described below) containing at least one web resource that is the source of the external trigger.

In FIG. 10 is a detailed high-level functional block diagram of a browsing history analyzer 804 according to various non-limiting embodiments of the present technology. As shown, the browsing history parser 804 uses a browsing history receiver 1002, a browsing history parser 1004, and a delta set builder 1006. It should be understood that there may be other elements not shown for simplicity and ease of interpretation.

Browsing history receiver 1002 is capable of receiving browsing history data 1100 (described below) associated with a portion of users 202 that form part of a plurality of crowdsourced tags 502 (e.g., tags 608 or 708) or all crowdsourced tags 502 from a web browsing history associated with many 202 users and stored on the server 232 tracking. There are no restrictions on the manner in which the tracking server 232 maintains a user activity log, which is well known in the art. For example, each user action or user event received by the tracking server 232 directly from the plurality of client devices 210 or from the first application server 226, the second application server 228, or the third application server 230 may be associated with an IP address, with a device ID, with a mark time containing date, time and time zone, with state, with content, etc.

In FIG. 11 is a non-limiting example of browsing history data 1100 associated with a portion of users 202 forming part of a set of crowdsourced tags 502 (eg, tags 608 or 708) or all crowdsourced tags 502 as an index stored on tracking server 232. Browsing history data 1100 consists of a set of 1102 URLs and a corresponding access time 1104 for each URL in the set 1102 of URLs. The set 1102 of URLs may be sorted chronologically based on the corresponding time 1104 accessed. Although the browsing history data 1100 is shown as containing only a set of 1102 URLs and the time 1104 accessed, the scope of the invention is not limited to this and may also contain other information such as an IP address, a device ID, a timestamp containing a date , time and time zone, status, etc.

It is assumed that the URL set 1102 is only a non-limiting example based on user actions taken in various application services, such as social media application services 306 hosted on the first application server 226, multimedia application services 308 hosted on the second application server 228; and other application services 310 hosted on the third application server 230.

The browser history receiver 1002 then provides the browsing history data 1100 to the browsing history parser 1004 for further processing. The browsing history parser 1004 is capable of dividing the browsing history data 1100 into a first browsing history group 1200 (described below) associated with users generating abnormal crowdsourcing tags (e.g., abnormal crowdsourcing tags 606 or 706) and a second browsing history group 1300 (described below). ) associated with the remaining users corresponding to part of the set of crowdsourced tags 502.

In FIG. 12 shows a first browsing history group 1200 associated with users generating abnormal crowdsourced tags (eg, abnormal crowdsourced tags 606 or 706) according to various non-limiting embodiments of the present technology. The first browsing history group 1200 consists of a URL set 1202 that is a subset of the URL set 1102 and a corresponding access time 1204 for each URL in the URL set 1202 . The set 1202 of URLs is sorted chronologically based on the corresponding access time 1204 .

In FIG. 13 shows a second browsing history group 1300 associated with remaining users corresponding to a portion of a plurality of crowdsourced tags 502, according to various non-limiting embodiments of the present technology. The second browsing history group 1300 consists of a URL set 1302 that is a subset of the URL set 1102 and a corresponding access time 1304 for each URL in the URL set 1302. The set 1302 of URLs is sorted chronologically based on the corresponding access time 1304 .

The browsing history parser 1004 then provides the first browsing history group 1200 and the second browsing history group 1300 to the delta set builder 1006 for further processing. The delta set generator 1006 is capable of parsing the first browsing history group 1200 and the second browsing history group 1300 and generating a delta set 1400 (described below). The delta set 1400 is generated based on the difference between the web resources associated with the first browsing history group 1200 and the second browsing history group 1300 (URL sets 1202 and 1302) visited by a portion of the users 202 forming part of the set of crowdsourced tags 502 (e.g., labels 608 or 708) or all crowdsourced labels 502.

In FIG. 14 shows a delta set 1400 generated by the browsing history parser 1004 and containing at least one web resource from the first anomalous browsing history set according to various non-limiting embodiments of the present technology. The delta set 1400 consists of a set 1402 of URLs, which is a subset of the set 1102 of URLs, and a corresponding hit time 1404 for each URL in the set 1402 of URLs. The set 1402 of URLs is sorted chronologically based on the corresponding access time 1404 .

In some non-limiting embodiments of the present technology, URL set 1402 also contains a URL subset 1406 and a URL subset 1408. The URL subset 1406 may be associated with a web resource from the first browsing history group 1200 not present in the second browsing history group 1300, and contain at least one web resource corresponding to the source of the external trigger (eg, www.aaaa.com or www.cccc.com or both). In some embodiments, the URL subset 1408 may be associated with web resources present in the first browsing history group 1200 and in the second browsing history group 1300 (for example, from or to www.llll.com or www.mmmm.com both). However, the proportion of users associated with the first browsing history group 1200 and accessing the subset 1408 of URLs is greater than the proportion of users associated with the second browsing history group 1300 accessing the subset 1408 of URLs.

It is contemplated that at least one web resource, such as a subset of URLs 1406 or 1408, may contain a plurality of web resources, each containing a corresponding version of an external trigger (eg, blogs, social media posts, news, etc.). .p.), and each external trigger may be a feedback about that content of the digital element 402. In addition, the feedback associated with the digital element 402 may be generated by an entity other than the digital element 402. Such entities may be bloggers, browsers, spammers, or any random users not associated with digital element 402.

In some non-limiting embodiments of the present technology, the delta set generator 1006 also parses the URL set 1102 associated with browsing history data 1100. For each URL in the URL set 1102, the delta set builder 1006 can calculate the number of users accessing that address. In addition, using Z-statistics indicating a statistically significant difference in the number of users associated with the first browsing history group 1200 accessing a URL from the URL set 1102 and the number of users associated with the second browsing history group 1300 and accessing the same URL from a set of 1102 URLs, the delta set builder 1006 may determine the URLs (eg, www.gggg.com or www.kkkk.com) as corresponding to at least one web resource, representing the source of the external triggering event, and store them in the delta set 1400 as a subset 1408 of URLs.

Browsing history analyzer 804 provides delta set 1400 to final label parser 806 for further processing (see FIG. 8). End label parser 806 may determine users from a set of 202 users accessing at least one web resource during a predetermined time period T. In some embodiments, end label parser 806 may remove some labels or at least assign a decreasing weight for some crowdsourced labels 502 generated by users from a set of 202 users accessing this at least one web resource during a predetermined period of time T. Thus, the final label parser 806 contributes to the credibility of the crowdsourced labels 502 by the anomalous crowdsourced label processor 404.

In some non-limiting embodiments of the present technology, the anomalous crowdsourcing tag processor 404 may communicate with the third application server 230 and with the tracking server 232 to determine a subset of the set of 202 users who have already accessed at least one web resource, but did not generate a corresponding label for the digital element 402 (see Fig. 4). Labels generated by such users in the future are highly likely to be biased. Therefore, the anomalous crowdsourcing label processor 404 may also remove such labels, or at least assign a decreasing weight to such labels in the future.

In some non-limiting embodiments of the present technology, the anomalous crowdsourcing tag processor 404 may also communicate with the third application server 230 and tracking server 232 to track the browsing history of a future user accessing at least one web resource in the future, and take corrective action when there is an external trigger source. Such corrective action may include removing any labels generated by future users, or at least assigning a decreasing weight to any labels generated by future users in the future. In addition, in some embodiments, the abnormal crowdsourced label processor 404, if there is an external trigger source, can analyze the crowdsourced label generated by the future user to verify that the external trigger source is the actual source of the external trigger.

In some non-limiting embodiments of the present technology, where at least one web resource contains a plurality of web resources identified as possible sources of an external trigger, the abnormal crowdsourcing label processor 404 may order the at least one web resource in a ranked list 1500 (described below) using the proportion of the plurality 202 of users who viewed the web resource from at least one web resource.

In FIG. 15 shows a ranked list 1500 generated by the anomalous crowdsourcing tag processor 404 and containing a set 1502 of URLs associated with at least one web resource and the proportion 1504 of users out of a set of 202 users who have viewed a particular web resource of at least one. one web resource. For example, the URL www.iiii.com has been viewed by 50 users, the URL www.ffff.com has been viewed by 45 users, and so on. In addition, the abnormal crowdsourcing label processor 404 can select a subset (eg, subset 1506) of at least one web resource using the absolute value of the user share 1504 as a cutoff threshold for the low-frequency subset (eg, subset 1508) viewed by the user share 1504 .

In some non-limiting embodiments of the present technology, the anomalous crowdsourcing label processor 404 may also use a machine learning (MLA) algorithm trained to predict the highest ranking external trigger source based on the highest ranking N sources of the multiple external trigger sources.

It is contemplated that although the abnormal crowdsourcing label processor 404 is shown as a separate module, according to various non-limiting embodiments of the present technology, the functions of the abnormal crowdsource label processor 404 may be implemented in any one server or all servers. of the first application server 226, the second application server 228, the third application server 230, and the tracking server 232.

In FIG. 16 is a flow diagram of a method 1600 for detecting an external trigger in accordance with various non-limiting embodiments of the present technology.

In some non-limiting embodiments of the present technology, method 1600 may be performed on a single service, such as among other application services 310. In another embodiment, method 1600 may be performed on multiple services, such as social media application services 306, multimedia application services 308 and other application services 310. Method 1600 is performed in accordance with non-limiting embodiments of the present technology. Method 1600 may be performed by third application server 230 and begin at step 1602.

Step 1602: parsing a portion of the set of crowdsourced tags.

In step 1602, the peak trend analyzer 902 receives the crowdsourced labels 502 and analyzes a portion of the set of crowdsourced labels 502 or all of the crowdsourced labels 502 over a period of time.

Step 1604: Determining an anomalous subset of crowdsourced tags.

In step 1604, the anomalous subset selector 904 determines an anomalous subset of crowdsourced tags based on the peak trends analyzed by the peak trend analyzer 902 in part of the set of crowdsourced tags 502 or across all crowdsourced tags 502.

Step 1606: Get browsing history from web browsing history.

At step 1606, the browsing history receiver 1002 obtains the browsing history data 1100 associated with a portion of the users 202 forming part of the crowdsourced tag set 502 or all of the crowdsourced tags 502 from the web browsing history associated with the user set 202 and stored on the tracking server 232.

Step 1608: Dividing the browsing history into a first browsing history group and a second browsing history group.

In step 1608, the browsing history parser 1004 separates the browsing history data 1100 into a first browsing history group 1200 associated with users generating abnormal crowdsourced tags and a second browsing history group 1300 associated with the remaining users corresponding to part of the set of crowdsourced tags 502.

Step 1610: Generate a delta web resource set.

In step 1610, the delta set builder 1006 analyzes the first browsing history group 1200 and the second browsing history group 1300 to generate a delta set 1400. The delta set 1400 is generated based on the difference between the web resources associated with the first browsing history group 1200 and the second browsing history group 1300 and visited by some of the users 202 forming part of the set of crowdsourced tags 502 or all of the crowdsourced tags 502. The delta set contains at least one web resource from the first set of search history associated with the anomalous nature of the visits and with the source of the external triggering event .

It should be apparent to those skilled in the art that at least some embodiments of the present technology are intended to expand the arsenal of technical solutions to a particular technical problem, namely, to reduce the cost of computing resources on various servers by reducing the actions with anomalous labels performed through a communication network, in resulting in optimization of network traffic and reduction of load on network servers. It is obvious that not all the technical effects mentioned in this description should be present in every embodiment of the present technology. For example, there may be implementations of the present technology when the user does not receive some of these technical effects, or other implementations when the user receives other technical effects or there is no technical effect.

For a person skilled in the art, possible changes and improvements in the above-described embodiments of the present technology may be obvious. The foregoing description is given by way of example only and is not intended to limit the scope of the invention. The scope of protection of this technology is determined solely by the scope of the appended claims.

Although the embodiments described above are given with reference to specific steps performed in a specific order, it should be understood that these steps can be combined, separated, or that their order can be changed without departing from the boundaries of the present technology. Accordingly, the order and grouping of steps is not limiting to the present technology.

Claims (45)

1. A method for detecting an external triggering event that caused the appearance of an abnormal crowdsourcing label for a digital element available on a digital platform, wherein the abnormal crowdsourcing label is a label from a plurality of crowdsourcing labels assigned to a digital element by a plurality of users, a plurality of crowdsourcing labels is associated with a typical distribution of labels in it, and the anomalous crowdsourced label is triggered by an external triggering event that refers to, but is not associated with, a digital item, executed by a server that stores a web browsing log associated with at least some users from a plurality of users, and includes : - analysis of a part of the set of crowdsourced tags collected over a period of time; - identifying an anomalous subset of crowdsourced tags potentially triggered by the occurrence of an external trigger event during that time period; - obtaining from the web browsing log a browsing history associated with a subset of the plurality of users forming this part of the plurality of crowdsourced tags; separating the browsing history associated with a subset of the plurality of users into a first browsing history group associated with the abnormal subset of crowdsourced tags and a second browsing history group associated with the remaining tags of a portion of the crowdsourced tags set; - forming a set of web resources based on the analysis of the first group of browsing history and the second group of browsing history for differences in web resources visited by the corresponding subset of the plurality of users, while said set of web resources contains at least one web resource from the first a set of search history associated with the anomalous nature of visits; and - mapping this at least one web resource to the source of the external triggering event. 2. The method according to claim. 1, characterized in that it further includes determining users from a plurality of users accessing said at least one web resource during said period of time. 3. The method of claim 2, further comprising removing at least some of the labels from the plurality of labels from users who have accessed said at least one web resource during said period of time. 4. The method of claim 2, further comprising assigning a decreasing weight to at least some of the labels from the plurality of labels from users accessing said at least one web resource during said period of time. 5. The method according to p. 1, characterized in that it in the future period of time after the said period of time additionally includes: - determining a subset of users from a plurality of future users sending tags in a future time period, accessing said at least one web resource before a future time period; - removing labels from a subset of users from a set of future users or assigning a decreasing weight to labels generated by a subset of users from a set of future users. 6. The method according to claim 1, characterized in that the determination of the abnormal subset includes the analysis of peak trends among all neighboring labels in the actual distribution of labels in the part of the set of crowdsourced labels based on peak trends that do not correspond to the typical distribution of labels that define the abnormal subset. 7. The method according to claim 6, characterized in that peak trends not corresponding to a typical label distribution are based on one set of two adjacent labels associated with an abnormal peak and/or on two sets of successive adjacent labels associated with abnormal peak. 8. The method according to claim 7, wherein one set of two adjacent labels contains a low level label or a high level label. 9. The method according to claim 7, characterized in that the two sets of successive adjacent labels contain low-level labels or high-level labels. 10. The method of claim. 1, characterized in that it further includes determining the anomalous nature of the visits. 11. The method according to claim 10, characterized in that the determination of the anomalous nature of the visits includes: - calculating the first proportion of users associated with the first browsing history group and accessing the at least one web resource; - calculating the second proportion of users associated with the second browsing history group and accessing the at least one web resource; and - determining that said at least one web resource is associated with an abnormal pattern of visits if the first share is greater than the second share. 12. The method of claim 11, further comprising comparing the first beat and the second beat using a Z-statistic. 13. The method of claim 12, wherein if the Z-statistic indicates a statistically significant difference, then said at least one web resource is determined to be the source of the external trigger. 14. The method according to claim 13, characterized in that it further includes ordering said at least one web resource in a ranked list using the proportion of a plurality of users who have viewed a particular web resource from among said at least one web resource . 15. The method according to claim 14, characterized in that it further includes selecting a subset that is the source of an external triggering event from among said at least one web resource, using as a cutoff threshold the absolute number of users in the set of users who viewed a specific web resource from among the mentioned at least one web resource. 16. The method of claim. 1, characterized in that at least one web resource contains a plurality of web resources, each of which contains a corresponding version of the external trigger. 17. The method of claim 1, wherein the digital element contains content and the external trigger is a feedback digital element containing feedback about the content of the digital element. 18. The method of claim 17, wherein the digital recall element is generated by an object other than the source of the digital element. 19. The method of claim. 1, characterized in that at a future point in time it further includes analyzing the user's browsing history for the presence of a source of an external triggering event and performing a corrective action if there is a source of an external triggering event. 20. The method of claim 19, further comprising, if there is an external trigger source, parsing the crowdsourced label generated by the user to verify that the external trigger source is the actual source of the external trigger . 21. The method of claim 1, wherein the external trigger source is a plurality of external trigger sources, and the method further includes: - selection of N sources with the highest rank from the set of sources and - training the machine learning algorithm using the N sources with the highest rank to predict the source of the external initiating event with the highest rank. 22. The method of claim. 1, characterized in that the anomalous pattern of visits corresponds to at least one resource that is not present in the second browsing history. 23. A system for detecting an external triggering event that caused the appearance of an abnormal crowdsourcing label for a digital element available on a digital platform, while the abnormal crowdsourcing label is a label from a set of crowdsourcing labels assigned to a digital element by many users, a set of crowdsourcing labels is associated with a typical distribution of labels in it, and the anomalous crowdsourced label is triggered by an external triggering event that refers to a digital element, but is not associated with this digital element, while the system contains: - a server that stores a web browsing log associated with at least some of the users of the plurality of users; - a machine-readable physical storage medium containing instructions; - a processor configured to perform the following actions when executing instructions: - analysis of a part of the set of crowdsourced tags collected over a period of time; - determination of an anomalous subset of crowdsourced tags potentially triggered by the occurrence of an external trigger event during this time period; - obtaining from the web browsing log a browsing history associated with a subset of the plurality of users forming this part of the plurality of crowdsourced tags; - separating the browsing history associated with a subset of the plurality of users into a first browsing history group associated with the abnormal subset of crowdsourced tags and a second browsing history group associated with the remaining tags from that part of the crowdsourced tags set; - forming a set of web resources based on the analysis of the first group of browsing history and the second group of browsing history for differences in web resources visited by the corresponding subset of the plurality of users, while said set of web resources contains at least one web resource from the first a set of search history associated with the anomalous nature of visits; and - mapping this at least one web resource to the source of the external triggering event.

Images