CN108628931B - Method, device and equipment for data driving service - Google Patents
Method, device and equipment for data driving service Download PDFInfo
- Publication number
- CN108628931B CN108628931B CN201810213068.6A CN201810213068A CN108628931B CN 108628931 B CN108628931 B CN 108628931B CN 201810213068 A CN201810213068 A CN 201810213068A CN 108628931 B CN108628931 B CN 108628931B
- Authority
- CN
- China
- Prior art keywords
- data
- service
- unified
- view model
- driving
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000012216 screening Methods 0.000 claims abstract description 30
- 238000004364 calculation method Methods 0.000 claims description 26
- 238000012545 processing Methods 0.000 claims description 26
- 230000008569 process Effects 0.000 claims description 18
- 230000003542 behavioural effect Effects 0.000 claims description 3
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims 2
- 238000010586 diagram Methods 0.000 description 16
- 230000006870 function Effects 0.000 description 11
- 238000004590 computer program Methods 0.000 description 9
- 230000006872 improvement Effects 0.000 description 9
- 230000006399 behavior Effects 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 238000001914 filtration Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008030 elimination Effects 0.000 description 2
- 238000003379 elimination reaction Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000000750 progressive effect Effects 0.000 description 2
- 230000004931 aggregating effect Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 229920001296 polysiloxane Polymers 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Stored Programmes (AREA)
Abstract
The embodiment of the specification discloses a method, a device and equipment for data driving service. The scheme comprises the following steps: the method comprises the steps of obtaining original data of a first service, generating a unified data view model through ETL according to the original data, screening out invalid data in required data at least comprising the unified data view model by utilizing a rule engine to obtain valid data, and driving a second service according to the valid data, wherein the required data further comprises characteristic data related to the unified data view model, other data obtained by copying to the local or remotely capturing and the like.
Description
Technical Field
The present disclosure relates to the field of computer software technologies, and in particular, to a method, an apparatus, and a device for data-driven services.
Background
In a service scene, other service behaviors are often required to drive the service scene, so as to achieve the purpose of scene connection.
When a user uses a certain service function, another service scenario is triggered, which is usually based on the message or interface connection and concatenation of the service application layer. When the number of the butt-joint scenes is too large, the research and development cost is increased, and the scene connection has business requirements such as rules and limits, and the business application is difficult to provide flexible expansibility. When the number of service parties interfacing a scene increases gradually, the message replication cost adopting the message mode is also increased in multiples.
Based on this, there is a need for efficient and flexible traffic driven schemes.
Disclosure of Invention
The embodiment of the specification provides a method, a device and equipment for data driving service, which are used for solving the following technical problems: there is a need for efficient and flexible service-driven schemes.
In order to solve the above technical problem, the embodiments of the present specification are implemented as follows:
the method for driving a service by data provided by the embodiment of the present specification includes:
acquiring original data of a first service;
generating a unified data view model by extracting-Transform-Load (ETL) according to the original data;
utilizing a rule engine to screen out invalid data in the required data at least comprising the unified data view model to obtain valid data;
and driving a second service according to the effective data.
An apparatus for data-driven services provided in an embodiment of the present specification includes:
the acquisition module acquires original data of a first service;
the generation module generates a unified data view model through ETL according to the original data;
the screening module screens out invalid data at least containing the required data of the unified data view model by using a rule engine to obtain valid data;
and the driving module drives the second service according to the effective data.
An apparatus for data-driven services provided in an embodiment of the present specification includes:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to cause the at least one processor to:
acquiring original data of a first service;
generating a unified data view model through ETL according to the original data;
utilizing a rule engine to screen out invalid data in the required data at least comprising the unified data view model to obtain valid data;
and driving a second service according to the effective data.
The embodiment of the specification adopts at least one technical scheme which can achieve the following beneficial effects: data such as online real-time data streams of services can be used as a driving source, the data is processed through the ETL and the rule engine, a driving scheme is arranged, and other services can be driven efficiently and flexibly.
Drawings
In order to more clearly illustrate the embodiments of the present specification or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only some embodiments described in the present specification, and for those skilled in the art, other drawings can be obtained according to the drawings without any creative effort.
Fig. 1 is a schematic diagram of an overall architecture involved in a practical application scenario according to the solution of the present specification;
fig. 2 is a flowchart illustrating a method for data-driven services according to an embodiment of the present disclosure;
fig. 3 is a schematic diagram illustrating an implementation principle of a method for driving a service by data according to an embodiment of the present disclosure in an actual application scenario;
fig. 4 is a schematic structural diagram of an apparatus corresponding to the data driven service in fig. 2 according to an embodiment of the present disclosure;
fig. 5 is a schematic structural diagram of an apparatus corresponding to the data driven service in fig. 2 according to an embodiment of the present disclosure.
Detailed Description
The embodiment of the specification provides a method, a device and equipment for data driving service.
In order to make those skilled in the art better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present specification will be clearly and completely described below with reference to the drawings in the embodiments of the present specification, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any inventive step based on the embodiments of the present disclosure, shall fall within the scope of protection of the present application.
Data Technology (DT) era internet architecture layering can divide the overall architecture into a Data layer and a service layer, the service layer focuses more on its own services, and the Data layer can serve as a base platform hub to connect independent service scenes. The scheme of the specification is mainly in a data layer, adopts an innovative data driving idea to complete a data business process, helps to quickly establish scene connection among businesses, improves a traditional application architecture into a data application architecture, can improve both the stability and the performance of the architecture, and is also beneficial to reducing the hardware cost.
Fig. 1 is a schematic diagram of an overall architecture related to the solution of the present specification in a practical application scenario. In the whole framework, a scheme is executed by a data driving system positioned on a data layer, and the data driving system takes related service data positioned on the data layer as a driving source, processes the driving source and is used for driving services positioned on a service layer. The service includes a plurality of service scenarios, and specifically, may drive a corresponding service scenario.
The solution of the present specification is explained below mainly based on the exemplary architecture in fig. 1.
Fig. 2 is a flowchart illustrating a method for driving a service by data according to an embodiment of the present disclosure. The flow in fig. 2 may include the following steps:
s202: original data of a first service is obtained.
In the embodiment of the present specification, after processing raw data of a service, the service itself or another service may be driven, where "driving" is understood to be triggering a service scenario and a service action in the service scenario. For convenience of description, a service to which the body data as a driving source belongs is referred to as a first service, and a service to be driven is referred to as a second service.
In the embodiments of the present specification, the raw data includes, for example, behavior data and/or pipeline data. Taking e-business service as an example: the behavior data can record behaviors of a user for browsing commodities, adding commodities to favorites, placing orders, evaluating merchants and the like, and can also record behaviors of the merchants for putting commodities on and off shelves, evaluating the user, adding logistics information and the like; the running data can contain information such as order details, account transfer details and the like of the user, and transaction behaviors between the user and the merchant are recorded more directly.
Further, the real-time performance requirement is higher for the online system, and accordingly, the online real-time raw data may also be obtained in step S202, so as to facilitate driving the service in time subsequently. For example, raw data may be streamed in an online data stream, and the streaming may be performed in an attrition data access manner, such as message queue, database synchronization, log synchronization, and the like.
S204: and generating a unified data view model through ETL according to the original data.
In the embodiment of the present specification, the ETL can implement a process of extracting, converting, and loading data from a source end to a destination end, and the usability of data processed by the ETL can be generally improved. The conversion process includes, for example: null value processing, normalization of data format, data splitting, data correctness verification, data replacement, data query, main foreign key constraint establishment and the like.
The original data is data of a partial bottom layer, which is not easy to understand, and may belong to different ranges (for example, different classes, different structures, different service scenarios, etc.), which is not favorable for subsequent unified processing. Based on this, by the ETL, the raw data is unified and visualized, and a more easily usable unified data view model is generated, which represents the raw data in a unified range and can also be represented in a view form for data that is not easily understood.
S206: and screening out invalid data in the required data at least comprising the unified data view model by using a rule engine to obtain valid data.
In the embodiment of the present specification, the required data refers to data required for implementing a subsequent driving service, and may include at least one of the following data in addition to the unified data view model: characteristic data associated with the unified data view model, other data copied to the local or obtained through remote capture.
The feature data can be generated from a unified data view model, typically containing some statistical data in the unified data view model. The other data is generally directly related to the set service driving condition, and it is necessary to combine the other data, such as environment data when the first service is performed, data of a service related to the first service, and the like, to decide whether to perform service driving and how to perform the driving.
In the embodiment of the present specification, in the rule engine, a data filtering rule may be set according to the service driving condition, where the data filtering rule includes one or more filtering conditions, and invalid data is filtered according to the filtering conditions.
For example, assuming that the required data is denoted as q, the data culling rule includes two culling conditions, condition 1 and condition 2.
Condition 1: let q have attributes a, b, requiring a >5 and b not belonging to set m;
condition 2: assuming that q has an attribute c, c% 3 is required to be 0, and if this condition is true, condition 1 is ignored.
Then according to the condition 1 and the condition, after invalid data in q is screened out, q satisfies:
“(q.a>5&&q.b not in m)||(c％3＝0)”。
in this embodiment, the required data after the invalid data is screened out may be used as valid data, or may be further processed after the required data after the invalid data is screened out and then used as valid data.
S208: and driving a second service according to the effective data.
In the embodiment of the present specification, the second service is driven when valid data exists, or the second service is driven when valid data exists and meets other service driving conditions.
By the method of fig. 2, data such as online real-time data stream of a service can be used as a driving source, and a driving scheme can be arranged by processing of the ETL and the rule engine, so that other services can be driven efficiently and flexibly.
Based on the method of fig. 2, the present specification also provides some specific embodiments of the method, and further embodiments, which are described below.
In this embodiment of the present specification, for step S206, the utilizing a rule engine to screen out invalid data in the required data at least including the unified data view model to obtain valid data may specifically include: obtaining required data outside the unified data view model, the required data including at least one of: characteristic data associated with the unified data view model are copied to local or other data obtained through remote capture; combining to obtain calculation data according to the unified data view model and the required data except the unified data view model; and screening invalid data in the calculated data by using a rule engine to obtain valid data.
In the embodiments of the present specification, the feature data may be extracted by means of feature engineering or manual work.
Take the characteristic engineering way as an example. Specifically, feature engineering processing may be performed on the unified data view model to obtain feature data associated with the unified data view model, where the feature data includes at least one of the following statistical results of specified attribute data in a specified time window, with a specified main body as a dimension: accumulating values, the most value, the mean value and the de-weight statistical value.
For e-commerce services, a specified subject is, for example, a user, a merchant, or a good. The accumulated value is, for example, the accumulated payment amount of the user in a week, or the accumulated consumption times of the user in a week, etc.; the maximum value is, for example, the maximum value or the minimum value; the duplication elimination statistic value is, for example, the number of brands purchased by the user in one week, and if the user purchases more than two commodities of the same brand, duplication occurs, and the brand is only once recorded, namely duplication elimination.
In the embodiment of the present specification, the number of times that some services can be driven in a certain time is limited, and if the upper limit of the number of driving times (referred to as a quota) has been reached, the corresponding calculation data may be discarded as excess data and not used in the subsequent processing procedure. It should be noted that the excess condition is not limited to this, for example, if multiple service scenarios are driven alternatively, after one of the multiple service scenarios is driven, the others are not driven any more, then after one of the multiple service scenarios is driven, the calculated data corresponding to the multiple service scenarios may be screened out as the excess data.
According to the analysis in the previous paragraph, the screening out invalid data in the calculation data to obtain valid data may specifically include: screening out invalid data in the calculation data; and carrying out limit calculation according to the calculated data after the invalid data is screened out so as to screen out excess data and obtain valid data. This process is typically performed based on a principal dimension, such as analyzing excess for a single user, store, or subject such as a good, and then performing an excess calculation.
For example, according to the consumption situation of a single user at a certain shop on the day, when the consumption reaches 500 yuan and when the consumption reaches 1000 yuan, a red packet is respectively awarded to the user. The consumption data of the user is a driving source, the bonus red packet is a service to be driven, the user is currently rewarded with at most two bonus packets, the consumption data within 1000 yuan is enough to drive the service correspondingly twice, and the consumption data exceeding 1000 yuan can belong to excess data.
In the embodiment of the present specification, in some scenarios, the second service may be directly driven by using the valid data, but in other scenarios, the second service may not be directly driven by using the valid data, but the valid data may need to be further processed and calculated to adapt to the scenario of the second service, and then the second service is driven by using the processed and calculated valid data. For example, assuming that the first service is a payment service, the valid data is processed user consumption data, the second service is a public service game, and virtual money exists in the public service game, the processing and calculating specifically includes converting the user consumption data into the virtual money according to a predetermined policy, and further triggering some scenes in the public service game, thereby implementing service driving.
In the embodiment of the present specification, the driving manner for driving the second service is various, for example, the driving manner may support multiple protocols such as message queue, remote invocation, HTTP2, dynamic scenario, and the like. In addition, for the online system, various data required in the service driving process can be synchronized by using various synchronization modes for the above processes.
According to the above description, the embodiment of the present disclosure further provides an implementation principle schematic diagram of the method for data-driven services in a practical application scenario, as shown in fig. 3.
In fig. 3, the method of the data driven service described above is performed by a corresponding data driven system. The system takes the data stream of a certain service as a driving source (corresponding to step 1 in fig. 3), arranges a driving scheme (corresponding to steps 2-5 of p1, p2 and p3 in fig. 3), and further drives another service. Taking p1 as an example, the service driving process mainly includes the following steps:
1. online real-time data d flows in;
2. d, generating a unified data view model d' through ETL processing;
2.1, inputting d 'into a real-time computing engine, performing characteristic engineering processing, and outputting d' associated characteristic data f1, f2, f3, … and fn;
2.2, aggregating other required data e, and combining d', f 1-fn and e into calculation data q;
3. inputting q into a rule engine, and screening out invalid data;
4. carrying out excess calculation on the q after the invalid data is screened out, and screening out excess data;
5. processing and calculating the q after screening the excess data to generate q';
6. another service is driven with q'.
The scheme has the following advantages: the data driving system is positioned in the integral architecture data layer and drives from bottom to top, so that each service of the application layer can be decoupled, and the architecture stability is improved; the connection between scenes is completed in a data operation mode, the arrangement of the whole driving scheme is completed in a configuration mode, and application layer development is not needed; the data driving system can adopt a quasi-real-time driving scheme, so that higher service timeliness is ensured; and the data statistical calculation of the characteristic engineering is supported, and the real-time data attribute is expanded.
Based on the same idea, the embodiments of the present specification further provide a device and an apparatus corresponding to the above method, see fig. 4 and fig. 5.
Fig. 4 is a schematic structural diagram of an apparatus for a data driven service corresponding to fig. 2 provided in an embodiment of the present specification, where the apparatus includes:
the acquiring module 401 acquires original data of a first service;
a generating module 402, which generates a unified data view model by extracting, converting and loading ETL according to the original data;
a screening module 403, configured to screen out invalid data in the required data at least including the unified data view model by using a rule engine, to obtain valid data;
and the driving module 404 drives the second service according to the valid data.
Optionally, the raw data comprises online real-time behavioral data and/or pipeline data.
Optionally, the screening module 403 screens out invalid data in the required data at least including the unified data view model by using a rule engine to obtain valid data, which specifically includes:
the screening module 403 obtains required data other than the unified data view model, the required data including at least one of: characteristic data associated with the unified data view model are copied to local or other data obtained through remote capture;
combining to obtain calculation data according to the unified data view model and the required data except the unified data view model;
and screening invalid data in the calculated data by using a rule engine to obtain valid data.
Optionally, the screening module 403 obtains feature data associated with the unified data view model, which specifically includes:
the screening module 403 performs feature engineering processing on the unified data view model to obtain feature data associated with the unified data view model;
the characteristic data comprises at least one of the following statistical results of the specified attribute data in a specified time window by taking a specified main body as a dimension: and accumulating the value, the most value, the mean value and the weight-removing statistic value.
Optionally, the screening module 403 screens invalid data in the calculation data to obtain valid data, and further includes:
the screening module 403 screens invalid data in the calculation data;
and carrying out limit calculation according to the calculated data after the invalid data is screened out so as to screen out excess data and obtain valid data.
Optionally, the driving module 404 drives the second service according to the valid data, which specifically includes:
the driving module 404 drives the second service according to the valid data by using at least one of the following manners: message queue, remote invocation, HTTP2, dynamic scripting.
Optionally, the driving module 404 drives the second service according to the valid data, specifically including:
the driving module 404 processes and calculates the valid data to adapt to a scene of a second service;
and driving the second service by using the effective data after the processing calculation.
Fig. 5 is a schematic structural diagram of an apparatus for a data driven service corresponding to fig. 2 provided in an embodiment of this specification, where the apparatus includes:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to:
acquiring original data of a first service;
generating a unified data view model through ETL according to the original data;
utilizing a rule engine to screen out invalid data in the required data at least comprising the unified data view model to obtain valid data;
and driving a second service according to the effective data.
Based on the same idea, the embodiments of the present specification further provide a non-volatile computer storage medium corresponding to fig. 2, which stores computer-executable instructions configured to:
acquiring original data of a first service;
generating a unified data view model through ETL according to the original data;
utilizing a rule engine to screen out invalid data in the required data at least comprising the unified data view model to obtain valid data;
and driving a second service according to the effective data.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the embodiments of the apparatus, the device, and the nonvolatile computer storage medium, since they are substantially similar to the embodiments of the method, the description is simple, and for the relevant points, reference may be made to the partial description of the embodiments of the method.
The apparatus, the device, the nonvolatile computer storage medium, and the method provided in the embodiments of the present specification correspond to each other, and therefore, the apparatus, the device, and the nonvolatile computer storage medium also have similar advantageous technical effects to the corresponding method.
In the 90 s of the 20 th century, improvements in a technology could clearly distinguish between improvements in hardware (e.g., improvements in circuit structures such as diodes, transistors, switches, etc.) and improvements in software (improvements in process flow). However, as technology advances, many of today's process flow improvements have been seen as direct improvements in hardware circuit architecture. Designers almost always obtain the corresponding hardware circuit structure by programming an improved method flow into the hardware circuit. Thus, it cannot be said that an improvement in the process flow cannot be realized by hardware physical modules. For example, a Programmable Logic Device (PLD), such as a Field Programmable Gate Array (FPGA), is an integrated circuit whose Logic functions are determined by programming the Device by a user. A digital system is "integrated" on a PLD by the designer's own programming without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Furthermore, nowadays, instead of manually making an Integrated Circuit chip, such Programming is often implemented by "logic compiler" software, which is similar to a software compiler used in program development and writing, but the original code before compiling is also written by a specific Programming Language, which is called Hardware Description Language (HDL), and HDL is not only one but many, such as abel (advanced Boolean Expression Language), ahdl (alternate Hardware Description Language), traffic, pl (core universal Programming Language), HDCal (jhdware Description Language), lang, Lola, HDL, laspam, hardward Description Language (vhr Description Language), vhal (Hardware Description Language), and vhigh-Language, which are currently used in most common. It will also be apparent to those skilled in the art that hardware circuitry that implements the logical method flows can be readily obtained by merely slightly programming the method flows into an integrated circuit using the hardware description languages described above.
The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer readable medium that stores computer readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, an Application Specific Integrated Circuit (ASIC), a programmable logic controller, and embedded microcontrollers, examples of which include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic for the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller as pure computer readable program code, the same functionality can be implemented by logically programming method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Such a controller may thus be considered a hardware component, and the means included therein for performing the various functions may also be considered as a structure within the hardware component. Or even means for performing the functions may be regarded as being both a software module for performing the method and a structure within a hardware component.
The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. One typical implementation device is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functions of the various elements may be implemented in the same one or more software and/or hardware implementations of the present description.
As will be appreciated by one skilled in the art, the present specification embodiments may be provided as a method, system, or computer program product. Accordingly, embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present description may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.
The description has been presented with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the description. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, the embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the description may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.
This description may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only an example of the present specification, and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement or the like made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.
Claims (13)
1. A method for data-driven service, which is applied to a data layer, comprises:
acquiring original data of a first service; the original data flows in the form of an online data stream;
loading ETL through extraction conversion according to the original data to generate a unified data view model;
utilizing a rule engine to screen out invalid data in the required data at least comprising the unified data view model to obtain valid data; the desired data includes at least one of: characteristic data associated with the unified data view model are copied to local or other data obtained through remote capture;
driving a second service according to the effective data; driving the data of the second service to be effective data or effective data after processing calculation; the second service is a service located in a service layer; the second service and the first service are mutually independent services corresponding to different service scenes;
the screening out invalid data in the required data at least containing the unified data view model to obtain valid data specifically includes:
screening out invalid data in the calculated data; the calculation data is obtained by combining the unified data view model and other required data;
calculating the amount according to the calculated data after the invalid data is screened out to screen out excess data and obtain valid data; the excess data represents data that the second service exceeds the upper limit of the driving times, or represents calculation data corresponding to the remaining services after one of the services corresponding to the service scenario is driven when one of the plurality of service scenarios is driven.
2. The method of claim 1, the raw data comprising online real-time behavioral data and/or pipeline data.
3. The method according to claim 1, wherein the using a rules engine to screen out invalid data in the required data at least including the unified data view model to obtain valid data specifically comprises:
acquiring required data outside the unified data view model;
combining to obtain calculation data according to the unified data view model and the required data except the unified data view model;
and screening invalid data in the calculated data by using a rule engine to obtain valid data.
4. The method according to claim 3, wherein the obtaining of the feature data associated with the unified data view model specifically comprises:
performing characteristic engineering processing on the unified data view model to obtain characteristic data associated with the unified data view model;
the characteristic data comprises at least one of the following statistical results of the specified attribute data in a specified time window by taking a specified main body as a dimension: and accumulating the value, the most value, the mean value and the weight-removing statistic value.
5. The method according to claim 1, wherein the driving the second service according to the valid data specifically includes:
and driving a second service by utilizing at least one of the following modes according to the effective data: message queue, remote invocation, HTTP2, dynamic scripting.
6. The method according to claim 1, wherein the driving the second service according to the valid data specifically includes:
processing and calculating the effective data so as to adapt to the scene of a second service;
and driving the second service by using the effective data after the processing calculation.
7. An apparatus for data-driven services, the apparatus being applied to a data layer, comprising:
the acquisition module acquires original data of a first service; the original data flows in the form of an online data stream;
the generation module is used for generating a unified data view model by extracting, converting and loading ETL according to the original data;
the screening module screens out invalid data at least containing the required data of the unified data view model by using a rule engine to obtain valid data; the desired data includes at least one of: characteristic data associated with the unified data view model are copied to local or other data obtained through remote capture; the screening module is also used for screening invalid data in the calculated data; the calculation data is obtained by combining the unified data view model and other required data; calculating the amount according to the calculated data after screening the invalid data to screen excess data to obtain valid data; the excess data represents data that the second service exceeds the upper limit of the driving times, or represents calculated data corresponding to the rest of services after one of the services corresponding to the service scene is driven when one of the plurality of service scenes is driven;
the driving module drives a second service according to the effective data; driving the data of the second service to be effective data or effective data after processing calculation; the second service is a service located in a service layer; the second service and the first service are services corresponding to different service scenes which are independent from each other.
8. The apparatus of claim 7, the raw data comprising online real-time behavioral data and/or pipeline data.
9. The apparatus of claim 7, wherein the culling module culls invalid data from the required data at least including the unified data view model by using a rules engine to obtain valid data, and specifically comprises:
the screening module acquires required data except the unified data view model;
combining to obtain calculation data according to the unified data view model and the required data except the unified data view model;
and screening invalid data in the calculated data by using a rule engine to obtain valid data.
10. The apparatus according to claim 9, wherein the screening module obtains feature data associated with the unified data view model, and specifically includes:
the screening module carries out feature engineering processing on the unified data view model to obtain feature data associated with the unified data view model;
the characteristic data comprises at least one of the following statistical results of the specified attribute data in a specified time window by taking a specified main body as a dimension: and accumulating the value, the most value, the mean value and the weight-removing statistic value.
11. The apparatus according to claim 7, wherein the driving module drives the second service according to the valid data, specifically including:
the driving module drives a second service according to the effective data by using at least one of the following modes: message queue, remote invocation, HTTP2, dynamic scripting.
12. The apparatus according to claim 7, wherein the driving module drives the second service according to the valid data, specifically including:
the driving module processes and calculates the effective data so as to enable the effective data to be suitable for a scene of a second service;
and driving the second service by using the effective data after the processing calculation.
13. An apparatus for data-driven services, the apparatus being applied to a data layer, comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to:
acquiring original data of a first service; the original data flows in the form of an online data stream;
loading ETL through extraction conversion according to the original data to generate a unified data view model;
utilizing a rule engine to screen out invalid data in the required data at least comprising the unified data view model to obtain valid data; the desired data includes at least one of: characteristic data associated with the unified data view model are copied to local or other data obtained through remote capture;
driving a second service according to the effective data; driving the data of the second service to be effective data or effective data after processing calculation; the second service is a service located in a service layer; the second service and the first service are mutually independent services corresponding to different service scenes;
the screening out invalid data in the required data at least containing the unified data view model to obtain valid data specifically includes:
screening out invalid data in the calculated data; the calculation data is obtained by combining the unified data view model and other required data;
calculating the amount according to the calculated data after the invalid data is screened out to screen out excess data and obtain valid data; the excess data represents data that the second service exceeds an upper limit of the driving times, or represents calculation data corresponding to the rest of services after one of the services corresponding to the service scenario is driven when one of the plurality of service scenarios is driven.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211110282.1A CN115563197A (en) | 2018-03-15 | 2018-03-15 | Method, device and equipment for data driving service |
| CN201810213068.6A CN108628931B (en) | 2018-03-15 | 2018-03-15 | Method, device and equipment for data driving service |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810213068.6A CN108628931B (en) | 2018-03-15 | 2018-03-15 | Method, device and equipment for data driving service |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211110282.1A Division CN115563197A (en) | 2018-03-15 | 2018-03-15 | Method, device and equipment for data driving service |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108628931A CN108628931A (en) | 2018-10-09 |
| CN108628931B true CN108628931B (en) | 2022-08-30 |
Family
ID=63706206
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810213068.6A Active CN108628931B (en) | 2018-03-15 | 2018-03-15 | Method, device and equipment for data driving service |
| CN202211110282.1A Pending CN115563197A (en) | 2018-03-15 | 2018-03-15 | Method, device and equipment for data driving service |
Family Applications After (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211110282.1A Pending CN115563197A (en) | 2018-03-15 | 2018-03-15 | Method, device and equipment for data driving service |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN108628931B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109614205A (en) * | 2018-10-18 | 2019-04-12 | 阿里巴巴集团控股有限公司 | A kind of method for processing business, device, equipment and system |
| CN111144950B (en) * | 2019-12-30 | 2023-06-30 | 北京顺丰同城科技有限公司 | Model screening method and device, electronic equipment and storage medium |
| CN112084179B (en) * | 2020-09-02 | 2023-11-07 | 北京锐安科技有限公司 | Data processing method, device, equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722582A (en) * | 2012-06-07 | 2012-10-10 | 陈浩 | System and method for integrating data on basis of reverse clearing |
| CN106874482A (en) * | 2017-02-20 | 2017-06-20 | 山东鲁能软件技术有限公司 | A kind of device and method of the patterned data prediction based on big data technology |
| CN107103548A (en) * | 2011-11-17 | 2017-08-29 | 阿里巴巴集团控股有限公司 | The monitoring method and system and risk monitoring and control method and system of network behavior data |
| CN107526600A (en) * | 2017-09-05 | 2017-12-29 | 成都优易数据有限公司 | A kind of visual numeric simulation analysis platform and its data cleaning method based on hadoop and spark |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102567796A (en) * | 2010-12-20 | 2012-07-11 | 北大方正集团有限公司 | Method and device for importing data |
| US8694918B2 (en) * | 2012-02-06 | 2014-04-08 | International Business Machines Corporation | Conveying hierarchical elements of a user interface |
| US9892178B2 (en) * | 2013-09-19 | 2018-02-13 | Workday, Inc. | Systems and methods for interest-driven business intelligence systems including event-oriented data |
| US20160162824A1 (en) * | 2014-12-05 | 2016-06-09 | International Business Machines Corporation | Monitoring the impact of information quality on business application components through an impact map to data sources |
| CN109564609A (en) * | 2016-08-05 | 2019-04-02 | 分形工业公司 | It mitigates and corrects using the detection of the computer attack of advanced computers decision-making platform |
| CN106339222B (en) * | 2016-08-23 | 2019-09-17 | 联动优势电子商务有限公司 | A kind of service implementing method and device |
| CN107766493A (en) * | 2017-10-19 | 2018-03-06 | 国网辽宁省电力有限公司 | A kind of patch source power network resources Model analysis system and method |
-
2018
- 2018-03-15 CN CN201810213068.6A patent/CN108628931B/en active Active
- 2018-03-15 CN CN202211110282.1A patent/CN115563197A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107103548A (en) * | 2011-11-17 | 2017-08-29 | 阿里巴巴集团控股有限公司 | The monitoring method and system and risk monitoring and control method and system of network behavior data |
| CN102722582A (en) * | 2012-06-07 | 2012-10-10 | 陈浩 | System and method for integrating data on basis of reverse clearing |
| CN106874482A (en) * | 2017-02-20 | 2017-06-20 | 山东鲁能软件技术有限公司 | A kind of device and method of the patterned data prediction based on big data technology |
| CN107526600A (en) * | 2017-09-05 | 2017-12-29 | 成都优易数据有限公司 | A kind of visual numeric simulation analysis platform and its data cleaning method based on hadoop and spark |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115563197A (en) | 2023-01-03 |
| CN108628931A (en) | 2018-10-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107424069B (en) | Wind control feature generation method, risk monitoring method and equipment | |
| CN110070399B (en) | Coupon pushing method and device | |
| CN108173706B (en) | Service marking method, device and equipment under multi-service system | |
| CN110717536A (en) | Method and device for generating training sample | |
| CN108628931B (en) | Method, device and equipment for data driving service | |
| CN110569428A (en) | recommendation model construction method, device and equipment | |
| CN112015626B (en) | User behavior recording method, device and equipment | |
| CN109389412B (en) | Method and device for training model, service equipment and user equipment | |
| CN109327715B (en) | Video risk identification method, device and equipment | |
| CN113079201B (en) | Information processing system, method, device and equipment | |
| CN109597678B (en) | Task processing method and device | |
| CN109359109B (en) | Data processing method and system based on distributed stream computing | |
| CN115878092A (en) | Code generation method, device and medium based on micro-service architecture | |
| CN107451204B (en) | Data query method, device and equipment | |
| CN110727700A (en) | Method and system for integrating multi-source streaming data into transaction type streaming data | |
| CN111930810B (en) | Data rule mining method and device | |
| KR101779529B1 (en) | Identifying software application events | |
| CN110738562A (en) | Method, device and equipment for generating risk reminding information | |
| CN111722995A (en) | Data processing method and device | |
| CN109597695A (en) | A kind of data processing method, device and equipment | |
| CN110689418B (en) | Bill generation method and device | |
| CN109039695B (en) | Service fault processing method, device and equipment | |
| CN110704742B (en) | Feature extraction method and device | |
| CN114661826A (en) | Blockchain-based data processing method, device and equipment | |
| CN114594944A (en) | Application construction system and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200918 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant after: Advanced innovation technology Co.,Ltd. Address before: A four-storey 847 mailbox in Grand Cayman Capital Building, British Cayman Islands Applicant before: Alibaba Group Holding Ltd. Effective date of registration: 20200918 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant after: Innovative advanced technology Co.,Ltd. Address before: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant before: Advanced innovation technology Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |