CN105515963A - Data gateway device and big data system - Google Patents

Abstract

The invention discloses a data gateway device and a big data system. Among them, the data gateway device includes: a user management module, which is used to manage user information; a service management module, which is used to manage the information of data services open to users; a rule management module, which is used for different users and different data services Set up different compliance check rules; data access management module, which is used to receive data access requests from users, send corresponding data access requests to the data center, and accept source data returned by the data center; and data desensitization processing module , which is used to check the compliance of the source data returned by the data center according to the rules set by the rule management module, so as to generate service data. It is used to provide the service data generated by the data masking processing module to the user.

Description

Data gateway device and big data system

technical field

The invention belongs to the technical field of data processing, and in particular relates to a data gateway (Gateway) device and a big data system including the data gateway device, which can guarantee the safe operation of big data.

Background technique

In the era of big data, revitalizing data assets and openly sharing data has become the general trend. Opening data services and realizing the operation and realization of big data are hot issues in the current development of big data. However, in the operation or realization of big data, how to ensure data compliance, avoid leakage of sensitive information, measure or bill transaction data, and audit data has become an urgent problem to be solved.

Contents of the invention

The object of the present invention is to provide a data gateway device and a big data system including the data gateway device, which can solve at least one of the above-mentioned problems in the prior art.

In order to achieve the purpose of the present invention, according to one aspect of the present invention, a data gateway device is provided, which includes:

User management module, which is used to manage user information;

A service management module, which is used to manage the information of data services open to users;

A rule management module, which is used to set different compliance check rules for different users and different data services;

A data access management module, which is used to receive the data access request sent by the user, send a corresponding data access request to the data center, and accept the source data returned by the data center; and

The data desensitization processing module is used to perform a compliance check on the source data returned by the data center according to the rules set by the rule management module, so as to generate service data, and the service data is data satisfying the compliance check, wherein

The data access management module is also used to provide the service data generated by the data desensitization processing module to the user.

Optionally, the rule management module can also be used to set different data filtering desensitization rules for different users and different data services, and the data desensitization processing module can be used to filter the source data returned by the data center Desensitization, and then perform a compliance check on the filtered and desensitized data.

Optionally, the data fetching module can also be used to measure and charge the output service data.

Optionally, the rule management module can also be used to set the number of output lines of the user's service data, and when the total number of lines of data satisfying the compliance check is greater than the set number of output lines of service data, the data The sensitive processing module can also be used to generate the service data by sequentially intercepting a certain number of rows of data from the first row of the data satisfying the compliance check according to the number of output rows of the service data set by the rule management module, The number of rows=the number of output rows of service data.

Optionally, the data retrieval management module can also be used to generate a data retrieval task according to the user's data retrieval request, and provide a viewing function of the data retrieval task.

Optionally, the user management module can also be used to configure permissions of different types of users, and the permissions include function menu access permissions.

The compliance checking rules may include: data dictionary matching, numerical range checking and field length checking.

The data filtering desensitization rules may include: service output field filtering, field condition filtering and field content processing.

Optionally, the data gateway device may further include a storage unit, which is used to cache the source data returned by the data center and the service data generated by the data desensitization processing module, wherein the data retrieval management module is configured to: first determine the Whether there is the data requested by the customer in the storage unit, if the requested data has been cached in the storage unit, then the requested data cached in the storage unit will be provided to the customer as service data, otherwise, a corresponding request will be sent to the data center number of requests.

Optionally, the data gateway device may also include a security audit module, which is used to record the operation log and the access log, and archive the operation log and the access log.

Optionally, the data gateway device may further include a work order management module, which is used to process system work order tasks, and the work order tasks include work order creation and work order viewing.

Optionally, the data gateway device may also include a system management module, which is used to configure the basic data of the system, the basic data includes a data cycle and a timer, and the data cycle is the source data returned by the data center and the The service data generated by the data desensitization processing module is cached in the data gateway device.

According to another aspect of the present invention, there is also provided a client-accessible big data system, which includes a data center and a data gateway device, the data gateway device is connected to the data center through an interface, wherein the data gateway The device is any one of the data gateway devices described above, and the data gateway device is configured to be able to connect with the client through an interface.

The data gateway device and the data center can realize data connection through a data service information synchronous update interface, a data access application interface, a service data generation notification interface, and a service data synchronization interface, wherein,

Through the service information synchronous update interface, the information of the data service in the service management module of the data gateway device is consistent with the data service opened by the data center;

Through the data access application interface, the data access management module of the data gateway device sends a service data access request to the data center, and when the service is a real-time service, the data center uses the data access The application interface returns the source data to the data access management module of the data gateway device. When the service is a periodic service and the account period of the service is a historical account period, the data center sends the data to the data access management module through the access application interface. The access management module of the data gateway device returns the storage information of the requested source data. When the service is a periodic service and the account period of the service is a future account period, the data center sends The number retrieval management module of the gateway device returns a message that the account period has not yet arrived;

Through the service data generation notification interface, the data center notifies the access management module of the data gateway device that the requested source data is ready and returns the storage information of the requested source data; and

Through the service data synchronization interface, the data access management module of the data gateway device acquires data according to the storage information of the requested source data. The data service information synchronous update interface, data access application interface, and service data generation notification interface can be realized through the WebService interface; the service data synchronization interface can be realized through the FTP interface.

The data gateway device is also configured to be able to connect with the client data through the access application interface and the service data push interface, wherein,

Through the access application interface, the access management module of the data gateway device accepts the access request of the service sent by the client, and the access management module of the data gateway device provides the requested service data to the user;

Through the service data push interface, the data access management module of the data gateway device pushes the requested service data to a specified location.

The data access application interface can be realized through the WebService interface; the service data push interface can be realized through the FTP interface.

In the data gateway device and the big data system provided by the present invention, for different users and different data types, different compliance inspection rules are used to review the source data provided by the data center, ensuring the standardization of service data provided to users ;For the sensitive information in the defined data, filter and desensitize according to the set rules to ensure that there is no sensitive content in the transaction output data (ie, service data); for the compliant desensitized output data, according to different users, The data volume and data value attributes of the transaction output are measured and billed; in addition, during the data output or transaction process, the data output object (that is, the transaction user), the time of the output data, and the format of the output data are permanently archived. , to facilitate future operational compliance audits.

Description of drawings

FIG. 1 is a schematic diagram of a data gateway device provided by an embodiment of the present invention;

FIG. 2 is a schematic diagram of a data gateway device provided by another embodiment of the present invention;

Fig. 3 is a working flow diagram of the data gateway device provided by the embodiment of the present invention; and

FIG. 4 is a schematic diagram of a big data system provided by an embodiment of the present invention.

detailed description

In order to enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

An embodiment of the present invention provides a data gateway device for ensuring safe operation of big data. As shown in FIG. 1 , the data gateway device may include a user management module, a service management module, a rule management module, a data access management module and a data desensitization processing module. The user management module is used to manage user information. The service management module is used to manage the information of data services open to users; the rule management module is used to set different compliance inspection rules for different users and different data services; the access management module is used to receive service access requests issued by users , send the corresponding access request to the data center, and accept the source data returned by the data center; the data desensitization processing module is used to check the compliance of the source data returned by the data center according to the rules set by the rule management module, so as to generate the service data, the service data is data that meets compliance checks; wherein, the access management module is also used to provide the service data generated by the data desensitization processing module to users.

According to other embodiments of the present invention, the data gateway device for ensuring safe operation of big data provided by the present invention may further include a security audit module, a rule management module, a work order management module and/or a system management module. For example, FIG. 2 shows a schematic diagram of a data gateway device provided by another embodiment of the present invention.

Hereinafter, each functional module of the data gateway device will be described in detail with reference to FIG. 1 and FIG. 2 .

User Management Module

The user management module is used to manage user information, and can configure the function menu access rights of different types of users, so as to support multiple types of user management and realize multi-tenant management.

Users can include data users and institutional users. Data users are data service demand users or third-party systems/applications, and each data user must be associated with an organization. Data users can apply for data services and obtain compliant service data. Institutional users are responsible persons of data users, enterprises or organizations. There is only one institutional user under an institution, but it can have multiple data users. It can be understood that both institutional users and data users can apply by themselves. By reviewing the applications and modifications of institutional users and data users, the flow of service data can be better tracked and recorded.

Users may also include background business management personnel, for example, data security administrators and auditors. The data security administrator configures data filtering, desensitization, and compliance checking rules. Auditors review user registration/modification/logout, assign user confidentiality levels, data service viewing permissions, access permissions, access priority permissions, and review data filtering, desensitization, and compliance inspection rules, etc.

The users may also include background system administrator users, for example, may include: operation and maintenance personnel and system administrators. Operators operate and maintain the system. The system administrator has full authority of the system. It can be understood that system administrators can create auditors, data security administrators, and operation and maintenance personnel.

The user management module can also configure user function menu access rights for different types of users, such as data users, institutional users, data security administrators, auditors, operation and maintenance personnel, and system administrators. In general, different users have different levels of access to function menus.

Service Management Module

The service management module is used to manage the information of data services open to users, for example, management operations such as creating new data services, modifying and deleting open data services. The data service information in the data management module is consistent with the data services that data operators open to users.

Data service information may include: service basic information, which may include service code, service name, service type (real-time service or periodic service), service cycle (year, quarter, month, week or day; for periodic service), periodic data ready Date (for periodic services), service output set code, service output set name, etc.; service output field information, including service output set code, field code, field name, field data type, field description, etc.; and service billing information, For example, price per unit field or per row etc.

Data access management module

The access management module is used to receive the user's access request, send a corresponding access request to the data center, and accept the source data returned by the data center. In the present invention, the source data returned by the data center includes: for real-time services, the source data returned by the data center, and for periodical services, the data access management module obtains the source data according to the storage information of the source data returned by the data center. After the data desensitization processing module (described later) processes the source data returned by the data center to generate service data (ie, compliant data), the access management module provides the service data to the user.

The data retrieval management module can also be used to generate data retrieval tasks according to user data retrieval requests. There are four ways to create data retrieval tasks. One is for real-time services, where the user manually initiates a real-time retrieval task; second, for periodic services, the user manually initiates the data retrieval task for the first time; After the data access task is initiated, the data access management module automatically generates the data access task periodically; fourth, the data security administrator modifies the compliance inspection rules and regenerates the data access task after it takes effect immediately.

The access management module can also provide a viewing function for service access tasks. Service retrieval tasks can be displayed in a list.

The information contained in the service retrieval task list may include: data user name, service code, service name, service type, service period, task account period, data ready status, task status, task start time, and task end time.

Wherein, the service type includes periodic or real-time.

Task accounting period This field has a value only when the service type is Periodic. If the task is a real-time fetching task initiated manually, this field displays real-time. According to the account type, the display format can be: year-yyyy, month-yyyymm, quarter-yyyy*Q, week-yyyy**, day-yyyy***.

Task status can include valid and invalid. After the task is created, the task status is valid by default. After the data security administrator modifies the rules (including data filtering and desensitization rules and compliance inspection rules) and chooses to take effect immediately, all current related unfinished data retrieval tasks will be terminated and restarted. Create a data retrieval task, implement new security rules, and set the task status of the terminated data retrieval task to invalid.

The value of the data readiness status can be ready to fetch, start fetching, fetch complete, desensitization, compliance check, data ready, illegal service termination, service delivered or service delivery failed, retrying, each Item descriptions are as follows:

The service task list can also support query functions, such as query based on data user name, service code, service name, start time, and end time.

The service access task list can also provide managers with functional operations such as inspection result viewing, service source data and service data sampling.

Inspection Results View the data desensitization and inspection results of the service source data provided for this data retrieval task, including the data user login name, data user name, affiliated institution user login name, affiliated institution user name, and service code for this data retrieval task , service name, service type, service period, task account period, task start time, service source data compliance, total number of compliance check lines, data desensitization/compliance check rules. If the inspection result of the service source data is non-compliant, all the non-compliant data detected will be displayed in a list. The list display information includes: the sequence number of the non-compliant data row, the row number in the service source data, and For all the data in the row where the compliant data is located, the non-compliant data will be marked in red fonts, and the compliance inspection rules that the data should comply with will pop up in a mouse-triggered manner, and the export function of the compliance inspection results is provided, and it supports txt and excel file formats export.

The sampling of service source data and service data supports the sampling of continuously specified row data content (designated from XXX row to XXX row), the maximum number of consecutive rows is 100, and the end row number must be less than or equal to the total number of rows of source data/service data.

If it is necessary to make statistics on the service data obtained by the user, the data access management module can also measure and charge the service data output to the user. For the output service data, according to different users, metering and billing are carried out according to the data volume and data value attributes. Display information in the service access task list to increase the service data volume and charging amount. For example, the amount of data and the charging amount of the data service can be calculated according to the following formula:

Data volume = number of fields × number of rows;

Data service charging amount = pricing per unit field/row x data volume x user grading standard coefficient.

Here, the user rating standard coefficient of the user may be predetermined in the user management module.

Generally speaking, data users can only view the information of authorized data services and make access requests.

It can be understood that the data gateway device may further include a storage unit for caching the source data returned by the data center and the service data generated by the data masking processing module. At this point, the access management module is configured to first confirm whether the data requested by the customer exists in the storage unit, and if the requested data has been cached in the storage unit, then provide the requested data cached in the storage unit as service data To the user, otherwise send a corresponding access request to the data center.

It should be noted that the storage unit can exist as an independent unit or as a part of the access management module, which is not limited in the present invention, as long as the source data returned by the data center or the source data obtained by the access management module can be The data masking module can read it, and the service data generated by the data masking module can be read by the access management module.

Rule Management Module

The rule management module is used to independently configure the permissions of institutional users and/or data users, and generate different viewing permissions, data access application authorization, data access priority permissions, and data filtering and desensitization for different users and different data services Rules and compliance check rules, so that resource isolation between different data users can be realized. Through the rule management module, a unified management function of security rules can be provided. It can be understood that any rule configuration can only be operated by the data security administrator. The viewing permission of data services refers to whether institutional users and/or data users can see the service and its detailed information in the data service list. Different users may see different service lists.

Data access application authorization for data services refers to whether data users can obtain the data of the service. Data users can submit data access applications in the data access management module for services with viewing authority. After review and configuration of data access application authorization, data access requests can be initiated through the data access management module.

The access priority authority of the data service refers to the sorting priority rules for the access tasks created by the access management module when multiple data users initiate access requests at the same time.

Since different institutional users or data users generally have different confidentiality requirements, in order to prevent multiple data users from obtaining sensitive information by sharing and reorganizing the obtained service data, it is preferable to formulate different secrets for different users and different data services. Data filtering desensitization rules and compliance check rules. Data filtering and desensitization rules and compliance inspection rules can be set randomly or in advance, as long as it can make it difficult for data users belonging to the same organizational user to recover sensitive information through acquired service data.

Compliance checking rules can include: data dictionary matching, numeric range checking, and field length checking. Compliance check rules can be imported in excel file format.

Filtering and desensitization rules may include: service output field filtering, field conditional filtering, and field content processing. The conditional filtering of fields may include: filtering according to a dictionary of fields, and for fields of numerical type, filtering according to numerical comparison expressions. Field content processing may include: processing of replacing special characters, and processing of intercepting and replacing continuous digits. Hereinafter, these filtering desensitization rules will be described in detail.

(1) Service output field screening

The service output field filter is the column filter of the data table. For example, the source data of a service includes 9 output fields, and the field names are province, user code, name, date of birth, ID number, mobile phone number, package type, model, and average monthly call charge. According to the "Regulations on the Protection of Personal Information of Telecommunications and Internet Users", fields such as user name, date of birth, ID card number, address, phone number, account number, and password are not allowed to be output. Therefore, for the service request of the data user, the rule management module sets the field screening rules so that the service only outputs five fields: province, user code, package type, model, and average monthly call fee. The 4 fields of date, ID card number and mobile phone number will be filtered out.

(2) Conditional filtering of fields

The conditional filtering of the field is the row filtering of the data table. As mentioned above, conditional filtering of fields can be divided into two types of filtering.

The first is filtering based on a dictionary of fields. For example, the province dictionary corresponding to the output field "province" in the above example service includes five values: "Beijing", "Hebei", "Tianjin", "Henan", and "Shandong". The rule management module can set this value for data users. If the value filtering condition of the output field "province" of the service is to include only "Beijing", then for this data user, the value of the province field in the service source data is "Hebei", "Tianjin", "Henan", "Shandong" data rows are filtered out.

The second is for fields of numeric type, filtering based on numeric comparison expressions. For example, the data type of the output field "average monthly phone bill" in the above example service is double, and the rule management module sets the value filter condition of the output field "average monthly phone bill" of the service for the data user as follows: the value range of the field is within 50 ~100, the data rows whose value of the monthly average call fee field in the service source data is less than 50 or greater than 100 will be filtered out.

(3) Field content processing

As shown above, field content processing includes two types of processing.

The first is to replace special characters. For example, in the service example above, the rule management module replaces special characters in the output field "user code" of the service according to the data user settings, and replaces the special character (123) with ***, then the field in the service source data " "123" in User Code" will be replaced with "***". For example, if the "user code" in the service source data is "1235678", it will be displayed as "***5678" after replacement.

The second is to intercept and replace characters with consecutive digits. Intercepting and replacing includes: replacing the front-end N-digit characters with specified characters, replacing the middle consecutive N-digit characters with specified characters, or replacing the characters starting from the number at the back-end with specified characters.

When it is required to replace the first 4 digits in the "User Code" with 6666, if the "User Code" in the service source data is "1235678", it will be displayed as: 6666678 after replacement;

When it is required to replace the 2nd to 5th digits in the "User Code" with 6666, if the "User Code" in the service source data is "1235678", it will be displayed as: 1666678 after replacement;

When it is required to replace the characters after the 4th digit in the "User Code" with 6666, if the "User Code" in the service source data is "1235678", it will be displayed as: 1236666 after replacement.

It should be noted that data compliance checking rules must be set in the rule management module, so as to ensure the compliance of service data provided to users. In the case where the data center has been configured with a filtering and desensitization function, the data gateway device does not need to configure data filtering and desensitization rules.

Data desensitization module

The data desensitization processing module is used to filter and desensitize the source data returned by the data center according to the configuration of the rule management module, and to check the compliance of the filtered and desensitized data. If the data is compliant, the compliance The data is provided to the user through the access management module as service data, otherwise, stop outputting the data.

If the rule management module also sets the number of output rows of service data for different users and different data services, when the total number of rows of data that meets the compliance check is greater than the number of output rows of service data set, the data will be removed. The sensitive processing module will intercept the data of a certain number of lines sequentially from the first line to generate the service data according to the output line number of the user's service data set by the rule management module, and the number of lines=service The number of output rows of data.

It can be understood that in the case where the rule management module does not need to configure data filtering and desensitization rules (that is, when the data center has been configured with a filtering and desensitization function), the desensitization and desensitization module directly returns the source data or retrieved data from the data center. The source data obtained through the data application interface is checked for compliance.

Security Audit Module

The security audit module is used to record operation logs and access logs, and archive the operation logs and access logs.

The operation log refers to the operation information of all users during the operation of the device, including: operating user account, operated user account, operation type, operation function, operation content (such as: query your own to-do task list, data volume: 25 items), operation time etc.

The access log refers to the acquisition operation information of all users' service data during the operation of the device, including: the time of acquiring data, data volume, basic service information, service output field information (that is, field information of service data), and executed service data Desensitization/compliance check rules, etc. The acquisition operation of service data includes two situations. One is that the management personnel actively download or sample download the service source data and service data, and the other is that the data gateway device successfully returns the service data to the data user.

The security audit module can periodically audit operation logs and access logs, and supports field-level auditing of service data obtained by institutional users or data users, so that the data flow of sensitive fields can be tracked more accurately, making big data operations Better meet operational compliance audits.

Work order management module

The work order management module is used to process work order tasks of the data gateway device, and the work order tasks include work order creation and work order viewing. For tasks that require approval and transfer, the data gateway device can automatically create a work order. For different businesses, set work order task trigger conditions at the business flow node. System administrators can configure the user types that handle each work order task, and when the work order task is triggered, a to-do work order is automatically generated for this type of user. For example, for the new user registration business, the submission of registration information can be set as the trigger condition, and the user type that handles the new user registration work order can be set as the reviewer. After the user submits the registration information, the new user registration work order will be triggered, and a pending work order will be automatically generated for the on-the-job review personnel to process.

In addition, work orders can be viewed through the work order management module, and managers can view pending/completed/withdrawn work orders.

The viewing of pending work orders provides work order query and processing functions to processing users. Users can perform fuzzy queries based on to-do type, to-do title, and to-do content. The system will sort and display in descending order according to the generation time of the to-do, and the user can directly click Process to enter the corresponding work order processing page. After the work order is processed, it will automatically flow to the completed work order.

The view of the completed work order will display the historical processing information of the processed work order, including: the previous work order processor, processing time, processing opinion, processing time and other information.

The view of the withdrawn ticket will display the details of the withdrawn ticket, including the reason for withdrawing the ticket. Through the work order management module, the data gateway system can realize the operation of the review process of user information and permissions of various users.

System Management Module

The system management module is used to configure the basic data of the system, which may include data periods and timers.

The data cycle is the cache cycle of the service source data and service data in the data gateway device, and the data cycle can be preset or specified or modified by the operation and maintenance personnel.

The type of data caching cycle can be year, quarter, month, week or day, and the cycle unit is day, that is, the data will be cleared after this cycle.

The system management module scans and checks whether the data is expired every day. For the expired data, it generates an expired data cleaning to-do work order, notifies the operation and maintenance personnel, and automatically performs the cleaning of expired data after the operation and maintenance personnel approve it.

Timers can include: automatic task creation monitoring timers for periodic services, service source data scanning intervals, data desensitization/regularity inspection intervals, re-push intervals after FTP push failures, and the number of FTP push resend failures.

Fig. 3 is a working flowchart of a data gateway device for ensuring safe operation of big data provided by an embodiment of the present invention. As shown in Figure 3, the workflow specifically includes the following steps.

Step 1: The user registers through the user management module of the data gateway device.

Step 2: The service management module of the data gateway device synchronizes information of various data services from the data center.

Step 3: The rule management module of the data gateway device authorizes the data service query authority to the user.

Step 4: The user submits a service access application through the access management module of the gateway device.

Step 5: The rule management module of the data gateway device configures data service access rights for users.

The rule management module can apply for authorization of service access, priority authorization of service data access, service data filtering and desensitization rules, service data compliance inspection rules, and the number of output rows of service data for users. The same service independently configures its data filtering desensitization rules, compliance inspection rules, and the number of output rows of service data for different users.

Step 6: The data access management of the data gateway device submits a data access application for the user authorization service to the data center.

Step 7: The data center generates service source data. In particular, if the user submits an application for accessing real-time service data, the data center generates real-time service source data. If the user submits a data access application for periodic services, the data center will generate periodic service source data according to the generation cycle of periodic services.

Step 8: The data center returns the service source data (real-time service source data or periodic service source data storage information) to the data gateway device.

Step 9: The data desensitization processing module of the data gateway device performs data filtering and desensitization on the service source data according to the data filtering and desensitization rules for the user set in the rule management module. Specific data filtering and desensitization can refer to the above, and will not be repeated here.

Step 10: The data desensitization processing module of the data gateway device performs a data compliance check on the service source data after data filtering and desensitization according to the compliance check rules set for the user in the rule management module. If there is any non-compliant data , the service data retrieval process will be interrupted.

As mentioned earlier, compliance checks can include data dictionary matches, numeric range checks, and field length checks.

Step 11: The data desensitization processing module of the data gateway device, according to the number of output rows of service data set for the user in the rule management module, for the filtered and desensitized service source data that meets the data compliance inspection rules, start from the first row At the beginning, the data of the specified number of rows is sequentially intercepted to generate service data.

Step 12: The data access management module of the data gateway device returns the service data to the user, and performs metering and charging.

It should be noted that the service data of the real-time service can be returned to the user through the WebService interface. The service data of the periodic service can be pushed to the user's FTP server through the FTP interface in the form of data files.

Step 13: The security audit module of the data gateway device regularly audits the operation log and data access log.

According to another aspect of the present invention, there is also provided a client-accessible big data communication system, as shown in Figure 4, which includes a data center and a data gateway device, the data gateway device is connected to the data center through an interface, And the data gateway device is configured to communicate data with the client through the interface. The data service request sent by the client (representing the data service demander) is sent to the data center through the data gateway device, and the service source data returned by the data center is subjected to allergy desensitization processing and compliance inspection through the data gateway device, and then the desensitized The service data that meets the compliance check is provided to the client, which prevents the leakage of key and sensitive data and meets the compliance. In addition, the data gateway device provided by the present invention can also measure or charge the service data sent to the client, and audit the service data. The data gateway device is the above-mentioned data gateway device, therefore, it will not be described again.

According to the flow direction of data in the big data system, the data center of the operator constitutes a data intranet area, and the data gateway device provided by the present invention constitutes an audit area, which can guarantee the security of data service open operation, and the client requesting data service constitutes In the extranet cooperation zone, clients can send various data service requests, as shown in Figure 4.

The data gateway device (audit area) and the data center (data intranet area) can be connected through interfaces. In one example, the data gateway device and the data center implement data connection through a data service information synchronization update interface, a data access application interface, a service data generation notification interface, and a service data synchronization interface.

The data service information synchronous update interface can be implemented through the WebService interface. When the data service information opened by the data center changes (for example, creating a new data service or modifying a data service), the data center actively pushes the data service information to the data gateway device through the WebService interface. Through the data service information synchronous update interface, the information in the service management module of the data gateway device is consistent with the data service opened by the data center

The access application interface can also be implemented through the WebService interface. After receiving the service data extraction application from the client, the data gateway device checks whether the requested data exists in the local service cache table, and if not, initiates a data extraction request for the service to the data center. If the service type of the service is real-time service, the data center will generate source data in real time and return it to the data gateway device; if the service type of the service is periodic service, and the requested service account period is a historical account period, the data center will The storage information of the service source data (including, for example, the FTP address of the file server, access port, login user name, password, source data file storage path, source data file name, etc.) is returned to the data gateway device; if the service type of the service is periodic service, and the requested service account period is a future account period, the data center will return a message that the account period has not yet arrived to the data gateway device, and then notify the data through the service data generation notification interface after the account period conditions are met and the source data is generated gateway device.

The service data generation notification interface can also be implemented through the WebService interface. As mentioned above, when the data gateway device initiates a data access application for a certain periodic service to the data center, since the requested service account period is a future account period, the data center will notify through this interface after the account period conditions are met and the source data is generated. The data gateway device is ready for data, and provides the storage information of the service source data (including, for example, FTP address of the file server, access port, login user name, password, source data file storage path, source data file name) to the data gateway device.

The service data synchronization interface can be realized through the FTP interface, and the data gateway device obtains the service source data through FTP according to the storage information of the service source data.

The data gateway device (audit area) and the client (external network cooperation area) can be connected through the interface. In an example, the data gateway device and the client implement a data connection through a data access application interface and a service data push interface.

The access application interface can be realized through the WebService interface. The client submits a service data extraction application to the data gateway device through the WebService interface. If the service type of the service is a real-time service, the data gateway device returns the service data to the client after desensitizing the source data, checking compliance, and other review processes.

The service data push interface can be realized through the FTP interface. After the data gateway device desensitizes the source data of periodic services and checks for compliance, the service data is pushed to the FTP specified by the data service demander through the service data push interface in the form of data files. The data service demander can specify FTP when registering, that is, specify the push FTP connection related information for receiving service data, which can include: FTP address, access port, login user name, password, file storage path, etc.

It can be understood that although the WebService interface and the FTP interface are used as examples to illustrate the present invention, the present invention is not limited thereto. Any interface that can realize real-time data exchange and periodic data reading and writing can be used in the present invention.

It can be understood that, the above embodiments are only exemplary embodiments adopted for illustrating the principle of the present invention, but the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also regarded as the protection scope of the present invention.

Claims (13)

1. A data gateway device, comprising: User management module, which is used to manage user information; A service management module, which is used to manage the information of data services open to users; A rule management module, which is used to set different compliance check rules for different users and different data services; A data access management module, which is used to receive the data access request sent by the user, send a corresponding data access request to the data center, and accept the source data returned by the data center; and The data desensitization processing module is used to perform a compliance check on the source data returned by the data center according to the rules set by the rule management module, so as to generate service data, and the service data is data satisfying the compliance check, wherein The data access management module is also used to provide the service data generated by the data desensitization processing module to the user. 2. The data gateway device according to claim 1, the rule management module is also used for setting different data filtering desensitization rules for different users and different data services, and the data desensitization processing module is used for data desensitization The source data returned by the center is filtered and desensitized, and then the filtered and desensitized data is checked for compliance. 3. The data gateway device according to claim 1, wherein the data fetching module is also used for measuring and charging the output service data. 4. The data gateway device according to claim 3, wherein the rule management module is also used to set the output line number of the user's service data, when the total line number of the data satisfying the compliance check is greater than the set service data When the number of output lines is set, the data desensitization processing module is also used to intercept a certain number of lines sequentially starting from the first line for the data that meets the compliance check according to the number of output lines of the service data set by the rule management module data to generate the service data, wherein the number of rows=the output row number of the service data. 5. The data gateway device according to claim 1, wherein the compliance checking rules include: data dictionary matching, value range checking and field length checking. 6. The data gateway device according to claim 1, wherein the filter desensitization rules include: service output field filtering, field conditional filtering and field content processing. 7. The data gateway device according to any one of claims 1-6, further comprising: A storage unit, which is used to cache the source data returned by the data center and the service data generated by the data desensitization processing module, wherein the access management module is configured to: firstly determine whether there is data requested by the customer in the storage unit, if If the requested data has been cached in the storage unit, the requested data cached in the storage unit is provided to the client as service data, otherwise a corresponding access request is sent to the data center. 8. The data gateway device according to any one of claims 1-6, further comprising: The security audit module is used for recording operation logs and access logs, and archiving the operation logs and access logs. 9. The data gateway device according to any one of claims 1-6, further comprising: The system management module is used to configure the basic data of the system, the basic data includes data cycle and timer, the data cycle is the source data returned by the data center and the service data generated by the data desensitization processing module in the data gateway The cache period in the device. 10. A client-accessible big data system, which includes a data center and a data gateway device, the data gateway device is connected to the data center through an interface, wherein the data gateway device is according to claim 1-8 The data gateway device described in any one, and the data gateway device is configured to be able to connect with the client through an interface. 11. The big data system according to claim 10, wherein the data gateway device and the data center are implemented through a data service information synchronous update interface, a data access application interface, a service data generation notification interface, and a service data synchronization interface data connection, where Through the service information synchronous update interface, the information of the data service in the service management module of the data gateway device is consistent with the data service opened by the data center; Through the data access application interface, the data access management module of the data gateway device sends a service data access request to the data center, and when the service is a real-time service, the data center uses the data access The application interface returns the source data to the data access management module of the data gateway device. When the service is a periodic service and the account period of the service is a historical account period, the data center sends the data to the data access management module through the access application interface. The access management module of the data gateway device returns the storage information of the requested source data. When the service is a periodic service and the account period of the service is a future account period, the data center sends The number retrieval management module of the gateway device returns a message that the account period has not yet arrived; Through the service data generation notification interface, the data center notifies the access management module of the data gateway device that the requested source data is ready and returns the storage information of the requested source data; and Through the service data synchronization interface, the data retrieval management module of the data gateway device obtains source data according to storage information of the requested source data. 12. The big data system according to claim 10 or 11, wherein the data gateway device is further configured to be connected to the client data through the access application interface and the service data push interface, wherein, Through the access application interface, the access management module of the data gateway device accepts the access request of the service sent by the client, and the access management module of the data gateway device provides the requested service data to the user; Through the service data push interface, the data access management module of the data gateway device pushes the requested service data to a specified location. 13. The big data system according to claim 12, wherein, the data service information synchronous update interface, the access application interface, and the service data generation notification interface are implemented through the WebService interface, and the service data synchronization interface is implemented through the FTP interface; The data access application interface is implemented through the WebService interface, and the service data push interface is implemented through the FTP interface.