Disclosure of Invention
The invention solves the problems existing in the prior art and provides a unified multi-platform portal system based on Kafka technology.
The technical scheme adopted by the invention is that the unified multi-platform portal system based on Kafka technology comprises:
the front end of the portal system comprises a plurality of front end pages of a third party system for displaying and interacting;
The portal system back end comprises a proxy server, a portal service platform and a corresponding message queue, wherein the portal service platform acquires a service request from a front end page of a third party system through the proxy server and returns data, and adopts a message queue service based on Kafka technology to register and cancel the service third party system;
Incremental synchronization of user information of the third-party system to the portal service platform is achieved.
Preferably, the synchronization of the user information of the third party system comprises the steps of:
Step 1.1, creating an incremental file index table for recording uploading files of each business module in any third party system;
Step 1.2, collecting and processing user information and attachment data of each business module in the project;
setting a timing task increment synchronization strategy, and pushing file increment to a target server by a data source side based on a preset data segmentation rule on the premise of evaluating a network environment, wherein evaluation of the network environment can be based on judgment of network bandwidth, and each sub-volume is larger as the bandwidth is higher;
Setting a timed task loading file strategy, and when a target server detects file pushing behavior, carrying out integrity judgment on the file, and carrying out file loading based on a judgment result until a complete file is obtained;
and 1.5, decompressing the file into the corresponding catalogue according to the path in the file name.
Preferably, in step 1.2, the user newly-added or modified interface or method return value parameters are obtained, the type of the return value is judged, different strategies are adopted to analyze according to different return value types, unified user file information is obtained, a corresponding path URL is generated, the URL is processed based on a module rule strategy, a complete directory and a corresponding file path in a server are generated, an incremental file is generated according to the file path, and the incremental file information is recorded.
Preferably, the step 1.3 includes the steps of:
Step 1.3.1, acquiring a file synchronization task and acquiring the last file synchronization completion time;
Step 1.3.2, obtaining an uploading file record to be pushed after the last file synchronization completion time, backing up data, and generally deleting the backed up data after the file synchronization completion;
Step 1.3.3, carrying out asymmetric encryption on an original file by using a public key to obtain an encrypted file and compressing the encrypted file;
If the compressed encrypted file exceeds the threshold value, taking a model according to the file size and the threshold value, determining the total number of to-be-split volumes, performing splitting treatment on the compressed encrypted file, and performing the next step, otherwise, directly pushing the compressed encrypted file to a target server and returning;
Step 1.3.5, creating a temporary directory for the split volumes, splitting the compressed encrypted files and outputting the split volumes to obtain a split volume file set;
And step 1.3.6, finally, pushing the split file sets to the target server one by one, and recording pushing result information.
Preferably, the step 1.4 includes the steps of:
Step 1.4.1, the target server monitors the file synchronization message, if the file synchronization message is monitored, the synchronous file loading is carried out, all file sub-volumes are obtained according to the prefix matching of the file name, otherwise, the step 1.4.1 is repeated;
step 1.4.2, creating a temporary directory, and generating an original compressed file by dividing all files into volumes;
step 1.4.3, decompressing the compressed file to obtain an encrypted file, decrypting the encrypted file by using an asymmetric algorithm private key, and placing the file in a corresponding directory;
And step 1.4.4, the target server analyzes the user file, generates corresponding JavaBean information and stores the JavaBean information in a database.
Preferably, any one of the third party systems registers with the back end of the portal system through the registration code, and any one of the third party systems de-registers with the back end of the portal system through the registration code.
Preferably, any third party system carries an authentication parameter registration code, a user name and a password of Kafka and request parameters to request authentication to Kafka, the Kafka authenticates the user name and the password, an authentication result is returned after the authentication is passed, and the request is stored in the Kafka;
The portal service at the back end of the portal system monitors Kafka, when a registration message exists, a registration code is acquired to carry out registration code verification, request parameters are acquired after verification passes, a public key and a private key for synchronous authentication of the user information of the registered user are generated, the public key and the private key are stored at the back end of the portal system, the public key information is asynchronously transmitted back to a third party system through a public key callback interface URL, and the third party system stores the public key information.
Preferably, when the users are synchronous, the third party system inquires local user information, records the inquiring time as the next user inquiring time parameter, obtains the public key encryption after the user information is obtained, and sends the encrypted user information to the Kafka service with Kafka authentication, the Kafka carries out login authentication, and the encrypted user information is stored after the authentication is passed;
The method comprises the steps of monitoring whether a Kafka service has a message or not by the back end of a portal system, decrypting encrypted user information by a private key if the Kafka service has the message to obtain the user information, carrying out new judgment according to a unique identification number, carrying out account automatic binding on a user which is not bound with the back end of the portal system, and carrying out association binding on the unique identification number of the user and an ID distributed by the back end of the portal system.
Preferably, any third party system carries CorpId, appKey, appSecret, a portal access URL, a public key callback interface URL, and Logo as request parameters.
Preferably, invoking the entire link through the portal system front end comprises the steps of:
step 2.1, performing login authentication of the front end of the current portal system;
Step 2.2, after the front end authentication of the current portal system is passed, acquiring portal list information from the back end of the portal system, and performing front end rendering display;
Clicking any third party system, and acquiring an authentication code according to CorpId, wherein the front end of the third party system initiates authentication to the rear end of the third party system according to the authentication code, the rear end of the third party system receives a request, firstly initiates the request to the third party system according to AppKey, appSecret, acquires authentication accessToken, and then acquires user information according to accessToken and the authentication code;
And 2.4, acquiring a unique identifier corresponding to the back-end user information of the third-party system according to the user information, acquiring the user information of the third-party system according to the mobile phone, and finally entering a corresponding portal system to finish portal unification if the user information of the third-party system can be inquired.
The invention relates to a unified multi-platform portal system based on Kafka technology, wherein the front end of the portal system comprises a plurality of front end pages of a third party system for display and interaction, the rear end of the portal system comprises a proxy server, a portal service platform and a corresponding message queue, the portal service platform acquires a service request from the front end pages of the third party system through the proxy server and returns data, the message queue based on Kafka technology is adopted to serve registration and cancellation services from the third party system, and synchronization of user information of the third party system is realized based on incremental synchronization.
The invention has the beneficial effects that:
(1) Supporting a plurality of third party systems as the front end of the portal system, and realizing unification through the rear end of the portal system;
(2) Unified user by Kafka technology, intelligent registration and cancellation are realized, and the third party system is supported to display and arrange the front end of the portal system;
(3) And the large file is transmitted in a fragmentation way based on incremental synchronization, so that the effect and the speed of file transmission are ensured.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention relates to a unified multi-platform portal system based on Kafka technology, which unifies application inlets of platforms of a plurality of third party systems, opens each platform to enable the platforms to have access rights to each other, opens user information of each platform by adopting Kafka technology, completes portal registration and portal cancellation, and supports portal display position arrangement.
In the invention, kafka is applied to the system as a high-throughput distributed publish-subscribe message system, can process all action stream data of consumers in websites, including but not limited to web browsing, searching and actions of other users, unifies online and offline message processing through a Hadoop parallel loading mechanism, and provides real-time messages through clusters.
The system comprises:
the front end of the portal system comprises a plurality of front end pages of a third party system for displaying and interacting;
the portal service platform acquires a service request from a front page of a third party system through a proxy server and returns data, and adopts a message queue service based on Kafka technology to register and cancel the service third party system;
Incremental synchronization of user information of the third-party system to the portal service platform is achieved.
In the invention, in order to ensure the applicability of the third party system, a timing heartbeat technology is generally adopted to monitor whether the front end of each portal system is online.
In the invention, in order to ensure the consistency of data and the real-time and reliability of synchronization in the synchronization process, user information including file synchronization is performed in an incremental mode, when the file is a large file, the large file is segmented and rolled, the segmented and rolled into an original file at a receiving party, the data are finally analyzed and synchronously stored, and after the data are synchronized to a target server, the segmented and rolled into the original file which is placed in a corresponding catalog.
In the implementation of the present invention, the synchronization of the user information of the third party system includes the following steps:
Step 1.1, creating an incremental file index table for recording uploading files of each business module in any third party system;
Step 1.2, collecting and processing user information and attachment data of each business module in the project;
The attachment data uploaded by each business module is collected by combining the section (UploadFileAspect) and the annotation (UploadFileRecord), wherein the annotation only acts on the method for uploading the file, and UploadFileAspect is a concrete implementation flow of the annotation.
Specifically, in step 1.2, obtaining new or modified interface or method return value parameters of the user, judging the type of the return value, analyzing by adopting different strategies according to different return value types, such as String, javabean types, and the like, obtaining unified user file information and generating a corresponding path URL, processing the URL based on a module rule strategy, and generating a complete directory and a corresponding file path in the server;
For example, when the return value is of String type, the return value file path is directly returned, i.e. the return value is directly used as the file path, when the return value is just one object, the object attribute is id, url, file name, etc., the strategy of obtaining the file path is to analyze the return value, the original object is generated by the java reflection mechanism, and finally the url is obtained as the file path according to the object.
And generating an incremental file according to the file path, wherein the incremental file is stored in the form of a compressed file, and the name of the compressed file is generally a file storage path, so that a target server conveniently loads the file and places the file in a corresponding directory, and incremental file information including, but not limited to, a service module, the file name, a file storage path, the compressed file path, file uploading time, creator information and the like is recorded for later file synchronization.
Setting a timing task increment synchronization strategy, and pushing file increment to a target server by a data source side based on a preset data segmentation rule on the premise of evaluating a network environment;
the step 1.3 comprises the following steps:
Step 1.3.1, acquiring a file synchronization task and acquiring the last file synchronization completion time;
Step 1.3.2, obtaining an uploading file record to be pushed which is created after the last file synchronization completion time (the creation time is later than the last synchronization completion time) from a t_uploading_file_record table, and preventing the data backup from being lost;
Step 1.3.3, carrying out asymmetric encryption on an original file by using a public key to obtain an encrypted file and compressing the encrypted file to obtain a zip file package ASSET-fileName.zip;
If the compressed encrypted file exceeds the threshold value, taking a model according to the file size and the threshold value, determining the total number of to-be-split volumes, performing splitting treatment on the compressed encrypted file, and performing the next step, otherwise, directly pushing the compressed encrypted file to a target server and returning;
Step 1.3.5, creating a temporary directory ASSET-fileName-X for the split volumes, splitting the compressed encrypted files and outputting the split volumes to obtain split volume file sets, ASSET-fileName.z1, ASSET-fileName.z2 and the like;
And step 1.3.6, finally pushing the partitioned file sets to the target server one by one, and recording pushing result information, wherein the number of retries of failed pushing is generally 3.
Setting a timed task loading file strategy, when a target server detects file pushing behavior, carrying out integrity judgment on the file, and carrying out file loading based on a judgment result until the complete file is obtained, namely firstly judging whether the file is a split and split file, if so, carrying out file folding and folding processing to generate an original file, and then carrying out loading, otherwise, directly loading, and finally decompressing the file into a corresponding directory according to a path in a file name.
The step 1.4 comprises the following steps:
step 1.4.1, the target server monitors the file synchronization message, if the file synchronization message is monitored, the synchronous file loading is carried out, and all files are obtained in a split mode according to the prefix matching of the file names, such as ASSET-fileName.z1 and ASSET-fileName.z2, and if not, the step 1.4.1 is repeated;
In practical application, after file synchronization is completed, the user record in the incremental file index table can be marked that the file synchronization is completed, the incremental file index table is synchronized to a third party system side, and the third party system side only needs to process the marked user file which is completed synchronously until the third party system side receives all the sub-volumes based on the index table, namely receives all the sub-volumes of the file, and does not need to judge any more;
step 1.4.3, decompressing the compressed file to obtain an encrypted file, decrypting the encrypted file by using an asymmetric algorithm private key, and placing the file in a corresponding directory;
And step 1.4.4, the target server analyzes the user file, generates corresponding JavaBean information and stores the JavaBean information in a database.
And 1.5, decompressing the file into the corresponding catalogue according to the path in the file name.
In the invention, a portal service platform at the back end of the portal system provides the following services:
distributing portal registration codes, namely generating portal registration codes to be used when a third party system is registered before portal registration, and simultaneously giving out a user name and a secret key of Kafka;
Portal registration, wherein a third party system carries a registration code to carry out portal registration, and the registration needs to be uploaded CorpId, appKey, appSecret and stored, and Logo, system name and the like can be uploaded and displayed;
Portal cancellation, namely cancellation is carried out on registered portals, and authentication cancellation is carried out by carrying registration codes when cancellation is carried out;
user synchronization, integrating user information of all third party systems in a portal service platform;
portal management, which manages registered portals, such as modification, deletion, etc.;
portal arrangement, which manages the display position, style, logo and the like of the portal system.
It should be noted in advance that, in the system of the present invention, any third party system carries CorpId, appKey, appSecret, a portal access URL, a public key callback interface URL, and a Logo as request parameters, where the Logo is in a binary format or in a base64 string form.
Any one of the third party systems registers to the back end of the portal system through the registration code, and any one of the third party systems logs out to the back end of the portal system through the registration code.
Specifically, any third party system carries an authentication parameter registration code, a user name and a password of Kafka and request parameters to request authentication to Kafka, the Kafka authenticates the user name and the password, an authentication result is returned after the authentication is passed, and the request is stored in the Kafka for consumption of other services;
The portal service at the back end of the portal system monitors Kafka, when a registration message exists, the message is processed in real time, registration codes are acquired to check the registration codes, request parameters are acquired through check, corpId, appKey, appSecret and Logo are obtained, corpId, appKey, appSecret is stored, binary format or base64 character strings of the Logo are converted into texts, file paths are generated and stored, public keys and private keys for synchronous authentication of the user information of the registered user are generated, the public keys and the private keys are stored at the back end of the portal system, the public key information is asynchronously returned to a third party system through a public key callback interface URL, and the third party system stores the public key information.
Obviously, the third party system flow of cancellation is the same as portal registration, a cancellation request is initiated with a registration code and a public key, a private key can be obtained by inquiring according to the registration code and the public key, and portal deletion cancellation operation is performed.
When the user is synchronous, the third party system inquires local user information, namely increment inquiry, records the inquiry time as the next user inquiry time parameter to obtain user information and encrypts the public key, and sends the encrypted user information to the Kafka service with Kafka authentication, wherein the Kafka carries out login authentication, and the encrypted user information is stored after the authentication is passed;
The method comprises the steps that a back end of a portal system monitors whether a message exists in Kafka service, if the message exists, the back end of the portal system carries out real-time processing, encrypted user information is decrypted through a private key to obtain the user information, if the encrypted user information passes through the decryption, new addition judgment is carried out according to a unique identification number, such as a mobile phone number, if the mobile phone number of the user exists, no processing is carried out, if the mobile phone number of the user does not exist, the new addition processing is carried out, account number automatic binding is carried out on the user which is not bound with the back end of the portal system, a request for acquiring the token is initiated to the back end of the portal system according to AppKey, appSecret information, the token is returned, then a user interface of the back end of the portal system is requested to acquire the user information through carrying token and mobile phone number parameters, if the authentication passes through the authentication, the user information is returned, wherein the ID of the back end of the portal system is included, the unique identification number of the user is associated and the ID distributed by the back end of the portal system, so that the user has all access rights of a portal, all accounts are opened, and the account numbers are logged in a login operation mode.
The user can arrange the interface of the front end of the portal system, carry out portal position association binding according to the AppKey of the sub-portal system, and finally store the portal position association binding in a database, and when the front end of the portal system is inquired and rendered, the portal position is displayed according to the AppKey and the portal position, so that the arrangement effect is achieved.
Invoking the entire link through the portal system front end includes the steps of:
step 2.1, performing login authentication of the front end of the current portal system;
Specifically, the front end of the portal system obtains an authentication code authCode from the back end of the portal system according to CorpId, the front end of the portal system initiates authentication to the back end of the portal system according to authCode, the back end of the portal system receives a request, firstly initiates the request to a third party system according to Appkey and APPSECRET to obtain authentication AccessToken, then obtains user information according to AccessToken and authCode, including a back end ID of the portal system, finally obtains back end user information of the portal system according to the back end ID of the portal system, such as a mobile phone number, obtains portal user information according to the mobile phone number, passes authentication if the portal user information can be queried, and finally enters the corresponding portal system.
Step 2.2, after the front end authentication of the current portal system passes, acquiring portal list information including, but not limited to CorpId, a portal name, a portal URL and a URL of a portal logo from the rear end of the portal system, and performing front end rendering display;
Clicking any third party system, and acquiring an authentication code according to CorpId, wherein the front end of the third party system initiates authentication to the rear end of the third party system according to the authentication code, the rear end of the third party system receives a request, firstly initiates the request to the third party system according to AppKey, appSecret, acquires authentication accessToken, and then acquires user information according to accessToken and the authentication code;
And 2.4, acquiring a unique identifier corresponding to the back-end user information of the third-party system according to the user information, acquiring the user information of the third-party system according to the mobile phone, and finally entering a corresponding portal system to finish portal unification if the user information of the third-party system can be inquired.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.