[go: up one dir, main page]

CN117235400B - A unified multi-platform portal system based on Kafka technology - Google Patents

A unified multi-platform portal system based on Kafka technology

Info

Publication number
CN117235400B
CN117235400B CN202311133052.1A CN202311133052A CN117235400B CN 117235400 B CN117235400 B CN 117235400B CN 202311133052 A CN202311133052 A CN 202311133052A CN 117235400 B CN117235400 B CN 117235400B
Authority
CN
China
Prior art keywords
file
portal
portal system
kafka
authentication
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202311133052.1A
Other languages
Chinese (zh)
Other versions
CN117235400A (en
Inventor
蒋廷金
金江焕
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Anheng Information Security Technology Co Ltd
Original Assignee
Hangzhou Anheng Information Security Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Anheng Information Security Technology Co Ltd filed Critical Hangzhou Anheng Information Security Technology Co Ltd
Priority to CN202311133052.1A priority Critical patent/CN117235400B/en
Publication of CN117235400A publication Critical patent/CN117235400A/en
Application granted granted Critical
Publication of CN117235400B publication Critical patent/CN117235400B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Information Transfer Between Computers (AREA)

Abstract

本发明涉及一种基于Kafka技术统一多平台门户系统,门户系统前端包括若干用于展示和交互的第三方系统的前端页面,门户系统后端包括代理服务器、门户服务平台及对应的消息队列,门户服务平台通过代理服务器获取来自第三方系统的前端页面的业务请求并返回数据,采用基于Kafka技术的消息队列服务于来自第三方系统的注册和注销业务;基于增量同步实现第三方系统的用户信息的同步。本发明支持多个第三方系统作为门户系统前端,并通过门户系统后端实现统一;采用Kafka技术统一用户,实现智能注册和注销,并且支持第三方系统对门户系统前端的展示和编排;基于增量同步实现大文件的分片传输,确保了文件传输的效果和速率。

This invention relates to a unified multi-platform portal system based on Kafka technology. The portal system frontend includes several frontend pages from third-party systems for display and interaction. The portal system backend includes a proxy server, a portal service platform, and corresponding message queues. The portal service platform obtains business requests from the frontend pages of the third-party systems through the proxy server and returns data. A message queue based on Kafka technology serves the registration and deregistration processes from the third-party systems. Incremental synchronization is used to synchronize user information from the third-party systems. This invention supports multiple third-party systems as the portal system frontend and achieves unification through the portal system backend. Kafka technology unifies users, enabling intelligent registration and deregistration, and supports the display and orchestration of the portal system frontend by third-party systems. Incremental synchronization enables the fragmented transmission of large files, ensuring both the efficiency and speed of file transfer.

Description

Unified multi-platform portal system based on Kafka technology
Technical Field
The invention relates to the technical field of digital information transmission, in particular to a unified multi-platform portal system based on Kafka technology.
Background
Portal (Portal) originally points to doors and entrances and exits of houses, and is currently used for Portal sites of the Internet. Portal refers to an application system that leads to a comprehensive internet information resource of some kind and provides related information services. Along with the development of technology, concepts such as industry portals, enterprise information portals and the like appear.
With the development of society, on the basis of informatization construction of government departments, a cross-department comprehensive business application system, namely a government portal website is established, so that the public, enterprises and office staff can quickly and conveniently know business applications, organization contents and information of related government departments, and personalized services are obtained.
The government portal serves the masses, but a part of the government portal has special roles, namely, the staff needs to work cooperatively along with the difference of the level and the attribution, and also needs to obtain different operation rights, for example, a part of provincial departments with special requirements need to enter subsystems of various municipal levels through an interface to realize information viewing, and need to carry out personnel management and control through intelligent registration and cancellation through the rear end of the portal, and do not need to operate and manage through a third party system.
However, the current state is that the current government portal does not directly unify a plurality of third party systems, but unifies a plurality of sub-modules in the same system, and the current government portal does not have unified registration, logout and user association functions, and does not have the expandable functions of portal display arrangement, so that the current application requirements cannot be met.
Furthermore, for a special portal, such as a government portal, the interface of the user synchronization mechanism is not external, so that only equipment can be used for synchronization, and the encrypted file can be used for synchronization in a synchronization form, and the equipment has limitation on the size and the transmission rate of the file, so that the inconvenience of data migration is caused.
Disclosure of Invention
The invention solves the problems existing in the prior art and provides a unified multi-platform portal system based on Kafka technology.
The technical scheme adopted by the invention is that the unified multi-platform portal system based on Kafka technology comprises:
the front end of the portal system comprises a plurality of front end pages of a third party system for displaying and interacting;
The portal system back end comprises a proxy server, a portal service platform and a corresponding message queue, wherein the portal service platform acquires a service request from a front end page of a third party system through the proxy server and returns data, and adopts a message queue service based on Kafka technology to register and cancel the service third party system;
Incremental synchronization of user information of the third-party system to the portal service platform is achieved.
Preferably, the synchronization of the user information of the third party system comprises the steps of:
Step 1.1, creating an incremental file index table for recording uploading files of each business module in any third party system;
Step 1.2, collecting and processing user information and attachment data of each business module in the project;
setting a timing task increment synchronization strategy, and pushing file increment to a target server by a data source side based on a preset data segmentation rule on the premise of evaluating a network environment, wherein evaluation of the network environment can be based on judgment of network bandwidth, and each sub-volume is larger as the bandwidth is higher;
Setting a timed task loading file strategy, and when a target server detects file pushing behavior, carrying out integrity judgment on the file, and carrying out file loading based on a judgment result until a complete file is obtained;
and 1.5, decompressing the file into the corresponding catalogue according to the path in the file name.
Preferably, in step 1.2, the user newly-added or modified interface or method return value parameters are obtained, the type of the return value is judged, different strategies are adopted to analyze according to different return value types, unified user file information is obtained, a corresponding path URL is generated, the URL is processed based on a module rule strategy, a complete directory and a corresponding file path in a server are generated, an incremental file is generated according to the file path, and the incremental file information is recorded.
Preferably, the step 1.3 includes the steps of:
Step 1.3.1, acquiring a file synchronization task and acquiring the last file synchronization completion time;
Step 1.3.2, obtaining an uploading file record to be pushed after the last file synchronization completion time, backing up data, and generally deleting the backed up data after the file synchronization completion;
Step 1.3.3, carrying out asymmetric encryption on an original file by using a public key to obtain an encrypted file and compressing the encrypted file;
If the compressed encrypted file exceeds the threshold value, taking a model according to the file size and the threshold value, determining the total number of to-be-split volumes, performing splitting treatment on the compressed encrypted file, and performing the next step, otherwise, directly pushing the compressed encrypted file to a target server and returning;
Step 1.3.5, creating a temporary directory for the split volumes, splitting the compressed encrypted files and outputting the split volumes to obtain a split volume file set;
And step 1.3.6, finally, pushing the split file sets to the target server one by one, and recording pushing result information.
Preferably, the step 1.4 includes the steps of:
Step 1.4.1, the target server monitors the file synchronization message, if the file synchronization message is monitored, the synchronous file loading is carried out, all file sub-volumes are obtained according to the prefix matching of the file name, otherwise, the step 1.4.1 is repeated;
step 1.4.2, creating a temporary directory, and generating an original compressed file by dividing all files into volumes;
step 1.4.3, decompressing the compressed file to obtain an encrypted file, decrypting the encrypted file by using an asymmetric algorithm private key, and placing the file in a corresponding directory;
And step 1.4.4, the target server analyzes the user file, generates corresponding JavaBean information and stores the JavaBean information in a database.
Preferably, any one of the third party systems registers with the back end of the portal system through the registration code, and any one of the third party systems de-registers with the back end of the portal system through the registration code.
Preferably, any third party system carries an authentication parameter registration code, a user name and a password of Kafka and request parameters to request authentication to Kafka, the Kafka authenticates the user name and the password, an authentication result is returned after the authentication is passed, and the request is stored in the Kafka;
The portal service at the back end of the portal system monitors Kafka, when a registration message exists, a registration code is acquired to carry out registration code verification, request parameters are acquired after verification passes, a public key and a private key for synchronous authentication of the user information of the registered user are generated, the public key and the private key are stored at the back end of the portal system, the public key information is asynchronously transmitted back to a third party system through a public key callback interface URL, and the third party system stores the public key information.
Preferably, when the users are synchronous, the third party system inquires local user information, records the inquiring time as the next user inquiring time parameter, obtains the public key encryption after the user information is obtained, and sends the encrypted user information to the Kafka service with Kafka authentication, the Kafka carries out login authentication, and the encrypted user information is stored after the authentication is passed;
The method comprises the steps of monitoring whether a Kafka service has a message or not by the back end of a portal system, decrypting encrypted user information by a private key if the Kafka service has the message to obtain the user information, carrying out new judgment according to a unique identification number, carrying out account automatic binding on a user which is not bound with the back end of the portal system, and carrying out association binding on the unique identification number of the user and an ID distributed by the back end of the portal system.
Preferably, any third party system carries CorpId, appKey, appSecret, a portal access URL, a public key callback interface URL, and Logo as request parameters.
Preferably, invoking the entire link through the portal system front end comprises the steps of:
step 2.1, performing login authentication of the front end of the current portal system;
Step 2.2, after the front end authentication of the current portal system is passed, acquiring portal list information from the back end of the portal system, and performing front end rendering display;
Clicking any third party system, and acquiring an authentication code according to CorpId, wherein the front end of the third party system initiates authentication to the rear end of the third party system according to the authentication code, the rear end of the third party system receives a request, firstly initiates the request to the third party system according to AppKey, appSecret, acquires authentication accessToken, and then acquires user information according to accessToken and the authentication code;
And 2.4, acquiring a unique identifier corresponding to the back-end user information of the third-party system according to the user information, acquiring the user information of the third-party system according to the mobile phone, and finally entering a corresponding portal system to finish portal unification if the user information of the third-party system can be inquired.
The invention relates to a unified multi-platform portal system based on Kafka technology, wherein the front end of the portal system comprises a plurality of front end pages of a third party system for display and interaction, the rear end of the portal system comprises a proxy server, a portal service platform and a corresponding message queue, the portal service platform acquires a service request from the front end pages of the third party system through the proxy server and returns data, the message queue based on Kafka technology is adopted to serve registration and cancellation services from the third party system, and synchronization of user information of the third party system is realized based on incremental synchronization.
The invention has the beneficial effects that:
(1) Supporting a plurality of third party systems as the front end of the portal system, and realizing unification through the rear end of the portal system;
(2) Unified user by Kafka technology, intelligent registration and cancellation are realized, and the third party system is supported to display and arrange the front end of the portal system;
(3) And the large file is transmitted in a fragmentation way based on incremental synchronization, so that the effect and the speed of file transmission are ensured.
Drawings
FIG. 1 is a schematic diagram of a system architecture of the present invention;
FIG. 2 is a flow chart of a method for synchronizing user information of a third party system according to the present invention;
FIG. 3 is a flow chart of a method for invoking an entire link through a portal system front end in the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention relates to a unified multi-platform portal system based on Kafka technology, which unifies application inlets of platforms of a plurality of third party systems, opens each platform to enable the platforms to have access rights to each other, opens user information of each platform by adopting Kafka technology, completes portal registration and portal cancellation, and supports portal display position arrangement.
In the invention, kafka is applied to the system as a high-throughput distributed publish-subscribe message system, can process all action stream data of consumers in websites, including but not limited to web browsing, searching and actions of other users, unifies online and offline message processing through a Hadoop parallel loading mechanism, and provides real-time messages through clusters.
The system comprises:
the front end of the portal system comprises a plurality of front end pages of a third party system for displaying and interacting;
the portal service platform acquires a service request from a front page of a third party system through a proxy server and returns data, and adopts a message queue service based on Kafka technology to register and cancel the service third party system;
Incremental synchronization of user information of the third-party system to the portal service platform is achieved.
In the invention, in order to ensure the applicability of the third party system, a timing heartbeat technology is generally adopted to monitor whether the front end of each portal system is online.
In the invention, in order to ensure the consistency of data and the real-time and reliability of synchronization in the synchronization process, user information including file synchronization is performed in an incremental mode, when the file is a large file, the large file is segmented and rolled, the segmented and rolled into an original file at a receiving party, the data are finally analyzed and synchronously stored, and after the data are synchronized to a target server, the segmented and rolled into the original file which is placed in a corresponding catalog.
In the implementation of the present invention, the synchronization of the user information of the third party system includes the following steps:
Step 1.1, creating an incremental file index table for recording uploading files of each business module in any third party system;
Step 1.2, collecting and processing user information and attachment data of each business module in the project;
The attachment data uploaded by each business module is collected by combining the section (UploadFileAspect) and the annotation (UploadFileRecord), wherein the annotation only acts on the method for uploading the file, and UploadFileAspect is a concrete implementation flow of the annotation.
Specifically, in step 1.2, obtaining new or modified interface or method return value parameters of the user, judging the type of the return value, analyzing by adopting different strategies according to different return value types, such as String, javabean types, and the like, obtaining unified user file information and generating a corresponding path URL, processing the URL based on a module rule strategy, and generating a complete directory and a corresponding file path in the server;
For example, when the return value is of String type, the return value file path is directly returned, i.e. the return value is directly used as the file path, when the return value is just one object, the object attribute is id, url, file name, etc., the strategy of obtaining the file path is to analyze the return value, the original object is generated by the java reflection mechanism, and finally the url is obtained as the file path according to the object.
And generating an incremental file according to the file path, wherein the incremental file is stored in the form of a compressed file, and the name of the compressed file is generally a file storage path, so that a target server conveniently loads the file and places the file in a corresponding directory, and incremental file information including, but not limited to, a service module, the file name, a file storage path, the compressed file path, file uploading time, creator information and the like is recorded for later file synchronization.
Setting a timing task increment synchronization strategy, and pushing file increment to a target server by a data source side based on a preset data segmentation rule on the premise of evaluating a network environment;
the step 1.3 comprises the following steps:
Step 1.3.1, acquiring a file synchronization task and acquiring the last file synchronization completion time;
Step 1.3.2, obtaining an uploading file record to be pushed which is created after the last file synchronization completion time (the creation time is later than the last synchronization completion time) from a t_uploading_file_record table, and preventing the data backup from being lost;
Step 1.3.3, carrying out asymmetric encryption on an original file by using a public key to obtain an encrypted file and compressing the encrypted file to obtain a zip file package ASSET-fileName.zip;
If the compressed encrypted file exceeds the threshold value, taking a model according to the file size and the threshold value, determining the total number of to-be-split volumes, performing splitting treatment on the compressed encrypted file, and performing the next step, otherwise, directly pushing the compressed encrypted file to a target server and returning;
Step 1.3.5, creating a temporary directory ASSET-fileName-X for the split volumes, splitting the compressed encrypted files and outputting the split volumes to obtain split volume file sets, ASSET-fileName.z1, ASSET-fileName.z2 and the like;
And step 1.3.6, finally pushing the partitioned file sets to the target server one by one, and recording pushing result information, wherein the number of retries of failed pushing is generally 3.
Setting a timed task loading file strategy, when a target server detects file pushing behavior, carrying out integrity judgment on the file, and carrying out file loading based on a judgment result until the complete file is obtained, namely firstly judging whether the file is a split and split file, if so, carrying out file folding and folding processing to generate an original file, and then carrying out loading, otherwise, directly loading, and finally decompressing the file into a corresponding directory according to a path in a file name.
The step 1.4 comprises the following steps:
step 1.4.1, the target server monitors the file synchronization message, if the file synchronization message is monitored, the synchronous file loading is carried out, and all files are obtained in a split mode according to the prefix matching of the file names, such as ASSET-fileName.z1 and ASSET-fileName.z2, and if not, the step 1.4.1 is repeated;
In practical application, after file synchronization is completed, the user record in the incremental file index table can be marked that the file synchronization is completed, the incremental file index table is synchronized to a third party system side, and the third party system side only needs to process the marked user file which is completed synchronously until the third party system side receives all the sub-volumes based on the index table, namely receives all the sub-volumes of the file, and does not need to judge any more;
step 1.4.3, decompressing the compressed file to obtain an encrypted file, decrypting the encrypted file by using an asymmetric algorithm private key, and placing the file in a corresponding directory;
And step 1.4.4, the target server analyzes the user file, generates corresponding JavaBean information and stores the JavaBean information in a database.
And 1.5, decompressing the file into the corresponding catalogue according to the path in the file name.
In the invention, a portal service platform at the back end of the portal system provides the following services:
distributing portal registration codes, namely generating portal registration codes to be used when a third party system is registered before portal registration, and simultaneously giving out a user name and a secret key of Kafka;
Portal registration, wherein a third party system carries a registration code to carry out portal registration, and the registration needs to be uploaded CorpId, appKey, appSecret and stored, and Logo, system name and the like can be uploaded and displayed;
Portal cancellation, namely cancellation is carried out on registered portals, and authentication cancellation is carried out by carrying registration codes when cancellation is carried out;
user synchronization, integrating user information of all third party systems in a portal service platform;
portal management, which manages registered portals, such as modification, deletion, etc.;
portal arrangement, which manages the display position, style, logo and the like of the portal system.
It should be noted in advance that, in the system of the present invention, any third party system carries CorpId, appKey, appSecret, a portal access URL, a public key callback interface URL, and a Logo as request parameters, where the Logo is in a binary format or in a base64 string form.
Any one of the third party systems registers to the back end of the portal system through the registration code, and any one of the third party systems logs out to the back end of the portal system through the registration code.
Specifically, any third party system carries an authentication parameter registration code, a user name and a password of Kafka and request parameters to request authentication to Kafka, the Kafka authenticates the user name and the password, an authentication result is returned after the authentication is passed, and the request is stored in the Kafka for consumption of other services;
The portal service at the back end of the portal system monitors Kafka, when a registration message exists, the message is processed in real time, registration codes are acquired to check the registration codes, request parameters are acquired through check, corpId, appKey, appSecret and Logo are obtained, corpId, appKey, appSecret is stored, binary format or base64 character strings of the Logo are converted into texts, file paths are generated and stored, public keys and private keys for synchronous authentication of the user information of the registered user are generated, the public keys and the private keys are stored at the back end of the portal system, the public key information is asynchronously returned to a third party system through a public key callback interface URL, and the third party system stores the public key information.
Obviously, the third party system flow of cancellation is the same as portal registration, a cancellation request is initiated with a registration code and a public key, a private key can be obtained by inquiring according to the registration code and the public key, and portal deletion cancellation operation is performed.
When the user is synchronous, the third party system inquires local user information, namely increment inquiry, records the inquiry time as the next user inquiry time parameter to obtain user information and encrypts the public key, and sends the encrypted user information to the Kafka service with Kafka authentication, wherein the Kafka carries out login authentication, and the encrypted user information is stored after the authentication is passed;
The method comprises the steps that a back end of a portal system monitors whether a message exists in Kafka service, if the message exists, the back end of the portal system carries out real-time processing, encrypted user information is decrypted through a private key to obtain the user information, if the encrypted user information passes through the decryption, new addition judgment is carried out according to a unique identification number, such as a mobile phone number, if the mobile phone number of the user exists, no processing is carried out, if the mobile phone number of the user does not exist, the new addition processing is carried out, account number automatic binding is carried out on the user which is not bound with the back end of the portal system, a request for acquiring the token is initiated to the back end of the portal system according to AppKey, appSecret information, the token is returned, then a user interface of the back end of the portal system is requested to acquire the user information through carrying token and mobile phone number parameters, if the authentication passes through the authentication, the user information is returned, wherein the ID of the back end of the portal system is included, the unique identification number of the user is associated and the ID distributed by the back end of the portal system, so that the user has all access rights of a portal, all accounts are opened, and the account numbers are logged in a login operation mode.
The user can arrange the interface of the front end of the portal system, carry out portal position association binding according to the AppKey of the sub-portal system, and finally store the portal position association binding in a database, and when the front end of the portal system is inquired and rendered, the portal position is displayed according to the AppKey and the portal position, so that the arrangement effect is achieved.
Invoking the entire link through the portal system front end includes the steps of:
step 2.1, performing login authentication of the front end of the current portal system;
Specifically, the front end of the portal system obtains an authentication code authCode from the back end of the portal system according to CorpId, the front end of the portal system initiates authentication to the back end of the portal system according to authCode, the back end of the portal system receives a request, firstly initiates the request to a third party system according to Appkey and APPSECRET to obtain authentication AccessToken, then obtains user information according to AccessToken and authCode, including a back end ID of the portal system, finally obtains back end user information of the portal system according to the back end ID of the portal system, such as a mobile phone number, obtains portal user information according to the mobile phone number, passes authentication if the portal user information can be queried, and finally enters the corresponding portal system.
Step 2.2, after the front end authentication of the current portal system passes, acquiring portal list information including, but not limited to CorpId, a portal name, a portal URL and a URL of a portal logo from the rear end of the portal system, and performing front end rendering display;
Clicking any third party system, and acquiring an authentication code according to CorpId, wherein the front end of the third party system initiates authentication to the rear end of the third party system according to the authentication code, the rear end of the third party system receives a request, firstly initiates the request to the third party system according to AppKey, appSecret, acquires authentication accessToken, and then acquires user information according to accessToken and the authentication code;
And 2.4, acquiring a unique identifier corresponding to the back-end user information of the third-party system according to the user information, acquiring the user information of the third-party system according to the mobile phone, and finally entering a corresponding portal system to finish portal unification if the user information of the third-party system can be inquired.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims (7)

1.一种基于Kafka技术统一多平台门户系统,其特征在于:所述系统包括:1. A unified multi-platform portal system based on Kafka technology, characterized in that: the system includes: 门户系统前端,包括若干用于展示和交互的第三方系统的前端页面;The portal system front-end includes several front-end pages of third-party systems used for display and interaction; 门户系统后端,包括代理服务器、门户服务平台及对应的消息队列,门户服务平台通过代理服务器获取来自第三方系统的前端页面的业务请求并返回数据,采用基于Kafka技术的消息队列服务注册和注销业务第三方系统;The portal system backend includes a proxy server, a portal service platform, and a corresponding message queue. The portal service platform obtains business requests from the front-end pages of third-party systems through the proxy server and returns data. It uses a message queue service based on Kafka technology to register and deregister business third-party systems. 任一所述第三方系统通过注册码向门户系统后端进行注册,任一所述第三方系统通过注册码向门户系统后端进行注销;Any of the aforementioned third-party systems registers with the portal system backend using a registration code, and any of the aforementioned third-party systems deregisters with the portal system backend using a registration code; 任一第三方系统携带认证参数注册码、Kafka的用户名和密码及请求参数向Kafka请求认证,Kafka对用户名和密码进行认证,认证通过后返回认证结果,并将请求存储在Kafka中;Any third-party system sends an authentication request to Kafka with the authentication parameter registration code, Kafka username and password, and request parameters. Kafka authenticates the username and password, returns the authentication result after successful authentication, and stores the request in Kafka. 门户系统后端的门户服务平台对Kafka进行监听,当有注册消息时,获取注册码进行注册码校验,校验通过则获取请求参数,生成该第三方系统用户信息同步认证的公钥和私钥,门户系统后端存储公钥和私钥,异步将该公钥信息通过公钥回调接口URL回传给第三方系统,第三方系统进行保存存储;The portal service platform on the backend of the portal system listens to Kafka. When a registration message is received, it obtains the registration code and verifies it. If the verification is successful, it obtains the request parameters and generates a public key and private key for the third-party system's user information synchronization authentication. The portal system backend stores the public key and private key and asynchronously sends the public key information back to the third-party system through the public key callback interface URL. The third-party system then saves and stores the public key. 用户同步时,第三方系统查询本地用户信息,记录本次查询时间作为下次用户查询时间参数,得到用户信息后公钥加密;将加密后的用户信息携带Kafka认证发送到Kafka服务上,Kafka进行登录认证,认证通过后存储加密后用户信息;When a user synchronizes, the third-party system queries the local user information, records the query time as the parameter for the next user query time, and encrypts the user information with the public key. The encrypted user information is then sent to the Kafka service along with Kafka authentication. Kafka performs login authentication, and stores the encrypted user information after successful authentication. 门户系统后端监听Kafka服务是否有消息,有消息则以私钥对加密的用户信息解密,得到用户信息;根据唯一识别号进行新增判断,对没有绑定门户系统后端的用户进行账号自动绑定,用户唯一识别号与门户系统后端分配的ID进行关联绑定;The portal system backend listens for messages on the Kafka service. If a message is found, the encrypted user information is decrypted using the private key to obtain the user information. The system then makes a new addition decision based on the unique identifier and automatically binds accounts to users who are not bound to the portal system backend. The user's unique identifier is associated with the ID assigned by the portal system backend. 实现第三方系统的用户信息至门户服务平台的增量同步。Enable incremental synchronization of user information from third-party systems to the portal service platform. 2.根据权利要求1所述的一种基于Kafka技术统一多平台门户系统,其特征在于:所述第三方系统的用户信息的同步包括以下步骤:2. The unified multi-platform portal system based on Kafka technology according to claim 1, characterized in that: the synchronization of user information from the third-party system includes the following steps: 步骤1.1:创建增量文件索引表,用于记录任一第三方系统中各个业务模块的上传文件;Step 1.1: Create an incremental file index table to record the uploaded files of each business module in any third-party system; 步骤1.2:收集并处理项目中各个业务模块的用户信息及附件数据;Step 1.2: Collect and process user information and attachment data from various business modules in the project; 步骤1.3:设定定时任务增量同步策略,在评估网络环境的前提下,基于预设的数据分割规则,由数据源方将文件增量推送至目标服务器;Step 1.3: Set up a scheduled task for incremental synchronization. Under the premise of evaluating the network environment, the data source pushes the file incrementally to the target server based on the preset data segmentation rules. 步骤1.4:设定定时任务加载文件策略,当目标服务器检测到文件推送行为,对此文件进行完整性判断,基于判断结果进行文件加载,直至获得完整文件;Step 1.4: Set a scheduled task to load files. When the target server detects a file push, it performs an integrity check on the file and loads the file based on the check result until the complete file is obtained. 步骤1.5:根据文件名中路径将文件解压到对应目录中。Step 1.5: Extract the file to the corresponding directory based on the path in the filename. 3.根据权利要求2所述的一种基于Kafka技术统一多平台门户系统,其特征在于:步骤1.2中,获取用户新增或修改接口或方法返回值参数,判断返回值的类型,根据不同返回值类型采取不同策略进行解析,得到统一的用户文件信息并生成对应的路径URL,基于模块规则策略处理URL,生成服务器中完整的目录及对应的文件路径;根据文件路径生成增量文件,记录增量文件信息。3. A unified multi-platform portal system based on Kafka technology according to claim 2, characterized in that: in step 1.2, the return value parameters of the user's new or modified interface or method are obtained, the type of the return value is determined, different strategies are adopted for parsing according to different return value types, unified user file information is obtained and the corresponding path URL is generated, the URL is processed based on the module rule strategy, and the complete directory and corresponding file path in the server are generated; incremental files are generated according to the file paths, and incremental file information is recorded. 4.根据权利要求2所述的一种基于Kafka技术统一多平台门户系统,其特征在于:所述步骤1.3包括以下步骤:4. A unified multi-platform portal system based on Kafka technology according to claim 2, characterized in that: step 1.3 includes the following steps: 步骤1.3.1:获取文件同步任务,获取上次文件同步完成时间;Step 1.3.1: Obtain the file synchronization task and get the last file synchronization completion time; 步骤1.3.2:获取创建于上次文件同步完成时间后的需要推送的上传文件记录,数据备份;Step 1.3.2: Obtain the records of uploaded files that need to be pushed after the last file synchronization completion time, and back up the data; 步骤1.3.3:对原始文件以公钥进行非对称加密,得到加密文件并压缩;Step 1.3.3: Encrypt the original file asymmetrically using the public key to obtain the encrypted file and compress it; 步骤1.3.4:若压缩后的加密文件超过阈值,则根据文件大小与阈值取模,确定待分卷总数,对此压缩后的加密文件进行分卷处理,进行下一步,否则,直接推送此压缩后的加密文件至目标服务器后返回;Step 1.3.4: If the compressed encrypted file exceeds the threshold, determine the total number of volumes to be split based on the modulo of the file size and the threshold, split the compressed encrypted file into volumes, and proceed to the next step; otherwise, directly push the compressed encrypted file to the target server and return. 步骤1.3.5:为分卷创建临时目录,对压缩后的加密文件进行分卷并输出,得到分卷文件集合;Step 1.3.5: Create a temporary directory for the split files, split the compressed encrypted files into multiple parts and output them to obtain a set of split files; 步骤1.3.6:最后将分卷文件集合逐个推送至目标服务器,记录推送结果信息。Step 1.3.6: Finally, push the multi-volume file set to the target server one by one and record the push result information. 5.根据权利要求2所述的一种基于Kafka技术统一多平台门户系统,其特征在于:所述步骤1.4包括以下步骤:5. A unified multi-platform portal system based on Kafka technology according to claim 2, characterized in that: step 1.4 includes the following steps: 步骤1.4.1:目标服务器监听文件同步消息,若监听到文件同步消息,则进行同步文件加载,根据文件名前缀匹配获取所有文件分卷,否则重复步骤1.4.1;Step 1.4.1: The target server listens for file synchronization messages. If a file synchronization message is detected, the server loads the synchronized files and obtains all file volumes based on the filename prefix. Otherwise, the server repeats step 1.4.1. 步骤1.4.2:创建临时目录,以所有文件分卷生成原始压缩文件;Step 1.4.2: Create a temporary directory and generate the original compressed file by splitting all files into multiple volumes; 步骤1.4.3:解压所述压缩文件后得到加密文件,使用非对称算法私钥进行解密,并把文件放置对应目录中;Step 1.4.3: After decompressing the compressed file, an encrypted file is obtained. The encrypted file is then decrypted using an asymmetric algorithm private key, and the file is placed in the corresponding directory. 步骤1.4.4:目标服务器解析用户文件,生成对应的JavaBean信息,存储至数据库中。Step 1.4.4: The target server parses the user file, generates the corresponding JavaBean information, and stores it in the database. 6.根据权利要求1所述的一种基于Kafka技术统一多平台门户系统,其特征在于:任一第三方系统携带CorpId、AppKey、AppSecret、门户访问URL、公钥回调接口URL以及Logo作为请求参数。6. A unified multi-platform portal system based on Kafka technology according to claim 1, characterized in that: any third-party system carries CorpId, AppKey, AppSecret, portal access URL, public key callback interface URL and Logo as request parameters. 7.根据权利要求6所述的一种基于Kafka技术统一多平台门户系统,其特征在于:通过门户系统前端调用整个链路包括以下步骤:7. A unified multi-platform portal system based on Kafka technology according to claim 6, characterized in that: calling the entire link through the portal system frontend includes the following steps: 步骤2.1:进行当前门户系统前端的登录认证;Step 2.1: Perform login authentication for the current portal system frontend; 步骤2.2:当前门户系统前端认证通过后,向门户系统后端获取门户列表信息,进行前端渲染展示;Step 2.2: After the current portal system's front-end authentication is successful, the portal system retrieves the portal list information from the back-end and renders and displays it on the front end. 步骤2.3:点击任意第三方系统,根据CorpId获取认证码;门户系统前端根据认证码向门户系统后端发起认证,门户系统后端接收到请求,首先根据AppKey、AppSecret向第三方系统发起请求,获取认证accessToken,然后根据accessToken和认证码获取用户信息;Step 2.3: Click on any third-party system and obtain the authentication code based on the CorpId; the portal system frontend initiates authentication to the portal system backend based on the authentication code. Upon receiving the request, the portal system backend first initiates a request to the third-party system based on the AppKey and AppSecret to obtain the authentication accessToken, and then obtains the user information based on the accessToken and the authentication code. 步骤2.4:根据用户信息获取门户系统后端用户信息对应的唯一标识号,并根据唯一标识号得到第三方系统的用户信息,若能查询到第三方系统的用户信息则认证通过,最终进入对应的门户系统当中,完成门户统一。Step 2.4: Obtain the unique identifier corresponding to the user information in the portal system backend based on the user information, and obtain the user information of the third-party system based on the unique identifier. If the user information of the third-party system can be found, the authentication is successful, and the user finally enters the corresponding portal system to complete the portal unification.
CN202311133052.1A 2023-09-04 2023-09-04 A unified multi-platform portal system based on Kafka technology Active CN117235400B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311133052.1A CN117235400B (en) 2023-09-04 2023-09-04 A unified multi-platform portal system based on Kafka technology

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311133052.1A CN117235400B (en) 2023-09-04 2023-09-04 A unified multi-platform portal system based on Kafka technology

Publications (2)

Publication Number Publication Date
CN117235400A CN117235400A (en) 2023-12-15
CN117235400B true CN117235400B (en) 2026-01-09

Family

ID=89081718

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311133052.1A Active CN117235400B (en) 2023-09-04 2023-09-04 A unified multi-platform portal system based on Kafka technology

Country Status (1)

Country Link
CN (1) CN117235400B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117527678A (en) * 2024-01-04 2024-02-06 贵州卓讯软件股份有限公司 Data exchange center and method

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113742700A (en) * 2021-11-08 2021-12-03 中国工程物理研究院计算机应用研究所 Cross-domain software system integration method based on portal
CN115001777A (en) * 2022-05-25 2022-09-02 公安部户政管理研究中心 Cross-service portal system management method

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8452959B2 (en) * 2009-02-24 2013-05-28 Research In Motion Limited Method and system for registering a presence user with a presence service
CN111555957B (en) * 2020-03-26 2022-08-19 孩子王儿童用品股份有限公司 Kafka-based synchronous message service system and implementation method
CN115842733A (en) * 2021-08-10 2023-03-24 中兴通讯股份有限公司 Resource sharing method, micro front-end system, application and computer readable storage medium
US12302127B2 (en) * 2021-09-27 2025-05-13 Omnissa, Llc Management service device platform creation and device configuration
CN114745164B (en) * 2022-03-25 2024-05-03 中国建设银行股份有限公司 Service processing method, device, electronic equipment and computer readable medium
CN115314509B (en) * 2022-07-27 2024-12-17 上海浦东发展银行股份有限公司 Synchronization method, device, equipment and storage medium of application registration information
CN115913682A (en) * 2022-11-04 2023-04-04 杭州安恒信息安全技术有限公司 Open interface processing method, device, system, electronic device and storage medium
CN116108420A (en) * 2023-02-23 2023-05-12 国核信息科技有限公司 Enterprise intranet portal system based on multi-service system integration and working method thereof

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113742700A (en) * 2021-11-08 2021-12-03 中国工程物理研究院计算机应用研究所 Cross-domain software system integration method based on portal
CN115001777A (en) * 2022-05-25 2022-09-02 公安部户政管理研究中心 Cross-service portal system management method

Also Published As

Publication number Publication date
CN117235400A (en) 2023-12-15

Similar Documents

Publication Publication Date Title
CN110971655B (en) Offline client playback and synchronization
CN110413595B (en) Data migration method applied to distributed database and related device
US11553196B2 (en) Media storage
JP2022529967A (en) Extracting data from the blockchain network
US8838679B2 (en) Providing state service for online application users
CN108491267B (en) Method and apparatus for generating information
CN110400221A (en) Data processing method, system, storage medium and computer equipment
CN103780700A (en) Application system and method for achieving compatibility and sharing among multi-source heterogeneous systems
Srivastava et al. Analysis of data extraction and data cleaning in web usage mining
CN106557269A (en) The method and apparatus of storage cloud disk resource
CN104992124A (en) Document safety access method for cloud storage environment
US9876776B2 (en) Methods for generating and publishing a web site based on selected items and devices thereof
CN112100534B (en) Information processing method, device, medium and electronic equipment in page sharing
CN114816499B (en) Electronic resource sharing method, device and storage medium
US20240020420A1 (en) Tamper-evident storage and provisioning of media streams
CN102946404B (en) At method and the device of user side display selectivity synchronization folder
CN117235400B (en) A unified multi-platform portal system based on Kafka technology
Schlepphorst et al. Digital forensic approaches for cloud service models: A survey
Cantelli-Forti et al. WEFT: a consistent and tamper-proof methodology for acquisition of automatically verifiable forensic web evidence
Kao Cybercrime investigation countermeasure using created-accessed-modified model in cloud computing environments
CN119690560A (en) Cross-border data multi-terminal display interaction system based on digital cockpit and its interaction method
CN116886758A (en) Cloud storage information security management method, device, system and storage medium
KR20160132854A (en) Asset collection service through capture of content
US20110219077A1 (en) Direct linked two way forms
CN115098837A (en) Data processing method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant