CN119988168A

CN119988168A - A web page performance monitoring and analysis method, system, device and storage medium

Info

Publication number: CN119988168A
Application number: CN202510068195.1A
Authority: CN
Inventors: 王晶
Original assignee: Space Shichuang Chongqing Technology Co ltd
Current assignee: Space Shichuang Chongqing Technology Co ltd
Priority date: 2025-01-16
Filing date: 2025-01-16
Publication date: 2025-05-13

Abstract

The invention discloses a webpage performance monitoring and analyzing method, a system, a device and a storage medium, relates to the technical field of webpage monitoring, and aims to overcome the limitation of the existing webpage performance monitoring technology. The method comprises the following steps of constructing a high-availability infrastructure to cope with large-scale concurrent access, collecting front-end performance data in real time, adopting a stream processing technology and a real-time algorithm to clean and aggregate, analyzing the data by using a machine learning model, mining performance trend and abnormality, establishing an intelligent reference, starting a scene intelligent adaptation module, accurately identifying a webpage application scene, dynamically customizing a monitoring strategy, constructing a monitoring system covering data collection, analysis, reporting, alarming and optimizing advice, optimizing a storage medium, and ensuring data integrity and accessibility. The method has the advantages of providing diversified monitoring indexes, intelligent and accurate prediction performance, high-efficiency real-time feedback and dynamic optimization of resource utilization, remarkably improving website performance and user experience, and providing firm guarantee for stable and efficient operation of the website.

Description

Webpage performance monitoring and analyzing method, system, device and storage medium

Technical Field

The present invention relates to the field of web page monitoring technologies, and in particular, to a web page performance monitoring analysis method, system, device, and storage medium.

Background

With rapid progress in internet technology, websites have become an important platform for enterprises to show their brands, provide services and products to the outside world. With the increase of the access amount of users and the improvement of the complexity of website functions, webpage performance problems become challenges which cannot be ignored gradually, such as problems of slow page loading speed, slow response, operation blocking and the like, so that user experience is reduced, and user loss and lower conversion rate are easy to cause. Therefore, web page performance monitoring becomes an essential ring to ensure that websites run smoothly and efficiently.

Currently, there are many tools and solutions for monitoring web page performance in the market, and they mainly depend on different technical means to monitor and optimize web page performance. The following are several typical prior art techniques and their shortcomings:

(1) YSlow and Pingdom, which mainly focus on few performance parameters such as page loading time and response time, and lack comprehensive consideration of multidimensional performance parameters such as user interaction experience, error occurrence frequency, user jump-out proportion and the like;

(2) Google analysis can acquire massive user behavior data, but has limitation on further data analysis capability, and the mode and trend of hiding behind the data are difficult to reveal;

(3) GTmetrix, although providing a detailed performance report, the real-time feedback mechanism is not sound, but has the defects in the aspect of real-time feedback, and the actual running state of the current webpage can not be reflected immediately;

(4) The dynamic optimization capability is lacking, most of the prior art stays in the performance monitoring and problem diagnosis stage, and the capability of automatically adjusting and optimizing the webpage performance according to the monitoring result is lacking, so that the manual intervention is needed to perform the intervention.

Therefore, there is a need for a more comprehensive, efficient, intelligent web page performance monitoring and analysis method, system, apparatus, and storage medium. The novel technology can monitor and collect comprehensive performance index data of the webpage in various stages of loading, rendering, interaction and the like in real time, and uses advanced data analysis algorithm and machine learning technology to carry out deep mining and analysis, discover potential performance problems and optimization space, and provide real-time early warning and dynamic optimization functions. By the method, the performance of the website can be remarkably improved, the user experience is ensured, and powerful support is provided for the stable and efficient operation of the website.

Disclosure of Invention

Aiming at the defects of the prior art, the invention provides a webpage performance monitoring and analyzing method, a system, a device and a storage medium. The invention aims at integrating a real-time insight and intelligent prediction engine, a precise abnormality detection and early warning system and a scene intelligent adaptation and dynamic optimization strategy to create a webpage performance monitoring analysis method, a system, a device and a storage medium. The innovation aims to thoroughly innovate a webpage performance monitoring mode, accurately predicts future trend by capturing performance data in real time, sensitively detects and responds to abnormal events in real time, and dynamically adjusts an optimization strategy according to a user access scene to realize omnibearing intelligent management of webpage performance.

Specifically, the invention improves the instantaneity, the accuracy and the foresight of monitoring, and the client can quickly master the real-time state of the webpage performance, and forecast and prevent the potential problems. Meanwhile, through accurate anomaly detection and automatic early warning mechanisms, the anomaly response time is greatly shortened, and potential losses are reduced. The intelligent adaptation of the scene is emphasized, the optimization measures are dynamically adjusted according to different user access scenes, and smoother and personalized access experience is provided for the users. And finally, providing comprehensive and clear performance overview and decision support for clients through visual and easy-to-use visual interfaces, assisting accurate decision and optimizing website operation strategies.

In order to achieve the above purpose, a method, a system, a device and a storage medium for monitoring and analyzing webpage performance are realized by the following technical scheme:

The method comprises the steps of S1, system deployment and configuration, namely, constructing a high-availability infrastructure in a cloud server or a local data center, namely, deploying a high-performance server, an SSD storage array and a high-speed network switching device, enabling hardware resources to fully meet the requirements of large-scale concurrent access and data processing, configuring an operating system to conduct safety reinforcement, guaranteeing the safety and stability of the system, deploying a Java running environment and a database management system (MySQL/PostgreSQL), and installing necessary middleware services APACHE HIVE for data warehouse and Apache flash for data stream processing. The APACHE KAFKA clusters are configured again to process real-time data streams, and low-delay stream data processing is realized by deployment APACHE FLINK;

s2, data acquisition and intelligent analysis, wherein performance data of a front-end page is acquired in real time through a front-end monitoring tool (SDK), wherein the performance data comprise page loading time, first Contact Time (FCT), time expiration rate (TLP), user interaction time, error rate, jump rate, throughput and the like. And then, performing primary cleaning and aggregation operation on the acquired data by using a stream processing technology and a real-time data analysis algorithm, and automatically analyzing the processed data by using a machine learning model to mine performance trends and potential anomalies. Training a deep learning model by using historical performance data, constructing an intelligent reference model, integrating an abnormal detection algorithm matrix such as a statistical method, cluster analysis and time sequence pattern recognition, and describing a normal fluctuation range of performance indexes;

And S3, generating scene intelligent adaptation and monitoring strategies, starting a scene intelligent adaptation module, and accurately identifying the webpage application scene by using machine learning or a rule engine. And dynamically customizing performance monitoring strategies and indexes according to scene recognition results, and providing personalized monitoring schemes for different webpage applications. Continuously optimizing a monitoring strategy to ensure the accuracy and pertinence of monitoring;

And S4, dynamically optimizing strategy and resource management, designing a dynamic optimizing strategy engine, analyzing monitoring data in real time, adjusting website resources, dynamically adjusting parameters of the optimizing strategy engine according to the monitoring result, and implementing resource loading priority adjustment, picture compression optimization and cache management strategy. Ensuring that the website maintains optimal performance under different network environments and user behaviors;

And S5, constructing a performance monitoring system, namely constructing a webpage performance monitoring system based on the monitoring analysis method, wherein the webpage performance monitoring system comprises a data acquisition module, an analysis module, a report generation module, an alarm module and an optimization suggestion module. The system comprises a data acquisition module, an analysis module, a report generation module, an alarm module and an optimization suggestion module, wherein the data acquisition module is used for acquiring loading data and user interaction data of a webpage in real time, the analysis module is used for carrying out deep analysis on the data by using an anomaly detection algorithm matrix and a model, the report generation module is used for timely generating a detailed performance report according to an analysis result, the alarm module is used for setting a performance threshold value and automatically triggering an alarm or early warning mechanism when anomaly is detected, and the optimization suggestion module is used for providing optimization suggestions according to the analysis result. The system continuously adjusts the monitoring and optimizing strategy according to the feedback and optimizing effect of the user to form a closed-loop continuous performance optimizing flow;

And S6, optimizing a storage medium, storing performance data by using a reliable storage medium, and ensuring the integrity and accessibility of the data. And storing the real-time performance data on the SSD and storing the history data in the HDD or cloud storage to improve the overall storage efficiency. And (3) carrying out backup on the key data at regular intervals, and carrying out recovery test at regular intervals to ensure the effectiveness and the recoverability of the backup data. And the performance indexes (read-write speed, delay, throughput and the like) of the storage medium are monitored in real time, and potential problems are found and solved in time.

Preferably, in the step S1, the method further includes:

A APACHE KAFKA cluster formed by a plurality of nodes is deployed in a cloud server or a local data center. Each node should have sufficient memory, CPU resources and disk space to support the large-scale concurrent access and data processing requirements. Setting the replication factor to 2 or higher ensures that the reliability of data and high availability of the system can be ensured even when a single point of failure occurs. Next, a APACHE FLINK cluster is deployed, including one master node (JobManager) for coordinating task allocation and state management, and a plurality of worker nodes (TASKMANAGERS) for performing actual data processing tasks. Configuration parameters of TASKMANAGER and JobManager are adjusted according to hardware resource conditions, JVM heap size, parallelism and the like to optimize performance. The Kafka producer API is configured to ensure that it can efficiently receive data from the front-end monitoring tool and publish it to the specified Kafka topic. And setting the Flink application as a Kafka consumer, subscribing related topics, starting to consume a data stream from a front-end monitoring tool, performing preliminary cleaning on the original data, and removing invalid or error records. And carrying out aggregation calculation on data in a period of time through a window function, training a deep learning model based on historical performance data, determining a normal fluctuation range of performance indexes, integrating the trained model into a task of the Flink, and realizing real-time data analysis and prediction.

Preferably, in the step S3, the method further includes:

The intelligent scene adaptation module is used for automatically identifying and analyzing the application scene of the webpage, customizing performance monitoring strategies and indexes for different application scales of the webpage, ensuring the monitoring accuracy and pertinence, and meeting the specific requirements of clients. The deep integration of intelligent scene adaptation and monitoring strategy generation is a static tool or system and is a continuous evolution process. Through continuous monitoring, analysis and optimization, the continuous improvement of the performance of the website can be ensured, and smoother and more satisfied access experience is provided for users.

Preferably, in the step S4, the method further includes:

The dynamic optimization strategy engine is also a core component of the scheme and is provided with real-time analysis monitoring data and automatically adjusts the optimization strategy. By continuously monitoring the performance of the website, the performance bottleneck and potential risk points are rapidly identified, and relevant parameter configuration is adjusted in real time according to the latest monitoring result, so that the website can always maintain the optimal performance under various different network conditions and user behavior modes, and a good complementary relationship is formed between the website and the scene intelligent adaptation module. The dynamic optimization strategy engine can receive and process the data flow from the performance monitoring system in real time, and flexibly adjust the optimization strategy according to the feedback information of the data. For example, it may reprioritize resource loading to speed page loading, optimize image compression algorithms to reduce bandwidth consumption, or more finely control cache resources to improve access efficiency. Regardless of the challenges faced, the dynamic optimization policy engine can quickly respond and implement corresponding corrective measures. The high flexibility and the real-time response capability ensure that the website always maintains the optimal running state even in complex and changeable network environments, and provides high-quality access experience for users.

Compared with the prior art, the invention discloses a webpage performance monitoring and analyzing method, a system, a device and a storage medium. The invention has the beneficial effects that:

1. The current system is not limited to detecting traditional indexes such as page loading time, response time and the like, but also comprises multidimensional performance parameters such as First Contact Time (FCT), time expiration rate (TLP), user interaction time, error rate, jump rate, throughput and the like. By comprehensively considering the indexes, the webpage performance can be more comprehensively evaluated, and the accuracy and the comprehensiveness of the monitoring result are ensured;

2. The intelligent and accurate model integrates an intelligent reference model, an anomaly detection algorithm matrix and a machine learning model, so that the system can automatically analyze the processed data, mine performance trend and potential anomaly, and realize accurate prediction. The intelligent method not only improves the accuracy and efficiency of anomaly detection, but also effectively filters false alarms, reduces missing alarms, and provides clear and reliable anomaly signals for operation and maintenance teams;

3. Real-time performance and high efficiency by deploying APACHE KAFKA clusters and APACHE FLINK, high-efficiency processing of real-time data streams is realized, and low delay of data processing is ensured. The system can respond to the performance change immediately and quickly capture and process the performance data. According to the invention, the average page loading time is reduced by 20%, the response time is accelerated by 30%, the user satisfaction is improved by 25%, the optimized resource management strategy effectively reduces the burden of a server, and the bandwidth cost is reduced by 30%;

4. Dynamic optimization and efficient resource utilization, namely designing a dynamic optimization strategy engine, dynamically adjusting website resources such as resource loading priority, a picture compression algorithm, a cache management strategy and the like according to real-time monitoring data, ensuring that the website can keep optimal performance under different network environments and user behaviors, and improving the resource utilization efficiency.

Drawings

For a clearer description of the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some examples of the present invention and should not be considered as limiting the scope, and that other related drawings can be obtained according to these drawings without inventive effort for a person skilled in the art:

FIG. 1 is a flow chart of system deployment and data acquisition analysis of the present invention;

FIG. 2 is a flow chart of web page performance monitoring and optimization in accordance with the present invention.

Detailed Description

Step one, system deployment and configuration

S11, constructing a high-availability infrastructure in a cloud server or a local data center, wherein the infrastructure comprises a high-performance server, an SSD storage array and high-speed network switching equipment. Ensuring that all hardware resources support the requirements of large-scale concurrent access and data processing;

S12, installing Java running environment, a database management system (MySQL/PostgreSQL) and necessary middleware services (APACHE HIVE for data warehouse and Apache flash for data stream processing);

s13 configuration APACHE KAFKA clusters real-time data streams, deployment APACHE FLINK implements low-latency stream data processing. The configuration of TASKMANAGER and JobManager is optimized according to system resources to achieve real-time data stream processing.

Step two, data acquisition and intelligent analysis

S21, acquiring performance data of front-end pages of the merchant website in real time through a front-end monitoring tool, wherein the performance data comprise page loading time, first Contact Time (FCT), time expiration rate (TLP), user interaction time, error rate, jump rate and throughput key performance indexes. The data acquisition device is ensured to have no interference to user access, the data is accurate and reliable, and the acquired data is sent to the designated Topic of the APACHE KAFKA clusters in a JSON format;

S22, configuring a Kafka cluster, setting a Topic to receive performance data from a front end, sending the acquired performance data to the Kafka Topic by using Kafka Producer API, reading a data stream by using Kafka Consumer API in APACHE FLINK, and carrying out real-time processing;

S23, in the stream processing task of the Flink, data cleaning and preprocessing are carried out by using TensorFlow.js, including invalid data removal, standardized data format and the like, a machine learning model (LSTM or other deep learning model) is constructed and trained to predict webpage performance trend, and the trained model is saved and loaded in the Flink to carry out real-time prediction.

Third, intelligent scene adaptation and monitoring strategy generation

S31, a development scene intelligent recognition module is used for automatically recognizing the webpage application scene by utilizing machine learning or a rule engine. The identification result accurately reflects the actual use scene and performance requirement of the user;

s32, dynamically generating a performance monitoring strategy and an index according to the scene recognition result. Customizing a personalized monitoring scheme for a specific webpage application, and ensuring the pertinence and the effectiveness of monitoring;

s33, continuously collecting user feedback and performance data, and performing iterative optimization on the monitoring strategy. And the data analysis result is utilized to guide performance optimization work, so that the overall performance and user experience of the website are improved.

Step four, dynamic optimization strategy and resource management

S41, intelligently adjusting the resource configuration of a website based on the performance monitoring data collected in real time, and carrying out abnormal aggregation and deduplication by using a Redis or other memory databases to ensure the accuracy and the effectiveness of the monitoring data;

S42, constructing a complete automatic optimization flow, automatically adjusting the website resource configuration according to the decision generated by the dynamic optimization strategy engine, establishing a mechanism to continuously monitor the optimized effect, and carrying out necessary adjustment on the optimization strategy according to the actual effect to form a closed-loop optimization process;

S43, designing and implementing dynamic resource scheduling and load balancing strategies aiming at different network environments and user access modes. Under the condition of high concurrent access, the website can still stably and efficiently run. Through means of intelligently distributing server resources, reasonably planning data transmission paths and the like, the problems of service interruption, performance degradation and the like caused by sudden increase of access quantity are effectively avoided.

Step five, constructing a performance monitoring system

S51, constructing a webpage performance monitoring system comprising a data acquisition module, an analysis module, a report generation module, an alarm module and an optimization suggestion module;

S52, acquiring webpage loading data and user interaction data in real time, performing deep analysis on the data by using an anomaly detection algorithm matrix and a model, supporting setting of a performance threshold, triggering an alarm immediately once anomaly is detected, and notifying corresponding personnel.

Step six, optimizing the storage medium

S61, storing real-time performance data by using SSD, improving the quick response of data processing, and storing historical data in a cloud storage system so as to facilitate long-term storage and data analysis;

S62, periodically backing up key data, performing recovery test, ensuring the validity and the recoverability of the data, monitoring the performance index of the storage medium in real time, and finding and solving the potential problem in time.

Claims

1. A web page performance monitoring and analysis method, system, device and storage medium, characterized in that the method comprises the following steps:

S1: Build a high-availability infrastructure, deploy high-performance servers, SSD storage arrays, and high-speed network switching devices to ensure that hardware resources support large-scale concurrent access and data processing requirements;

S2: Collect performance data of front-end pages in real time, including but not limited to page loading time, first contact time (FCT), time to expiration rate (TLP), user interaction time, error rate, bounce rate, throughput, etc., and use stream processing technology and real-time data analysis algorithms for preliminary cleaning and aggregation operations;

S3: Use machine learning models to automatically analyze processed data, mine performance trends and potential anomalies, build intelligent benchmark models, and integrate anomaly detection algorithm matrices such as statistical methods, cluster analysis, and time series pattern recognition;

S4: Start the scene intelligent adaptation module to accurately identify web application scenarios, dynamically customize performance monitoring strategies and indicators, and provide personalized monitoring solutions for different web applications;

S5: Design a dynamic optimization strategy engine to analyze monitoring data in real time and adjust website resources to ensure that the website maintains optimal performance under different network environments and user behaviors;

S6: Build a web performance monitoring system, covering data collection, analysis, report generation, alarm and optimization suggestion modules, forming a closed-loop continuous performance optimization process;

S7: Optimize storage media, ensure the integrity and accessibility of performance data, improve overall storage efficiency, regularly back up critical data, and perform recovery tests.

2. The method according to claim 1, characterized in that said S1 further comprises:

Deploy an Apache Kafka cluster consisting of multiple nodes on a cloud server or local data center;

Each node should have sufficient memory, CPU resources, and disk space to support large-scale concurrent access and data processing requirements;

Set the replication factor to 2 or higher to ensure data reliability and high system availability even when a single point of failure occurs;

Deploy an Apache Flink cluster, including a master node (JobManager) for coordinating task allocation and state management, and multiple worker nodes (TaskManagers) for performing actual data processing tasks; adjust the configuration parameters of TaskManager and JobManager, JVM heap size, parallelism, etc. according to hardware resources to optimize performance;

Configure the Kafka producer API to ensure that it can efficiently receive data from the front-end monitoring tool and publish it to the specified Kafka topic;

Set the Flink application as a Kafka consumer, subscribe to relevant topics, start consuming data streams from the front-end monitoring tool, perform preliminary cleaning on the raw data, and remove invalid or erroneous records;

Through window functions, data within a period of time is aggregated and calculated, and deep learning models are trained based on historical performance data to determine the normal fluctuation range of performance indicators. The trained models are then integrated into Flink tasks to achieve real-time data analysis and prediction.

3. The method according to claim 1, characterized in that said S2 further comprises:

Develop a scenario intelligent recognition module to automatically identify web application scenarios using machine learning or rule engines;

Dynamically generate performance monitoring strategies and indicators based on the identification results.

4. The method according to claim 1, characterized in that said S3 further comprises:

Intelligently adjust website resource configuration based on performance monitoring data collected in real time;

Establish automated optimization processes and implement dynamic resource scheduling and load balancing strategies.

5. The method according to claim 1, characterized in that said S4 further comprises:

By continuously monitoring website performance, performance bottlenecks and potential risk points can be quickly identified;

Adjust relevant parameter configurations in real time based on the latest monitoring results;

The dynamic optimization strategy engine can receive and process data streams from the performance monitoring system in real time;

Flexibly adjust the optimization strategy based on the feedback information from these data.

6. The method according to any one of claims 1 to 4, further comprising:

Use SSD to store real-time performance data and store historical data in cloud storage;

Regularly perform backup and recovery tests on critical data;

Monitor storage media performance indicators in real time to identify and resolve potential problems promptly.