CN119127975A - An intelligent information management method and system based on cloud computing - Google Patents
An intelligent information management method and system based on cloud computing Download PDFInfo
- Publication number
- CN119127975A CN119127975A CN202411613478.1A CN202411613478A CN119127975A CN 119127975 A CN119127975 A CN 119127975A CN 202411613478 A CN202411613478 A CN 202411613478A CN 119127975 A CN119127975 A CN 119127975A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- read
- write
- cache
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000007726 management method Methods 0.000 title claims abstract description 71
- 238000004458 analytical method Methods 0.000 claims abstract description 138
- 238000000034 method Methods 0.000 claims abstract description 108
- 238000012545 processing Methods 0.000 claims abstract description 76
- 238000005516 engineering process Methods 0.000 claims abstract description 37
- 238000000547 structure data Methods 0.000 claims description 70
- 238000012937 correction Methods 0.000 claims description 60
- 238000004088 simulation Methods 0.000 claims description 57
- 238000004364 calculation method Methods 0.000 claims description 55
- 238000005457 optimization Methods 0.000 claims description 49
- 230000004044 response Effects 0.000 claims description 45
- 230000001960 triggered effect Effects 0.000 claims description 38
- 238000013461 design Methods 0.000 claims description 34
- 238000000926 separation method Methods 0.000 claims description 27
- 230000006978 adaptation Effects 0.000 claims description 19
- 238000013507 mapping Methods 0.000 claims description 10
- 238000003032 molecular docking Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 description 68
- 238000004422 calculation algorithm Methods 0.000 description 38
- 230000007246 mechanism Effects 0.000 description 19
- 238000011156 evaluation Methods 0.000 description 16
- 230000006399 behavior Effects 0.000 description 11
- 238000009826 distribution Methods 0.000 description 11
- 230000008859 change Effects 0.000 description 10
- 238000009472 formulation Methods 0.000 description 7
- 239000000203 mixture Substances 0.000 description 7
- 238000003860 storage Methods 0.000 description 7
- 230000002159 abnormal effect Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 239000007787 solid Substances 0.000 description 5
- 230000003044 adaptive effect Effects 0.000 description 4
- 238000000540 analysis of variance Methods 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 239000012634 fragment Substances 0.000 description 4
- 238000012163 sequencing technique Methods 0.000 description 4
- 230000015556 catabolic process Effects 0.000 description 3
- 238000011217 control strategy Methods 0.000 description 3
- 238000007405 data analysis Methods 0.000 description 3
- 238000013523 data management Methods 0.000 description 3
- 238000006731 degradation reaction Methods 0.000 description 3
- 230000003111 delayed effect Effects 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 238000005192 partition Methods 0.000 description 3
- 238000013439 planning Methods 0.000 description 3
- 238000013468 resource allocation Methods 0.000 description 3
- 238000007619 statistical method Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 238000012300 Sequence Analysis Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 2
- 238000004140 cleaning Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000010921 in-depth analysis Methods 0.000 description 2
- 238000003780 insertion Methods 0.000 description 2
- 230000037431 insertion Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000013178 mathematical model Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 238000010845 search algorithm Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000012098 association analyses Methods 0.000 description 1
- 238000012550 audit Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000010485 coping Effects 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000012854 evaluation process Methods 0.000 description 1
- 238000005429 filling process Methods 0.000 description 1
- 239000003102 growth factor Substances 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000012858 packaging process Methods 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 238000000611 regression analysis Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/604—Tools and structures for managing or administering access control systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5072—Grid computing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/21—Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/2141—Access rights, e.g. capability lists, access control lists, access tables, access matrices
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Quality & Reliability (AREA)
- Automation & Control Theory (AREA)
- Fuzzy Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of intelligent information management, in particular to an intelligent information management method and system based on cloud computing. The method comprises the steps of carrying out load time sequence bias analysis on log data of an enterprise database to obtain load fluctuation time sequence bias data, carrying out table structure analysis on the enterprise database based on enterprise database management authority and carrying out table structure performance bottleneck analysis to obtain table structure performance bottleneck data, carrying out asynchronous read-write cache processing on a database table structure according to the table structure performance bottleneck data and the load fluctuation time sequence bias data to obtain an asynchronous read-write cache database, carrying out compatibility matching on the asynchronous read-write cache database to obtain an asynchronous read-write compatibility cache database, and butting the asynchronous read-write compatibility cache database into the enterprise database to execute intelligent information management. The invention improves the intelligent information management technology through optimizing the intelligent information management technology.
Description
Technical Field
The invention relates to the technical field of intelligent information management, in particular to an intelligent information management method and system based on cloud computing.
Background
Cloud computing, which is an emerging computing model, has become one of the core technologies of modern information management by virtue of its powerful computing power, mass data storage and elastic expansibility. In an enterprise data management system, a database often carries a large number of business operations, such as order processing, customer management and financial settlement, and with the increase of data volume and concurrent requests, load fluctuation and performance bottleneck of the database become key problems affecting the operation efficiency of an enterprise. The intelligent information management method relies on the cloud computing infrastructure, can provide a high-efficiency, reliable and flexible solution, and is particularly suitable for processing complex data streams, mass data analysis and diversified business requirements. The conventional information management method has the problems of low processing efficiency, limited storage capacity, poor expansibility and the like in the aspects of data diversity, real-time performance and dynamic change. The intelligent information management method based on cloud computing can dynamically allocate and adjust computing resources by fully utilizing the resource scheduling capability and the virtualization technology of the cloud platform, so that the on-demand allocation of the resources is realized, the management cost is reduced, and the response speed and the flexibility of the system are improved. However, the traditional intelligent information management method based on cloud computing has the problems that the analysis of the structural performance bottleneck of the database table is inaccurate, and high concurrent mass data cannot be handled.
Disclosure of Invention
Based on this, it is necessary to provide an intelligent information management method and system based on cloud computing, so as to solve at least one of the above technical problems.
In order to achieve the above purpose, an intelligent information management method based on cloud computing comprises the following steps:
Step S1, acquiring enterprise database log data and enterprise database management authority, carrying out load time sequence bias analysis on the enterprise database log data to obtain load fluctuation time sequence bias data;
step S2, carrying out table structure analysis on an enterprise database based on enterprise database management authority to obtain a database table structure, carrying out table structure performance bottleneck analysis on the database table structure to obtain table structure performance bottleneck data, carrying out asynchronous read-write cache processing on the database table structure according to the table structure performance bottleneck data and load fluctuation time sequence bias data to obtain an asynchronous read-write cache database;
step S3, carrying out compatibility matching on the asynchronous read-write cache database based on a containerization technology in cloud computing to obtain an asynchronous read-write compatible cache database;
and S4, interfacing the asynchronous read-write compatible cache database into an enterprise database to execute intelligent information management.
The method is important to obtain the log data of the enterprise database and the management authority of the enterprise database. The method not only ensures the comprehensiveness of the analysis process, but also lays a foundation for the subsequent performance optimization. By carrying out load time sequence bias analysis on the log data, the load change condition of the database in different time periods can be identified, including peak load and valley load. Such analysis can help enterprises understand the usage patterns of the system over different time periods and provide important basis for capacity planning, resource allocation, and fault prevention. In addition, the load fluctuation time sequence bias data provides real-time monitoring capability for database management staff of enterprises, and potential performance bottlenecks or abnormal conditions can be timely identified, so that corresponding measures are adopted for optimization. Finally, the stability and response speed of the database are effectively improved, and the good service quality of enterprises under the high-load condition is ensured. Based on enterprise database management rights, it is necessary to parse the database table structure, which enables a manager to fully understand the design and architecture of the database. Through the table structure performance bottleneck analysis, it can be determined which tables or fields perform poorly under high load conditions, resulting in delays in query and write operations. Such in-depth analysis provides a means to optimize the specific direction of the database, such as by redesigning the table structure, adding an index, or archiving data. Next, it is very effective to implement asynchronous read-write caching processing based on the analysis results of the table structure performance bottleneck data and the load fluctuation time sequence bias data. The asynchronous cache can remarkably improve the read-write performance of the database, reduce response time and improve the overall throughput of the system. The process not only improves the access speed of the database, but also reduces the pressure on the back-end database, can effectively improve the operation experience of the user, and provides powerful support for the business sustainable development of enterprises. This step ultimately helps the enterprise achieve more efficient data management and processing capabilities under high load conditions. Compatibility matching is a key link for an asynchronous read-write cache database by utilizing a containerization technology in cloud computing. The containerization technique allows applications and all of their dependencies to be packaged in a lightweight, portable environment, thereby ensuring consistent running results in different computing environments. In the process, by carrying out compatibility matching on the asynchronous read-write cache database, potential compatibility problems among different system components can be identified and solved, and seamless data flow among different containers can be ensured. The matching process can improve the flexibility of the system, so that enterprises can rapidly deploy and expand database services according to requirements, and meanwhile, the maintenance complexity is reduced. Finally, the obtained asynchronous read-write compatible cache database can be operated in various cloud environments with high efficiency, and the data access speed and the reliability of the system are remarkably improved. This not only optimizes the enterprise's IT architecture, but also lays a solid foundation for subsequent intelligent information management. The interfacing of the asynchronous read-write compatible cache database with the enterprise database is an important step for achieving intelligent information management. The docking process can realize real-time updating and accessing of data, so that enterprises can keep high efficiency when processing massive data. By integrating the compatible cache database into the enterprise database, the enterprise can perform data analysis and decision more quickly, and the intelligent level of business operation is improved. The realization of intelligent information management can not only improve the efficiency of data processing, but also utilize machine learning and data analysis technology to provide deep insight for enterprises, help decision makers to identify trends, optimize business processes and reduce cost. In addition, the real-time data flow is also beneficial to enhancing the agility of enterprises, so that the enterprises can quickly respond to market changes and customer demands, and the competitiveness of the enterprises is improved. The series of optimization finally enables the power-assisted enterprises to better utilize data resources in digital transformation, and promotes sustainable development of the business.
Preferably, step S1 comprises the steps of:
S11, acquiring enterprise database log data and enterprise database management authorities;
Step S12, filling missing values of the enterprise database log data to obtain database log filling data;
S13, carrying out load fluctuation analysis on the database log filling data to obtain database load fluctuation data;
And S14, carrying out load time sequence deviation analysis on the load fluctuation data of the database to obtain the load fluctuation time sequence deviation data.
The invention firstly obtains the log data and the management authority of the enterprise database, which is the basis for analyzing and optimizing the performance of the database. This process ensures that the analyst has full access to the database's operational records, thereby accurately understanding the usage and performance bottlenecks of the system. Meanwhile, the acquisition of the management authority provides necessary authority guarantee for subsequent data processing and optimization. The log data of the enterprise database contains rich information such as query frequency, execution time, error information and the like, which provide precious basis for analyzing and diagnosing problems. By comprehensively acquiring the data, an enterprise can better monitor the running state of the database and identify potential risk points, so that a solid foundation is laid for optimizing the database architecture and improving the performance. The completion of this step creates a good data base for subsequent analysis work. Filling missing values into database log data is a key link for improving data quality and analysis accuracy. The database log data have missing values, and the missing values can cause deviation to subsequent analysis, so that the scientificity of decision making is affected. By adopting a proper filling strategy (such as mean filling, median filling or predictive filling based on a machine learning model), missing parts in the data set can be effectively reduced, so that complete database log filling data can be generated. This step not only improves the integrity of the data, but also more fully reflects the actual operating conditions of the database. In addition, the filled data can play a larger role in subsequent load fluctuation analysis and performance evaluation, and the reliability and effectiveness of analysis results are ensured, so that a more accurate basis is provided for optimization of a database. Load fluctuation analysis is carried out on the database log filling data, so that the use condition and the load characteristics of the database in different time periods can be effectively disclosed. Such analysis typically involves statistical calculations, trend analysis, and periodic checks of the data to identify peak loads, valley loads, and their law of variation. By acquiring the load fluctuation data of the database, enterprises can deeply understand the performance of the system in different time periods, including response time, concurrent connection number, query efficiency and the like. The information can help IT team of enterprises identify potential bottlenecks of the system, optimize resource allocation, and further improve stability and usability of the system. In addition, knowing the load fluctuation characteristics is also beneficial to enterprises to conduct more reasonable capacity planning, so that the system can still keep good service quality in peak hours, and user experience is improved. The load time sequence bias analysis is carried out on the load fluctuation data of the database so as to deeply mine the load characteristics of the database and the change rule of the load characteristics with time. Load time sequence bias analysis can help identify the distribution characteristics of data and detect potential abnormal fluctuation and extreme values. The process not only can reveal the bias distribution condition of the load data, but also can evaluate the performance of the database under different load conditions. by obtaining the load fluctuation time sequence deviation data, enterprises can clearly know key indexes such as response time, processing capacity, resource utilization rate and the like of the system under the condition of high load. Such information is critical to the formulation of optimization strategies, helping enterprises to consider load characteristics in designing and implementing database architectures, and thus more efficiently performing performance tuning and risk management. Finally, the analysis provides important decision support for subsequent database optimization, and ensures that enterprises can still keep high-efficiency operation in continuously-changing business environments.
Preferably, step S2 comprises the steps of:
S21, carrying out table structure analysis on an enterprise database based on enterprise database management authority to obtain a database table structure;
s22, performing table structure performance bottleneck analysis on the database table structure according to the load fluctuation time sequence bias data to obtain table structure performance bottleneck data;
step S23, performing high-load read-write delay response simulation according to the table structure performance bottleneck data and the load fluctuation time sequence bias data to obtain high-load read-write delay response data;
step S24, performing read-write separation adaptation on the database table structure based on the high-load read-write delay response data to obtain database read-write separation adaptation data;
And S25, carrying out asynchronous read-write cache processing on the database table structure according to the database read-write separation adaptive data to obtain an asynchronous read-write cache database.
The invention analyzes the table structure of the database through the management authority of the enterprise database, and can comprehensively understand the design and the architecture of the database. This process involves analyzing the various tables, fields and their data types, constraints, and relationship structures in the database to form a clear understanding of the database table structure. Through table structure analysis, enterprises can identify the logic layout of data storage, and basic data support is provided for subsequent performance optimization and problem investigation. Furthermore, deep knowledge of the table structure helps to find potential design flaws, such as redundant data or unreasonable relationships, that can affect the performance and scalability of the database. Finally, this step not only improves the maintainability and flexibility of the database, but also provides advantages for future expansion and adjustment of the enterprise, thereby supporting the business development strategy thereof. And carrying out performance bottleneck analysis on the database table structure according to the load fluctuation time sequence bias data, wherein the performance bottleneck analysis is used for identifying key factors affecting the performance of the database. By comparing load fluctuations with the table structure, it can be found which tables or fields perform poorly under high load conditions, such as long response times, low query efficiency, or poor concurrent processing capability. The analysis not only can help enterprises identify performance bottlenecks, but also can deeply understand the performances of the database under different loads, thereby providing data support for the formulation of an optimization scheme. In addition, the results of the performance bottleneck analysis are beneficial to the enterprises to make targeted decisions in the aspects of database design adjustment, query optimization, index addition and the like, and finally, the overall performance and the response capacity of the database are obviously improved, so that the enterprises can still operate stably under the high-load condition. And performing high-load read-write delay response simulation based on the table structure performance bottleneck data and the load fluctuation time sequence bias data, so as to evaluate the performance of the database under extreme load. By simulating a high-load scene, enterprises can obtain high-load read-write delay response data, so that the behavior of the database under the pressure condition is deeply understood. Such simulations not only reveal the response time and processing power of the database in high concurrency situations, but also identify specific performance bottlenecks and potential failure points. Through the obtained delayed response data, enterprises can take corresponding optimization measures, such as adjusting inquiry strategies, improving index design or upgrading hardware facilities, so as to ensure that challenges caused by high loads can be better dealt with in actual operation. The process helps enterprises to foresee problems before actual business operation, so that the stability and reliability of the system are improved, and the high efficiency of data processing is ensured. The read-write separation adaptation of the database table structure based on the high-load read-write delay response data is an important means for improving the performance of the database. The read-write separation technology can effectively lighten the pressure of the main database and improve the data processing efficiency by distributing the read request and the write request to different database examples. By analyzing the read-write latency response data under high load conditions, the enterprise can identify which queries can be optimized for read operations, thereby implementing read-write separation. The process not only improves the concurrent processing capacity of the system, but also reduces the influence on the reading operation during data updating and improves the response speed of user access. In addition, the read-write separation can help enterprises to realize better load balancing, enhance the expandability of the system and ensure that good user experience can be maintained under a high-load scene. This adaptation process will lay a solid foundation for the subsequent asynchronous read-write caching process. Asynchronous read-write caching processing is carried out on the database table structure according to the database read-write separation adaptation data, so that the performance and the user experience of the database are further improved. The asynchronous read-write cache can enable read operation and write operation to be processed in parallel, so that response time is greatly reduced, and overall throughput of the system is improved. The processing mode not only reduces the pressure of the database under the condition of high load, but also improves the reading efficiency of the data by reducing the frequency of directly writing the data into the database. Asynchronous caching allows data to be processed in the background without affecting the operation of the front-end user, ensuring that the user can get faster feedback when accessing. In addition, through asynchronous read-write caching, enterprises can more flexibly cope with high concurrency requests, and the availability and stability of the system are enhanced. This optimization will provide powerful support for enterprise data management, ensuring efficient and stable database operation in a rapidly evolving business environment.
Preferably, step S22 comprises the steps of:
s221, performing field constraint structure analysis on a database table structure to obtain field constraint structure data;
Step S222, performing constraint nesting level calculation on the database table structure according to the field constraint structure data to obtain field constraint nesting level data;
Step S223, performing trigger association logic retrieval on the database table structure to obtain trigger association logic data;
step S224, carrying out event triggering logic complexity evaluation on the database table structure according to the trigger association logic data to obtain event triggering logic complexity data;
Step S225, performing nonlinear time complexity calculation based on event triggering logic complexity data and field constraint nested level data to obtain nonlinear time complexity data;
Step S226, performing execution complexity load fluctuation mapping on the execution nonlinear time complexity data according to the load fluctuation time sequence bias data to obtain execution complexity load fluctuation data;
And step S227, performing table structure performance bottleneck analysis on the database table structure according to the execution complexity load fluctuation data and the execution nonlinear time complexity data to obtain table structure performance bottleneck data.
The invention analyzes the field constraint structure of the database table structure by identifying and extracting the constraint conditions (such as a main key, an external key, a unique constraint, a non-null constraint and the like) of each field. The purpose of this analysis is to fully understand the relationships and constraints between fields to facilitate subsequent nested hierarchy calculations. By obtaining field constraint structure data, a database designer can effectively identify data integrity problems and ensure consistency and accuracy of data in a storage process. The effective constraint structure can not only prevent the insertion of invalid data, but also improve the query efficiency, because the existence of the constraint can enable the database optimizer to better select an execution plan. In addition, the clear field constraint structure lays a solid foundation for subsequent complexity evaluation and performance analysis, so that the database is more stable and reliable in performance in complex transaction processing. And performing constraint nesting level calculation on the field constraint structure data. The process involves deep analysis of hierarchical relationships between the field constraints to form a hierarchical constraint structure map. By calculating the nesting level of constraints, it is possible to identify which field constraints are interdependent, and thus analyze the most complex combination of constraints. This is critical to the design and maintenance of databases because excessive or complex nesting constraints can lead to performance problems, particularly when processing large amounts of data. Meanwhile, the result of the step can also be used for optimizing the design of the database, simplifying the constraint structure, thereby reducing redundancy and improving the efficiency of data operation. Through a clear hierarchy, database administrators can better understand and manage the constraint characteristics of data, thereby improving maintainability of the system. The trigger association logic retrieval of the database table structure is performed to obtain the relationship data between the trigger and the table structure. Triggers are a mechanism automatically implemented in databases that can execute specific logic upon data modification, such as verifying data integrity, maintaining audit logs, etc. Thus, accurately identifying the associated logic of a trigger is critical to understanding the behavior of the database and its performance. This step may help developers and database administrators understand the execution conditions of triggers and their interactions with fields in the table structure. This analysis not only helps to find potential performance bottlenecks, but also evaluates the impact on trigger execution when modifying the table structure, ensuring the integrity and consistency of data operations. By the manipulation of the trigger logic, the predictability and reliability of the database is improved, thereby reducing data problems due to logic errors. And carrying out event triggering logic complexity evaluation on the trigger association logic data. The purpose of this step is to systematically analyze the logic performed by the trigger, evaluating its complexity in the event of various events. complex trigger logic can lead to performance degradation at high loads and thus is critical for its evaluation of complexity. Through this process, the execution efficiency of the trigger under different conditions can be identified, its performance at high concurrent accesses is analyzed, and the direction of optimization is found. The evaluation result of the event-triggered logic complexity provides data support for subsequent performance optimization and database adjustment, and ensures that the system can cope with challenges of different loads in actual operation. By comprehensively considering the logic complexity, performance bottlenecks caused by excessively complex logic can be effectively avoided, and therefore the response speed and processing capacity of the whole database system are improved. And carrying out combined analysis on the event triggering logic complexity data and the field constraint nesting level data, and calculating the nonlinear time complexity. This calculation can reveal how the temporal complexity of database operations varies with the amount of input data under certain event-triggered conditions. From this analysis, it can be identified which operations will cause an exponential increase in execution time under certain conditions, thereby affecting the overall performance of the database. Particularly in big data environment, the nonlinear time complexity identification is helpful for the database manager to make more intelligent design decisions, and optimize the database structure and the query logic so as to improve the processing efficiency and the response speed of the system. The result of this step provides an important mathematical basis for database optimization, enabling developers to take into account performance factors during the design phase. the execution complexity load fluctuation mapping is performed on load fluctuation time sequence bias data, and aims to disclose the change rule of the execution complexity under different load conditions. This process helps database administrators understand how the behavior of the system changes under high and low load conditions by modeling the relationship between load and execution complexity. The mapping analysis can not only provide the performance data of the database under different workloads, but also provide reference basis for capacity planning and performance tuning of the system. By analyzing the influence of load fluctuation on the execution complexity, a database manager can effectively predict the performance bottleneck appearing in the peak period, so that optimization measures are adopted in advance, and the system can be ensured to stably operate under various load conditions. The result of this analysis helps to achieve a more efficient allocation of resources, optimizing the overall performance of the database system. And carrying out table structure performance bottleneck analysis according to the execution complexity load fluctuation data and the execution nonlinear time complexity data. By integrating the data obtained in the previous steps, the performance bottleneck of the database table structure in processing high load and complex events is identified. This analysis can reveal which specific field constraints, trigger logic, or execution paths lead to performance degradation. By identifying these bottlenecks, database administrators may more specifically optimize, such as redesigning the table structure, optimizing the index, or adjusting the trigger logic to improve overall performance. the final objective of this process is to ensure that the database system achieves optimal performance under a variety of operating conditions, avoiding performance problems due to design defects. The deep bottleneck analysis provides an important basis for continuous optimization of the database, so that the system can adapt to the continuously changing business requirements.
Preferably, performing the nonlinear time complexity calculation based on the event-triggered logic complexity data and the field constraint nested hierarchy data comprises the steps of:
respectively carrying out nonlinear execution efficiency simulation on the event triggering logic complexity data and the field constraint nesting level data to respectively obtain event triggering nonlinear execution efficiency data and nonlinear field constraint execution efficiency data;
performing hierarchical execution complexity error correction calculation on the event trigger nonlinear execution efficiency data to obtain trigger execution complexity error correction data;
performing efficiency convergence average difference calculation on the nonlinear field constraint execution efficiency data to obtain constraint efficiency convergence average difference data;
And performing nonlinear time complexity calculation based on the trigger execution complexity error correction data and the constraint efficiency convergence average difference data to obtain the execution nonlinear time complexity data.
The invention firstly needs to simulate the nonlinear execution efficiency of event triggering logic complexity and field constraint nested level data. This process involves modeling the trigger logic of different types of events and analyzing their complexity under multi-level, multi-dimensional conditions. For example, mathematical models or simulation tools may be used to simulate response time and resource consumption under different event trigger conditions. Meanwhile, similar simulation is needed to be carried out on nested level data of field constraint, and execution efficiency under different constraint levels is studied. The key of this step is that the execution efficiency of the two types of data is obtained through detailed analysis of various variables and conditions, which provides basic data support for subsequent complexity analysis and optimization. After the event-triggered nonlinear execution efficiency data is acquired, an error correction calculation of the hierarchical execution complexity is required. Since there are certain errors in the simulation process, these errors can affect the final execution complexity analysis. To solve this problem, an error evaluation can be performed by comparing the event trigger efficiency at the time of actual operation with the simulation result using a statistical method. Common correction methods include regression analysis, least square method, and the like, by which analog data can be effectively adjusted so as to be closer to the actual situation. The process ensures that the follow-up execution complexity calculation is based on corrected and more accurate data, thereby improving the reliability and the effectiveness of the analysis result. In the processing of non-linear field constraint execution efficiency data, convergence mean difference calculation is a key step. The goal of this step is to analyze the stability and consistency of execution efficiency under different constraints. By collecting execution efficiency data under different levels of constraints and calculating their mean and variance, the extent to which each constraint affects overall execution efficiency can be evaluated. Specifically, the difference between the sample mean and each sample value in the dataset may be utilized for analysis, resulting in a convergence mean difference. The data not only helps identify which field constraints lead to performance bottlenecks, but also provides basis for system optimization, ensuring that the system can still maintain efficient execution performance under complex constraint conditions. After acquiring the trigger execution complexity error correction data and the constraint efficiency convergence average difference data, the calculation of the nonlinear time complexity needs to be performed based on the data. This calculation typically involves the integration and analysis of the multidimensional data, with a corresponding algorithm model (e.g., nonlinear regression or other optimization algorithm) being employed to determine the final execution time complexity. At this stage, the sufficiency and accuracy of the data is ensured, and appropriate mathematical tools are employed for complexity analysis and modeling. The final result provides powerful support for performance evaluation, resource allocation and optimization of the system, and lays a foundation for future improvement and development. The smooth completion of the process is not only helpful for accurately grasping the performance condition of the current system, but also provides guidance for the subsequent optimization scheme, and ensures the efficient operation of the system in a complex application environment.
Preferably, performing the hierarchical execution complexity error correction calculation on the event-triggered nonlinear execution efficiency data comprises the steps of:
Performing event trigger execution hierarchy retrieval on the event trigger nonlinear execution efficiency data to obtain event trigger execution hierarchy data;
Performing nested level-to-level execution efficiency variance calculation on the event trigger nonlinear execution efficiency data according to the event trigger execution level data to obtain nested level execution efficiency variance data;
performing nested overlapping redundant execution efficiency identification on the event-triggered nonlinear execution efficiency data based on the nested hierarchy execution efficiency variance data to obtain nested overlapping redundant execution efficiency data;
Overlapping error correction processing is carried out on the event trigger nonlinear execution efficiency data according to the nested overlapping redundant execution efficiency data, so that trigger nonlinear execution correction efficiency data is obtained;
and performing hierarchical execution complexity error correction calculation on the trigger nonlinear execution correction efficiency data based on the divide-and-conquer method to obtain trigger execution complexity error correction data.
The invention performs event-triggered execution hierarchy retrieval on event-triggered nonlinear execution efficiency data, and aims to define the execution hierarchy relation among different events. Through this process, complex nonlinear execution efficiency data can be organized and categorized in a hierarchical structure for facilitating subsequent analysis. In particular, event-triggered execution hierarchy retrieval involves identifying and marking each event in a dataset, forming a hierarchy tree from the dependency relationships between the events and the triggering order. The structure can help identify the level position of each event, thereby laying a foundation for calculation and analysis of execution efficiency. The result of this step will provide the necessary framework for subsequent variance calculation and error correction, ensuring the accuracy and reliability of the data. And performing nested inter-level execution efficiency variance calculation on the event-triggered nonlinear execution efficiency data based on the event-triggered execution level data. The purpose of this calculation is to identify execution efficiency fluctuations between different tiers and reveal potential performance bottlenecks. Specifically, the variance between each level is calculated by using a statistical method, and the variance is found by comparing the execution efficiency of the same level with that of different levels. Such analysis of variance may help to understand at what levels the performance is inefficient, as well as the impact of these inefficient levels. Finally, the step is to generate nested level execution efficiency variance data which is used as a basis for subsequent redundancy identification and error correction, and provide guidance for optimizing the execution efficiency. After the nested-hierarchy execution efficiency variances are identified, a nested overlapping redundant execution efficiency identification phase is entered. The core of this process is to analyze whether there is a repetition or overlap of execution efficiency in different levels, thereby identifying cases of redundant execution efficiency. Redundancy not only results in resource waste, but also reduces the execution efficiency of the overall system to some extent. By performing an in-depth analysis of the efficiency variance data on the nested hierarchy, repeated work and unnecessary steps in the execution can be identified. After redundancy is identified, it is clear which links are optimizable and which resources are reasonably configurable in the event execution chain. the result of this step provides a solid foundation for the subsequent error correction and the formulation of an optimization scheme. After identifying nested overlapping redundant execution efficiencies, the next step is to perform overlapping error correction processing on the event-triggered nonlinear execution efficiency data. The goal of this stage is to improve the accuracy and reliability of the overall execution efficiency by eliminating redundant parts. Specifically, the redundant execution efficiency identified in the previous step is quantitatively analyzed, the influence degree of the redundant execution efficiency on the whole execution efficiency data is evaluated, and targeted correction is performed accordingly. This may involve optimizing certain steps, adjusting the order of execution, or allocating resources appropriately to reduce duplication. By the correction, the event execution efficiency can be improved, and a good foundation can be laid for subsequent complexity error correction. The end result of this step is triggering nonlinear execution of correction efficiency data that will be more accurate to facilitate subsequent analysis and decision making. And performing hierarchical execution complexity error correction calculation on the trigger nonlinear execution correction efficiency data based on a divide-and-conquer method. The divide-and-conquer method is an efficient computational strategy that solves one by decomposing a complex problem into a plurality of smaller problems, and finally summarizing the results to obtain a global solution. (in the conventional divide-and-conquer method, if there is an overlapping or dependency relationship between sub-problems, the divide-and-conquer method can cause repeated computation, and the repeated parts are removed, so that the accuracy of the data computation result can be greatly improved by using the divide-and-conquer method) in this step, firstly, event-triggered nonlinear execution correction efficiency data is divided according to the hierarchy, and complexity analysis and error correction are independently carried out for each hierarchy. By the method, the complexity source in the execution of each level can be efficiently identified, and corresponding error correction can be implemented. The hierarchical processing mode can not only improve the calculation efficiency, but also ensure that the interaction between different layers is considered in the correction process. The final result is triggering execution complexity error correction data, which greatly enhances the execution efficiency and reliability of the system and provides important basis for future optimization.
Preferably, step S25 comprises the steps of:
step S251, carrying out concurrent read-write frequency simulation on the database table structure according to the database read-write separation adaptation data to obtain concurrent read-write frequency simulation data;
Step S252, performing read-write cache structure design on a database table structure based on concurrent read-write frequency simulation data to respectively obtain read cache structure data and write cache structure data;
Step 253, performing asynchronous thread matching on the read cache structure data and the write cache structure data to respectively obtain read cache asynchronous thread structure data and write cache asynchronous thread structure data;
Step S254, conflict concurrency control logic design is carried out on the read cache asynchronous thread structure data and the write cache asynchronous thread structure data, and read-write conflict logic control data is obtained;
And S255, carrying out asynchronous read-write cache processing on the database table structure according to the read cache asynchronous thread structure data, the write cache asynchronous thread structure data and the read-write conflict logic control data to obtain an asynchronous read-write cache database.
According to the principle of database read-write separation, the invention simulates the concurrent read-write frequency of the database table structure. The simulation process can provide actual data support for subsequent database design, and ensure that the system can adapt to high concurrent access requirements. By analyzing the read-write behavior of the database table in a high concurrency environment, potential bottlenecks and optimization points can be identified. In addition, the step establishes a foundation for a cache mechanism with reasonable design by collecting and analyzing the simulation data, so that resources can be more efficiently allocated when a database processes a large number of concurrent requests, response time is reduced, and user experience is improved. And designing a proper read-write buffer structure according to the concurrent read-write frequency analog data obtained in the previous step. By deeply analyzing concurrent data, the system can identify the characteristics and requirements of read operations and write operations, thereby respectively constructing an optimized read cache structure and a write cache structure. The structural design can effectively reduce the frequency of direct access of the database, relieve the pressure of the database and improve the overall data processing efficiency. The reasonable configuration of the read-write cache not only can improve the reading speed of the data, but also can ensure the consistency and the integrity of the data, and provides high-efficiency support for subsequent read-write operation. Asynchronous thread matching is performed on the read cache structure data and the write cache structure data. The process is used for optimizing concurrent operation by analyzing the characteristics of read-write operation and distributing the characteristics to corresponding asynchronous threads. The asynchronous processing mechanism enables a plurality of read-write requests to be carried out simultaneously, fully utilizes system resources and reduces waiting time. Through effective thread matching, the system not only can improve the throughput of data processing, but also can enhance the expandability of the system, adapt to larger-scale concurrent requests and improve the overall performance of the system. By analyzing the read cache asynchronous thread structure data and the write cache asynchronous thread structure data, the system can identify potential read-write conflicts and establish corresponding control logic. This logic ensures that read and write operations can be performed safely and orderly, avoiding data inconsistencies and errors, in the event of high concurrency. The conflict control mechanism is crucial to maintaining the data integrity of the system, can effectively coordinate operation in a multi-user environment, and improves the reliability and stability of the system. And combining the read cache asynchronous thread structure data, the write cache asynchronous thread structure data and the read-write conflict logic control data obtained in the previous step, and performing asynchronous read-write cache processing on the database table structure. Through the processing, the system can realize a more flexible and efficient database access mode, and the overall response speed and processing capacity are improved. The introduction of asynchronous read-write cache means that the system can process more requests at the same time, thereby reducing delay and improving user experience. Meanwhile, the optimized caching strategy also enables the database to continuously run under the high-load condition, and stability and usability of the system are ensured.
Preferably, the present invention also provides a cloud computing-based intelligent information management system for executing the cloud computing-based intelligent information management method as described above, the cloud computing-based intelligent information management system comprising:
The load time sequence deviation analysis module is used for obtaining the log data of the enterprise database and the management authority of the enterprise database, carrying out load time sequence deviation analysis on the log data of the enterprise database and obtaining load fluctuation time sequence deviation data;
The asynchronous read-write cache processing module is used for carrying out table structure analysis on the enterprise database based on the enterprise database management authority to obtain a database table structure, carrying out table structure performance bottleneck analysis on the database table structure to obtain table structure performance bottleneck data, carrying out asynchronous read-write cache processing on the database table structure according to the table structure performance bottleneck data and the load fluctuation time sequence bias data to obtain an asynchronous read-write cache database;
The compatibility matching module is used for carrying out compatibility matching on the asynchronous read-write cache database based on a containerization technology in cloud computing to obtain an asynchronous read-write compatibility cache database;
And the execution docking module is used for docking the asynchronous read-write compatibility cache database into the enterprise database so as to execute intelligent information management.
The cloud computing-based intelligent information management method has the beneficial effects that the cloud computing-based intelligent information management method is optimized, the problems that the traditional cloud computing-based intelligent information management method is inaccurate in analysis of the structural performance bottleneck of the database table and cannot cope with high concurrent mass data are solved, the accuracy of the structural performance bottleneck analysis of the database table is improved, and the capability of coping with high concurrent mass data is improved.
Drawings
FIG. 1 is a schematic flow chart of steps of an intelligent information management method based on cloud computing;
fig. 2 is a flowchart illustrating a detailed implementation step of step S2 in fig. 1.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
The following is a clear and complete description of the technical method of the present patent in conjunction with the accompanying drawings, and it is evident that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, are intended to fall within the scope of the present invention.
Furthermore, the drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. The functional entities may be implemented in software or in one or more hardware modules or integrated circuits or in different networks and/or processor methods and/or microcontroller methods.
It will be understood that, although the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of example embodiments. The term "and/or" as used herein includes any and all combinations of one or more of the associated listed items.
To achieve the above objective, please refer to fig. 1 to 2, a cloud computing-based intelligent information management method, the method includes the following steps:
Step S1, acquiring enterprise database log data and enterprise database management authority, carrying out load time sequence bias analysis on the enterprise database log data to obtain load fluctuation time sequence bias data;
step S2, carrying out table structure analysis on an enterprise database based on enterprise database management authority to obtain a database table structure, carrying out table structure performance bottleneck analysis on the database table structure to obtain table structure performance bottleneck data, carrying out asynchronous read-write cache processing on the database table structure according to the table structure performance bottleneck data and load fluctuation time sequence bias data to obtain an asynchronous read-write cache database;
step S3, carrying out compatibility matching on the asynchronous read-write cache database based on a containerization technology in cloud computing to obtain an asynchronous read-write compatible cache database;
and S4, interfacing the asynchronous read-write compatible cache database into an enterprise database to execute intelligent information management.
In the embodiment of the present invention, as described with reference to fig. 1, a step flow diagram of an intelligent information management method based on cloud computing according to the present invention is provided, and in this example, the intelligent information management method based on cloud computing includes the following steps:
Step S1, acquiring enterprise database log data and enterprise database management authority, carrying out load time sequence bias analysis on the enterprise database log data to obtain load fluctuation time sequence bias data;
In the embodiment of the invention, in the process of acquiring the log data of the enterprise database and the management authority of the enterprise database, the data connection is established with the database system of the enterprise, and the database log is read by means of a database access tool (such as a database connection driver or a cloud platform database interface). Database log data typically includes read and write operations of the database, transaction execution conditions, error information, index usage conditions, and the like. The log data are subjected to time sequence load analysis through a special time sequence data processing algorithm, wherein the used load time sequence bias analysis technology can adopt a frequency domain and time domain mixed analysis method such as Fourier transformation, hilbert transformation and the like, and the load fluctuation characteristics of the database in different time periods are identified. The specific analysis flow comprises preprocessing log data, processing time sequence data in a segmented mode through a sliding window technology after irrelevant data points are removed, calculating load peak values, average values and deviation coefficients of each segment, and outputting load fluctuation time sequence deviation data. The data provides input basis for the subsequent optimization of the database table structure to reveal the database load condition during the bottleneck of the system.
Step S2, carrying out table structure analysis on an enterprise database based on enterprise database management authority to obtain a database table structure, carrying out table structure performance bottleneck analysis on the database table structure to obtain table structure performance bottleneck data, carrying out asynchronous read-write cache processing on the database table structure according to the table structure performance bottleneck data and load fluctuation time sequence bias data to obtain an asynchronous read-write cache database;
In the embodiment of the invention, after the management authority of the enterprise database is acquired, the table structure analysis step acquires the table structure information of the database by traversing the metadata table (such as information_schema or pg_category of PostgreSQL in MySQL). Specific operations include extracting field names, data types, index configurations, constraints, etc. of each table, and generating table structure map data based on these information. And then, carrying out deep analysis on indexes, connection modes and query modes of each table by adopting a database access cost model (such as a cost estimation model or a QEP query plan evaluation technology) by utilizing a table structure performance bottleneck analysis method and combining performance bottleneck information in a database query log. The analysis method is to consider the bottlenecks of response time, index use efficiency, locking mechanism and the like of the database under high load. And calculating the local bottleneck of the query performance of the database through the association analysis of the log data and the table structure information, and generating the table structure performance bottleneck data. Based on the performance bottleneck data of the table structure and the load fluctuation time sequence bias data obtained in the step S1, asynchronous read-write caching is performed by introducing an asynchronous data caching mechanism into a database layer. In the specific operation, the data of the high-frequency read-write operation is temporarily cached in the memory by combining with a Hybrid cache strategy such as an LRU cache replacement algorithm and a frequency and recently used algorithm, and meanwhile, the asynchronous read-write processing of the database table is realized through the asynchronous I/O operation, so that the load of directly writing the database into the disk is reduced. The realization logic of the process is that hot data is directly obtained from a cache during reading, the load peak value of a database is reduced, and during writing, the consistency and durability of the data are ensured by means of a log advance record (WAL) technology, so that the delay of frequent disk I/O operation to a system is avoided. The finally output asynchronous read-write cache database has obvious response performance improvement, and ensures the stable operation of the database under high load fluctuation.
Step S3, carrying out compatibility matching on the asynchronous read-write cache database based on a containerization technology in cloud computing to obtain an asynchronous read-write compatible cache database;
In the embodiment of the invention, in the process of carrying out compatibility matching on the asynchronous read-write cache database based on the containerization technology in cloud computing, the introduction of the containerization technology aims to ensure that the asynchronous read-write cache database can keep consistent performance and compatibility in different environments. The specific operation adopts container arrangement technology such as Docker or Kubernetes, and the like, packages the asynchronous read-write cache database into a container, and ensures that the running environment (including an operating system, a dependent package, network configuration, and the like) of the asynchronous read-write cache database is consistent with other services of the cloud computing platform. In the containerized packaging process, all the dependent components and configuration files of the asynchronous read-write cache database are firstly extracted, and a unified container mirror image is created. Then, the containerization management tool performs dependency analysis and compatibility check on the mirror image, and mainly analyzes file system support of a database, a network I/O model and interaction behaviors of an asynchronous caching strategy and a cloud platform container network. Through the construction of the container mirror image and the standardized operation of the environment in the container, the compatibility of the database with storage service and network service in the cloud computing environment is ensured, and the stability of the asynchronous read-write mechanism in the high concurrency environment is verified through a series of load tests and pressure tests. And finally, generating an asynchronous read-write compatible cache database, and ensuring that the asynchronous read-write compatible cache database can run seamlessly under different cloud computing environments.
And S4, interfacing the asynchronous read-write compatible cache database into an enterprise database to execute intelligent information management.
In the embodiment of the invention, an asynchronous read-write compatible cache database is butted into an enterprise database, and in order to execute intelligent information management, the asynchronous read-write compatible cache database is connected with a core management module of the enterprise database through a database middleware or an API interface. The process adopts a database gateway technology or a database proxy technology based on a connection pool, so that efficient transfer of data between an asynchronous read-write cache layer and an enterprise database main storage layer is ensured. First, database connection pool parameters are defined and configured to ensure that the data flow achieves asynchronous non-blocking transfer between the cache and the database. Secondly, by means of a load balancing algorithm (such as a polling load balancing algorithm or a consistency hash algorithm), query requests of enterprises are distributed to a cache layer or a main storage layer, high-frequency query operation is processed in the cache preferentially, and access pressure of the main storage layer is reduced. For write operations, the consistency and reliability of the data is ensured by means of asynchronous write logging. In the data circulation process, the response time, concurrency and cache hit rate of database inquiry are monitored in real time so as to adjust the cache capacity and asynchronous read-write strategy. Finally, the intelligent information management automatically performs cache optimization and scheduling according to service requirements by monitoring the load condition of the enterprise database in real time, so as to realize intelligent management of a large-scale database.
Preferably, step S1 comprises the steps of:
S11, acquiring enterprise database log data and enterprise database management authorities;
Step S12, filling missing values of the enterprise database log data to obtain database log filling data;
S13, carrying out load fluctuation analysis on the database log filling data to obtain database load fluctuation data;
And S14, carrying out load time sequence deviation analysis on the load fluctuation data of the database to obtain the load fluctuation time sequence deviation data.
In the embodiment of the invention, in the process of acquiring the log data of the enterprise database and the management rights of the enterprise database, a rights management system is configured by a database manager (DBA) to acquire a rights set required by operation. The specific operation comprises the steps of configuring an access control strategy in a database management system, and acquiring the reading authority of log data through a role authority allocation method. The log data is typically stored in a system log table or in a separate log file within the database, which logs historical information of database operations, including query statements, transaction execution, resource usage, and the like. The latest log data is obtained periodically by adopting a script or an automatic scheduling tool (such as a Cron task) and is stored in a designated analysis node. After the data acquisition is completed, the log data is transmitted to a subsequent data processing unit through a standardized transmission protocol (such as FTP, SFTP or Intranet) so as to ensure the integrity and the safety of the data. And for missing value filling of the log data of the enterprise database, firstly cleaning the log data through a data preprocessing technology. The cleaning step includes deleting duplicate log entries, eliminating irrelevant operation information, and marking abnormal formats or incomplete logs. For the filling processing of the missing values, a time sequence-based interpolation method or a historical trend fitting technology is adopted. Specifically, the missing time period data in the log is estimated and filled by using a linear interpolation method or a lagrangian interpolation method. If the missing data amount is large, a history window regression prediction method is adopted in combination with the periodicity and time sequence relativity of the log data, missing points are filled according to the load change trend of the front and rear time periods, and complete database log filling data are generated. The filling process ensures the integrity and consistency of the data, and provides accurate data input for subsequent load analysis. When load fluctuation analysis is carried out on the database log filling data, parameters such as query frequency, transaction concurrency number, CPU and I/O resource occupancy rate and the like in the log are analyzed by using a load calculation formula, so that characteristic values of load fluctuation are obtained. The method comprises the steps of dividing log data into a plurality of time periods by adopting a sliding window method, and respectively calculating the average load, the peak load and the fluctuation amplitude of each time period by utilizing a calculation formula. Specifically, the load fluctuation analysis comprises the following calculation of setting a time window length, calculating the change rate of the load window by window, and evaluating the resource use condition of the database in different time periods. The frequency and magnitude of the load fluctuations are identified by comparing the peak and average values of the load for each time window. The analysis process is performed using a database-specific resource usage assessment tool (e.g., DB Performance Analyzer) and outputting database load fluctuation data to provide accurate load characteristic information for subsequent steps. In the process of carrying out load time sequence bias analysis on the load fluctuation data of the database, adopting a time sequence analysis method to further process the load data. The specific operation comprises the steps of converting the load fluctuation data of the database into time sequence data, and extracting time sequence characteristics of the load fluctuation by using a time domain and frequency domain analysis technology. The time domain analysis identifies the bias characteristic of the load distribution by calculating the average value, standard deviation, bias, kurtosis and other parameters of the time sequence data, and the frequency domain analysis converts the time sequence data into frequency components by Discrete Fourier Transform (DFT) or Short Time Fourier Transform (STFT) to analyze the frequency distribution of the load fluctuation. The core of the load time sequence bias analysis is to reveal the distribution non-uniformity of the database load in different time periods, and the output load fluctuation time sequence bias data can reflect the concentration degree and abnormal fluctuation characteristics of the load in peak time periods, so that scientific basis is provided for database optimization.
Preferably, step S2 comprises the steps of:
S21, carrying out table structure analysis on an enterprise database based on enterprise database management authority to obtain a database table structure;
s22, performing table structure performance bottleneck analysis on the database table structure according to the load fluctuation time sequence bias data to obtain table structure performance bottleneck data;
step S23, performing high-load read-write delay response simulation according to the table structure performance bottleneck data and the load fluctuation time sequence bias data to obtain high-load read-write delay response data;
step S24, performing read-write separation adaptation on the database table structure based on the high-load read-write delay response data to obtain database read-write separation adaptation data;
And S25, carrying out asynchronous read-write cache processing on the database table structure according to the database read-write separation adaptive data to obtain an asynchronous read-write cache database.
As an example of the present invention, referring to fig. 2, the step S2 in this example includes:
S21, carrying out table structure analysis on an enterprise database based on enterprise database management authority to obtain a database table structure;
in the embodiment of the invention, when the table structure analysis is performed on the database based on the enterprise database management authority, the table structure metadata is firstly required to be extracted from the database. The process obtains the structure information of each table through the metadata management function of the database management system, wherein the structure information comprises the column name, the data type, the index, the foreign key constraint, the data distribution and the like of the table. At the heart of table structure parsing is the conversion of the physical storage form of a database into a logical structure representation, which can be accomplished by parsing a "Data Dictionary" that is self-contained in the database system. For accurate parsing, SQL statements, such as commands like DESCRIBE table_name or SHOW CREATE TABLE table _name, are used to read the creation details of the table. The analysis process also needs to combine the access frequency in the load data to identify the key table and column, lay a foundation for the subsequent optimization step, and finally obtain the database table structure.
S22, performing table structure performance bottleneck analysis on the database table structure according to the load fluctuation time sequence bias data to obtain table structure performance bottleneck data;
In the embodiment of the invention, when the table structure performance bottleneck analysis is carried out on the database table structure according to the load fluctuation time sequence deviation data, peak load and low peak load differences identified in the load data are used, and the query frequency, the data quantity and the index service condition of the table structure are combined for evaluation. And identifying the performance bottleneck position by carrying out detailed analysis on the index of the table structure, the foreign key constraint, the data distribution and the association relation among the data tables. In the analysis process, a specific performance bottleneck evaluation algorithm, such as an efficiency evaluation method based on B-tree or hash index, is adopted to measure the index failure rate of the table and the occurrence frequency of full table scanning, and the bottleneck in the query process is found out by combining with an SQL Execution Plan (Execution Plan). The bottleneck analysis result reveals the problems of index missing, redundant data, external key locking and the like in the table structure, and finally generates the performance bottleneck data of the table structure.
Step S23, performing high-load read-write delay response simulation according to the table structure performance bottleneck data and the load fluctuation time sequence bias data to obtain high-load read-write delay response data;
In the embodiment of the invention, when high-load read-write delay response simulation is carried out according to the performance bottleneck data of the table structure and the load fluctuation time sequence bias data, a load generator tool (such as Apache JMeter) is adopted to simulate the actual running environment of the database under the high-load condition, and a plurality of read-write request combinations are set for carrying out a large number of concurrent accesses so as to simulate the response conditions of the database under different loads. The simulation process generates a delay curve by monitoring performance indexes such as I/O response time, CPU utilization rate, transaction processing delay and the like of the database in real time. In particular, specific response times for read and write operations are identified by analyzing data access latency variations under high load for table structure performance bottleneck locations. The simulation is also dynamically adjusted by combining with time sequence data to ensure that the simulation result can cover different load scenes, and finally high-load read-write delay response data is generated for the subsequent optimization adaptation process.
Step S24, performing read-write separation adaptation on the database table structure based on the high-load read-write delay response data to obtain database read-write separation adaptation data;
In the embodiment of the invention, when the read-write separation adaptation is performed on the database table structure based on the high-load read-write delay response data, delay bottlenecks of read operation and write operation are firstly identified according to the simulation data, and particularly, the parts affecting the performance of the database under high load are identified. Then, a read-write separation strategy is implemented, and the read operation and the write operation are respectively distributed to different database examples or partitions so as to reduce resource competition. The core of the read-write separation adaptation is to replicate the read operations in the database, synchronize the data to multiple read-only instances, and concentrate the write operations in the master database. In order to ensure data consistency, a real-time data synchronization mechanism based on transaction logs, such as a replying function in MySQL, is adopted to ensure that the synchronization delay between the main database and the read-only database is minimized. The whole adapting process also needs to optimize the routing rule of the read-write request, dynamically distributes the read request to different examples through a load equalizer, and finally obtains the read-write separation adapting data of the database.
And S25, carrying out asynchronous read-write cache processing on the database table structure according to the database read-write separation adaptive data to obtain an asynchronous read-write cache database.
In the embodiment of the invention, when asynchronous read-write caching processing is carried out on the database table structure according to the database read-write separation adaptation data, the caching technology is used for optimizing the read-write efficiency of the database. First, a cache layer is introduced into a database, and a distributed cache system (such as Redis or Memcached) is used for storing frequently accessed read operation data. The key of asynchronous read-write buffer is to realize asynchronous processing of read-write request, read operation is preferentially obtained from buffer, and write operation is asynchronously submitted to database through transaction queue. Delayed commit of write operations by introducing a multithreaded queue manager while maintaining consistency requirements for database write operations. The asynchronous read-write caching process also needs to consider a cache failure policy, adopts an LRU (LEAST RECENTLY Used) algorithm to dynamically update the cache data, ensures that the system maintains stable read-write performance in high load, and finally generates an asynchronous read-write cache database for supporting the efficient operation of the enterprise database.
Preferably, step S22 comprises the steps of:
s221, performing field constraint structure analysis on a database table structure to obtain field constraint structure data;
Step S222, performing constraint nesting level calculation on the database table structure according to the field constraint structure data to obtain field constraint nesting level data;
Step S223, performing trigger association logic retrieval on the database table structure to obtain trigger association logic data;
step S224, carrying out event triggering logic complexity evaluation on the database table structure according to the trigger association logic data to obtain event triggering logic complexity data;
Step S225, performing nonlinear time complexity calculation based on event triggering logic complexity data and field constraint nested level data to obtain nonlinear time complexity data;
Step S226, performing execution complexity load fluctuation mapping on the execution nonlinear time complexity data according to the load fluctuation time sequence bias data to obtain execution complexity load fluctuation data;
And step S227, performing table structure performance bottleneck analysis on the database table structure according to the execution complexity load fluctuation data and the execution nonlinear time complexity data to obtain table structure performance bottleneck data.
In the embodiment of the invention, when the field constraint structure analysis is performed on the database table structure, firstly, the definition information of the field is required to be extracted from the table structure, including the type, the length, the default value, whether the field is allowed to be empty, the unique constraint, the foreign key constraint and the like. The definition of these field constraints directly affects the integrity and data consistency of the database tables. A field constraint structure model is constructed by systematically analyzing constraint conditions of each field, and the constraint relations among the fields, such as dependencies of foreign keys and primary keys, are identified by combining the relevance of the fields in the table. In the specific analysis process, a constraint analysis algorithm (such as a recursion constraint tree analysis algorithm) is adopted to conduct hierarchical decomposition, constraint relations of fields are extracted step by step, field constraint structure data are finally generated, and a foundation is laid for subsequent nested hierarchical analysis. When constraint nesting level calculation is performed on the database table structure according to the field constraint structure data, the level relation of the field constraint is needed to be defined, and the nesting situations of main key constraint, foreign key constraint, uniqueness constraint and the like are included. And constructing a constraint nested graph of each table by adopting a tree structure through analyzing hierarchical dependence in the field constraint structure. The graph evaluates the nesting level of the fields by calculating the hierarchical depth, hierarchical nesting frequency, and complexity of each constraint. To accurately compute nesting levels, a graph traversal algorithm (e.g., depth-first search DFS) is applied to analyze the dependency and constraint chains among fields layer by layer, identifying bottlenecks and redundancy constraints in nesting relationships. Finally, field constraint nested level data is generated, and a foundation is laid for subsequent logic complexity evaluation. When the trigger association logic search is performed on the database table structure, firstly, trigger logic defined in the database needs to be traversed, and event trigger conditions, trigger occasions (such as trigger during insertion, update or deletion operation) and trigger execution operation logic related to the trigger on each table are searched by querying the metadata table of the database. The execution order of each trigger and the conditional logic at the time of triggering are obtained by an SQL query (e.g., SHOW TRIGGERS or access to a trigger management table in a database). And then, reversely analyzing the triggers by applying a logic backtracking technology, analyzing the association relation between the triggers, and identifying the dependency of the triggers. finally, constructing a correlation diagram of the trigger through a recursive logic search algorithm, and generating trigger correlation logic data, thereby providing basis for subsequent complexity evaluation. When the event triggering logic complexity evaluation is carried out on the database table structure according to the trigger association logic data, firstly, the length of the trigger chain, the quantity of concurrent triggers and the execution sequence of the trigger events are identified by traversing the dependency relationship in the trigger association graph. In the complexity evaluation process, a combined complexity analysis algorithm is applied to calculate the number of logic branches in the trigger chain and the number of events triggered by each branch. By analyzing the execution logic (such as recursive calls, nested calls, etc.) of each trigger, the nesting depth and execution path complexity of the trigger are determined. In particular for those logic with multi-level trigger dependencies, a multi-level trigger complexity model (e.g., a nested tree model) is used to quantify the execution complexity of the triggers. Event-triggered logic complexity data is ultimately generated, indicating a critical performance bottleneck in the database trigger system. When nonlinear time complexity calculation is executed based on event triggering logic complexity data and field constraint nested level data, firstly, the nested level of the trigger execution logic complexity and the field constraint is combined, and the response time of a read-write request of a database under high load is comprehensively considered. The time complexity function is constructed by analyzing the constraint relation between the execution path and the fields of the trigger. The function calculates worst case response time and best case response time of the trigger chain under a concurrent scene according to a complexity analysis model, such as a large O sign analysis method. Meanwhile, by recursively traversing the structure of the database table, the nonlinear influence of the constraint nesting depth on the performance is measured, and the execution nonlinear time complexity data is generated. This data reflects the response time behavior of the database in the case of multi-level nesting and trigger concurrency. When complex load fluctuation mapping is performed on the execution nonlinear time complexity data according to the load fluctuation time sequence bias data, time sequence features in the load fluctuation data are analyzed to identify time periods of a load peak period and a load low peak period. Then, matching the time complexity function in the nonlinear time complexity data with the time sequence data in the load fluctuation data, and constructing a mapping relation between the time complexity and the load fluctuation. And (3) adopting a time sequence cross mapping algorithm (such as Granger causality test), evaluating the dynamic influence of load fluctuation on the execution complexity of the database, and generating time complexity change curves under different load levels. By the mapping analysis, key time periods affecting the execution time complexity in the load fluctuation are identified, and the execution complexity load fluctuation data is finally generated, so that accurate load mapping information is provided for performance bottleneck analysis. And when the table structure performance bottleneck analysis is carried out on the database table structure according to the execution complexity load fluctuation data and the execution nonlinear time complexity data, the table structure performance bottleneck under the condition of high load is identified by combining the relation between the execution time complexity and the load fluctuation. In the analysis process, firstly, according to the time sequence characteristics of load fluctuation, determining the performance degradation point of the database operation under high load. Then, the factors such as trigger execution delay, constraint nesting bottleneck and index conflict marked in the nonlinear time complexity data are taken as key analysis objects. By adopting a performance backtracking analysis technology, the position and the reason of the performance bottleneck are determined through slow query, lock waiting and resource contention problems in the process of the Execution Plan (Execution Plan) backtracking operation of the database. And finally generating table structure performance bottleneck data for optimizing the database table structure design and trigger logic.
Preferably, performing the nonlinear time complexity calculation based on the event-triggered logic complexity data and the field constraint nested hierarchy data comprises the steps of:
respectively carrying out nonlinear execution efficiency simulation on the event triggering logic complexity data and the field constraint nesting level data to respectively obtain event triggering nonlinear execution efficiency data and nonlinear field constraint execution efficiency data;
performing hierarchical execution complexity error correction calculation on the event trigger nonlinear execution efficiency data to obtain trigger execution complexity error correction data;
performing efficiency convergence average difference calculation on the nonlinear field constraint execution efficiency data to obtain constraint efficiency convergence average difference data;
And performing nonlinear time complexity calculation based on the trigger execution complexity error correction data and the constraint efficiency convergence average difference data to obtain the execution nonlinear time complexity data.
In the embodiment of the invention, nonlinear execution efficiency simulation is performed aiming at event-triggered logic complexity data. This step calculates the execution efficiency of event triggers by analyzing the execution paths of the triggers in the database. In the execution efficiency simulation process, a trigger path tracking algorithm (such as a dependency analysis method based on a directed acyclic graph) is adopted to analyze each operation of the trigger chain and the dependency relationship thereof step by step. In the parsing process, concurrency and nesting behaviors among the triggers are identified, and particularly performance bottlenecks of the triggers under large-scale concurrency operation are identified. In order to perform nonlinear simulation, time-consuming operation in a trigger path is simulated by using a polynomial approximation method, particularly aiming at the influence of trigger nesting logic and concurrent conflict, the growth trend of nonlinear behavior is deduced, and finally event-triggered nonlinear execution efficiency data is generated. when nonlinear execution efficiency simulation is performed based on field constraint nested level data, field constraint in a database is deeply analyzed, and simulation analysis is performed particularly on nested levels of foreign keys, primary keys and unique constraints. A recursive traversal algorithm (e.g., a tree-based hierarchical recursive algorithm) is employed to hierarchically divide the nested levels and evaluate the execution cost of each level of constraints. In the simulation process, the constraint execution behaviors in the nested hierarchy are simulated by using an exponential growth model, and the execution efficiency decline trend in the constraint verification process is deduced when the constraint hierarchy is continuously increased. And evaluating the execution time and resource consumption of each level of constraint under the large data volume processing by a nonlinear time complexity analysis method, and finally generating nonlinear field constraint execution efficiency data. When performing hierarchical execution complexity error correction on event-triggered nonlinear execution efficiency data, firstly, the influence of trigger logic of different levels on the execution efficiency needs to be considered. The level error in the trigger execution path is analyzed and adjusted by an error correction algorithm, such as a kalman filter algorithm. The complexity of the execution path of the trigger is reduced due to logic nesting or the execution error is increased, so that bottleneck points and delay hot spots in the trigger chain are identified in the error correction process, and the execution time is adjusted according to the bottlenecks. Particularly in high concurrency situations, the trigger experiences an abnormal drop in execution efficiency due to waiting or lock contention, and the error correction algorithm ensures the accuracy of the final output data by correcting these abnormal points. Finally, trigger execution complexity error correction data is generated, and the fluctuation of the execution efficiency of the trigger system and the corrected execution complexity are reflected. When efficiency convergence average difference calculation is performed based on nonlinear field constraint execution efficiency data, firstly, efficiency fluctuation generated in the field constraint execution process is subjected to statistical analysis, and average difference analysis is performed on constraint execution efficiency of each layer by using an average difference calculation method (such as a Bayesian mean method). In the specific operation, the execution efficiency data of each layer constraint is subjected to layering processing, and the convergence speed and the fluctuation range of each layer are calculated. And (3) identifying the stability of the execution efficiency in the constraint nesting level through convergence average difference calculation, and evaluating whether the execution efficiency of the field constraint can be stabilized in a certain range in long-time execution. The step can not only identify the execution bottleneck of the constraint through the average difference calculation, but also provide an efficiency reference for the subsequent complexity calculation. And finally generating constraint efficiency convergence average difference data, which shows the balance of the efficiency in different levels of constraint. When nonlinear time complexity calculation is performed based on trigger execution complexity error correction data and constraint efficiency convergence average difference data, firstly, the influence of trigger execution efficiency and field constraint efficiency on time complexity is comprehensively analyzed by combining the two types of data. In the process, a complex complexity analysis method is adopted to combine the complexity of triggering execution and the complexity of constraint execution, so as to construct a comprehensive time complexity function. by means of time complexity theory (such as a log-linear complexity model), the execution efficiency of the trigger and the field constraint is dynamically weighted and calculated, and the execution time of the trigger and the field constraint under different load conditions is calculated. Especially, aiming at scenes with unbalanced complexity, nonlinear growth factors in complexity calculation are adjusted, and the time complexity model is ensured to accurately reflect the overall execution behavior of the system. And finally generating the execution nonlinear time complexity data, and providing important reference basis for performance optimization and load prediction of the database.
Preferably, performing the hierarchical execution complexity error correction calculation on the event-triggered nonlinear execution efficiency data comprises the steps of:
Performing event trigger execution hierarchy retrieval on the event trigger nonlinear execution efficiency data to obtain event trigger execution hierarchy data;
Performing nested level-to-level execution efficiency variance calculation on the event trigger nonlinear execution efficiency data according to the event trigger execution level data to obtain nested level execution efficiency variance data;
performing nested overlapping redundant execution efficiency identification on the event-triggered nonlinear execution efficiency data based on the nested hierarchy execution efficiency variance data to obtain nested overlapping redundant execution efficiency data;
Overlapping error correction processing is carried out on the event trigger nonlinear execution efficiency data according to the nested overlapping redundant execution efficiency data, so that trigger nonlinear execution correction efficiency data is obtained;
and performing hierarchical execution complexity error correction calculation on the trigger nonlinear execution correction efficiency data based on the divide-and-conquer method to obtain trigger execution complexity error correction data.
In the embodiment of the invention, when the event trigger nonlinear execution efficiency data is subjected to event trigger execution level retrieval, the execution level of each trigger in the event trigger process is identified by analyzing the logic structure of the trigger in the database. In the step, a hierarchical traversal algorithm (such as a depth-first search algorithm) is adopted to analyze the trigger chains layer by layer, and the calling relation and the nesting depth of each trigger are recorded. In the execution process, the direct and indirect dependency relations among the triggers are classified, the triggers of all levels are arranged into an execution level table through a tree structure, and event triggering execution level data are generated. The data clarifies the execution sequence and the nesting relation of the triggers, and lays a foundation for subsequent efficiency analysis. And performing nested inter-level execution efficiency variance calculation on the event-triggered nonlinear execution efficiency data based on the event-triggered execution level data. In this step, the execution efficiency of each level trigger is compared layer by layer using an analysis of variance method (such as an ANOVA-based analysis of variance model). The method comprises the steps of calculating the mean value and variance of the execution efficiency of each trigger level by counting the execution time of each trigger level, comparing the efficiency fluctuation among different levels, and analyzing the imbalance of the execution efficiency in different level nesting. In particular, for the hierarchy with deeper nesting, the phenomenon of obviously reduced efficiency is easy to occur, and the occurrence hierarchy of the phenomenon can be accurately positioned through variance calculation. and finally, generating nested level execution efficiency variance data, and providing basis for subsequent redundancy identification. And performing nested overlapping redundant execution efficiency identification on the event-triggered nonlinear execution efficiency data according to the nested hierarchical execution efficiency variance data. In this step, an overlapping redundancy detection algorithm (such as a redundancy detection algorithm based on pattern recognition) is adopted, and the redundant trigger execution paths of the hierarchy with larger variance are further analyzed. By pattern matching the call chains of each trigger, redundant operations that are repeatedly performed in different levels, particularly the portion of repeated triggers in multiple nesting, are identified. In the redundancy recognition process, the dependency relationship among triggers, repeated logic conditions and calling conflict of parallel triggers are focused. Finally, nested overlapping redundant execution efficiency data is generated that clearly shows the trigger redundancy portion and its impact on overall efficiency. And based on the nested overlapping redundant execution efficiency data, overlapping error correction processing is carried out on the event-triggered nonlinear execution efficiency data. This step optimizes and adjusts the execution time of the redundant paths by correcting the detected redundant execution paths using an error correction algorithm (e.g., a taylor series expansion method). In a specific operation, by removing or simplifying redundant trigger chains, repeated calls are reduced, and corrected execution time and efficiency are recalculated. Particularly, under the high concurrency condition, the execution path of the trigger redundancy has a great influence on the whole system load, and the resource consumption of the system can be obviously reduced by correcting the redundancy path. And finally generating trigger nonlinear execution correction efficiency data, wherein the data can reflect the optimized trigger execution path and the time efficiency change condition thereof. And performing hierarchical execution complexity error correction calculation on the trigger nonlinear execution correction efficiency data based on a divide-and-conquer method. Divide-and-conquer approach breaks down the complex trigger execution path into multiple smaller sub-questions, each of which is solved independently, and then merges the results. In the correction process, firstly, the hierarchical execution path of the trigger is divided, the execution complexity of each hierarchy is analyzed independently, and the error of each hierarchy is corrected layer by performing step calculation of the complexity. In each sub-problem, the execution time and complexity are optimized by using a dynamic programming method, so that the error is ensured to be continuously converged in the correction process. And finally, merging correction results of all the sub-problems to obtain global level execution complexity error correction data, wherein the data is used for measuring the overall execution complexity of the trigger system and providing basis for subsequent performance optimization.
Preferably, step S25 comprises the steps of:
step S251, carrying out concurrent read-write frequency simulation on the database table structure according to the database read-write separation adaptation data to obtain concurrent read-write frequency simulation data;
Step S252, performing read-write cache structure design on a database table structure based on concurrent read-write frequency simulation data to respectively obtain read cache structure data and write cache structure data;
Step 253, performing asynchronous thread matching on the read cache structure data and the write cache structure data to respectively obtain read cache asynchronous thread structure data and write cache asynchronous thread structure data;
Step S254, conflict concurrency control logic design is carried out on the read cache asynchronous thread structure data and the write cache asynchronous thread structure data, and read-write conflict logic control data is obtained;
And S255, carrying out asynchronous read-write cache processing on the database table structure according to the read cache asynchronous thread structure data, the write cache asynchronous thread structure data and the read-write conflict logic control data to obtain an asynchronous read-write cache database.
In the embodiment of the invention, when the concurrent read-write frequency simulation is performed on the table structure of the database, the read-write request frequency per second is obtained by analyzing the historical read-write operation log of the database and adopting the concurrent transaction analysis technology (such as a timestamp ordering algorithm in a transaction concurrent model) to count the read-write requests of different tables. And then, according to read-write separation adaptive data of a database, a read-write proportion calculation formula based on a mathematical model is applied, concurrent read-write operation in a multithreading environment is simulated, and the read-write frequency is gradually adjusted to ensure that the read-write frequency is consistent with an actual running environment. In the concurrent simulation process, the conflict and bottleneck problem of read-write operation in the peak period are focused, and finally concurrent read-write frequency simulation data are generated, wherein the data can quantify the read-write pressure distribution of different table structures under the concurrent condition. And designing a read-write cache structure of the database table structure based on the concurrent read-write frequency simulation data. In the step, the hierarchical cache design theory is used, and access data of the table is distributed according to the frequency of read-write operation in the database and the two parts of the read cache and the write cache. Specifically, for a table with frequent read operations, the read cache structure is preferably designed, a LRU-like (least recently used) cache replacement algorithm is applied, and cache optimization is performed on frequently accessed data blocks in the table. For a table with frequent write operation, an asynchronous write cache mechanism is adopted, and the write cache efficiency is improved through a merging write technology (such as a batch write technology). Finally, respectively generating read cache structure data and write cache structure data, wherein the data records the cache design details of each table in detail, including the cache size and the cache strategy. And performing asynchronous thread matching operation on the cache system according to the designed read cache structure data and write cache structure data. In this step, a thread pool optimization algorithm (e.g., a dynamic thread allocation algorithm) is used to allocate an independent asynchronous thread to each read cache and write cache structure to implement asynchronous parallel processing. The number of threads in the thread pool is dynamically adjusted by analyzing the access frequency and the operation complexity of each cache area, so that the concurrency performance of read-write operation is ensured to be maximized. In the matching process, high-frequency read-write operation is preferentially distributed to multi-thread processing, so that thread blocking is avoided. Finally, generating read cache asynchronous thread structure data and write cache asynchronous thread structure data, and describing the matching relation and allocation strategy of the cache structure and the asynchronous threads respectively. And after the read-write cache structure completes asynchronous thread matching, conflict concurrency control logic design is carried out on asynchronous read-write operation. This step applies a control strategy combining a lock mechanism And a lock-free mechanism, and uses, for example, optimistic locks (Optimistic Locking) And CAS (computer-And-Swap) based lock-free algorithms to ensure that read And write operations do not compete for resources And collide with data in a high concurrency environment. In specific operations, a lock-free mechanism is preferentially used for read operations to improve concurrent processing performance, and optimistic locks are applied for write operations to avoid multi-threaded write conflicts. Meanwhile, the conflict control is refined by utilizing a transaction isolation technology (such as multi-version concurrency control, MVCC) so as to ensure the correctness of the read-write operation. Finally, read-write conflict logic control data are generated, and conflict solutions under various concurrent scenes are clearly recorded. And carrying out asynchronous read-write caching processing on the database table structure according to the read-cache asynchronous thread structure data, the write-cache asynchronous thread structure data and the read-write conflict logic control data. According to the method, the read cache and the write cache are independently operated in the asynchronous thread, so that parallelization of read-write operation is realized, and the overall processing capacity of the database is improved. In the specific operation, the LRU strategy is used in the process of the read cache processing, the common data is asynchronously read and the query result is quickly returned, and the write cache processing ensures the high efficiency and consistency of the write operation by combining batch writing and transaction log. By combining conflict control logic, the concurrency problem between read-write operations is effectively avoided, and the system is ensured to be still stable under the condition of high load. Finally, generating an asynchronous read-write cache database, wherein the structure of the asynchronous read-write cache database is optimized, and the processing efficiency and stability of the database in a large-scale concurrency scene can be remarkably improved.
Preferably, the read-write cache structure design for the database table structure based on the concurrent read-write frequency simulation data comprises the following steps:
Performing field reading hot spot frequency sequencing of different time periods on the database table structure based on the concurrent reading and writing frequency simulation data to obtain field reading hot spot frequency sequencing data;
giving a hot spot field priority value to the field reading hot spot frequency ordering data to obtain the hot spot field priority value;
performing LRU (line-of-sight) preferential starting strategy formulation on field reading hotspot frequency sequencing data according to the hotspot field priority value to obtain a reading field preferential starting strategy;
performing read cache structure design on field read hot spot frequency sequencing data according to a read field priority starting strategy to obtain read cache structure data;
performing field writing request density analysis on the database table structure based on the concurrent reading and writing frequency simulation data to obtain field writing request density data;
Performing cache slicing processing on the field writing request density data by using a cache database to obtain field writing cache slicing data;
Performing batch write-back cache policy formulation on field write-in cache fragmented data to obtain a batch write-back cache policy;
And performing write cache structure design according to the field write cache fragment data and the batch write-back cache policy to obtain write cache structure data.
In the embodiment of the invention, based on concurrent read-write frequency simulation data, the read frequencies of all fields in a database table structure in different time periods are statistically analyzed, and the read times in each time period are quantified by adopting a time sequence analysis method. By ordering the read operations in different time periods, the read hotspot frequency for each field is derived. In the specific operation, a day is divided into a plurality of time slices by using a discrete time model, the reading times of each field in each time slice are recorded, and finally the field reading hotspot frequency ordering data are obtained according to the ordering of the reading frequencies from high to low. The data can clearly reflect the access hot spots of each field in different time periods, and after the field reading hot spot frequency ordering data is obtained, the giving of the priority value is carried out based on the reading hot spot degree of the field. Firstly, fields are grouped according to the reading frequency, and different priorities are given to the fields in different groups by adopting a hierarchical weight distribution method. The assignment of weight values is based on the degree of hot spots for each field, and fields with high reading frequency will be given higher priority. The priority is set by normalizing the priority values of all the fields to a specific interval in a linear normalization mode, so that obvious differences among the field priority values are ensured. And finally obtaining a hotspot field priority value, wherein the hotspot field priority value is used for guiding the subsequent cache policy establishment. And according to the hotspot field priority value, performing LRU (least recently used) optimization starting policy formulation on the field read hotspot frequency ordering data. In this step, the LRU cache replacement algorithm is used to preferentially select the hot spot field during the field read cache start. Specifically, a high priority field is placed at the front end of the cache queue, and a low priority field enters the back end, based on the priority value of the hot spot field. The LRU algorithm dynamically adjusts the data in the cache according to the frequency of use of the field, and when the cache space is insufficient, the field which is not accessed for the longest time is evicted from the cache. Through the formulation of the strategy, the fields accessed by high frequency are ensured to be processed preferentially in the cache, and finally, the read field priority starting strategy is generated. And according to the read field priority starting strategy, performing read cache structure design on the field read hot spot frequency ordering data. In the step, a hierarchical cache design method is used, the hot spot field is cached in a partition mode according to the priority, and a multi-level cache mechanism is set. The primary cache is used for storing the most frequently accessed fields, and the secondary cache is used for storing secondary hot spot fields. Through the multi-level cache structure, the reading efficiency of fields with different priorities is ensured to be maximized. Meanwhile, for the cache allocation of the high priority field, a dynamic adjustment algorithm can be used to automatically adjust the size of the cache space to cope with the change of data access. Finally, the read cache structure data is obtained, and the allocation situation and the management strategy of the cache area are described in detail. And analyzing the field writing request density in the database table structure based on the concurrent reading and writing frequency simulation data. the density of the write requests of the fields is calculated by counting the number of writes of each field and the size of the data block using a density estimation method. The density Estimation is based on the writing frequency and data volume of the fields, and a writing density distribution map of each field is generated by adopting a nuclear density Estimation (KERNEL DENSITY Estimation) technology. The distribution diagram can intuitively display the writing frequency and the data pressure condition of each field, provide data support for the subsequent writing cache design, and finally obtain the field writing request density data. And performing cache slicing processing on the field writing request density data by using a cache database. Firstly, according to the writing request density data, the writing pressure of the fields is segmented, and a partition strategy is adopted to distribute the high-density fields to a plurality of cache segments so as to balance the pressure of each cache region. The slicing process uses a hash slicing algorithm to map frequently written fields into different cache blocks by hashing the fields. Therefore, write-in conflict and data bottleneck in the cache can be effectively avoided, and stable performance under the high concurrent write-in scene is ensured. Finally, field writing cache fragment data is generated, and the data describes the cache allocation scheme of each field in detail. And according to the field write-in cache fragment data, making a batch write-back cache strategy for the cache write-back operation. The step adopts a batch write optimization technology, and a plurality of small write requests are combined into one large request, so that the write operation times of a database are reduced, and the write efficiency is improved. In a specific operation, based on a delayed write (Write Behind) mechanism, data in a cache is accumulated, a write-back threshold is set, and when the data quantity reaches the threshold, batch write-back operation is triggered. And in combination with the writing request density of the fields, the high-density fields are written back to the database preferentially, so that the writing efficiency of the system is ensured to be maximized. And finally obtaining a batch write-back caching strategy. And performing write cache structure design according to the field write cache fragment data and the batch write cache strategy. In this step, an asynchronous write cache mechanism is used to design a multi-level write cache architecture by combining partitioning with batch writing. Firstly, an independent write buffer zone is set up for a high-frequency write-in field, and a dedicated write-in thread is allocated for the write-in field, so that the write-in operation is ensured not to be influenced by other fields. Secondly, the data in the cache are written back to the database in batches through a batch write-back strategy, so that the write-in pressure of the database is reduced. Finally, an asynchronous writing mechanism is used to ensure that the writing operation is carried out in the background, and the reading and writing performance of the foreground is not affected. And finally generating write cache structure data.
Preferably, the conflict concurrency control logic design for the read cache asynchronous thread structure data and the write cache asynchronous thread structure data comprises the following steps:
Performing conflict field simulation on the read cache asynchronous thread structure data and the write cache asynchronous thread structure data to obtain conflict field simulation data;
performing cache field line level lock granularity optimization on the conflict field simulation data to obtain conflict field line level lock granularity optimization data;
Performing optimistic lock concurrency control processing on conflict field positioning data according to conflict field line-level lock granularity optimization data to obtain conflict field optimistic lock control data;
and carrying out conflict concurrency control logic design based on the conflict field optimistic lock control data and the conflict field line level lock granularity optimization data to obtain read-write conflict logic control data.
In the embodiment of the invention, the simulation analysis of the conflict field is carried out on the read cache asynchronous thread structure data and the write cache asynchronous thread structure data, so as to identify the field with concurrent conflict in the read-write operation process. The method adopts a conflict detection algorithm based on concurrent access, and identifies the fields which are read or written by a plurality of threads simultaneously in the same time period by analyzing the accessed fields in the cache. By modeling these conflict fields, detailed statistics of the access frequency and the degree of conflict of the conflict fields can be generated. And comparing the behaviors of different threads accessing the same field at different times by adopting a space-time overlap analysis technology, and finally obtaining the conflict field simulation data. The data provides basic information for subsequent lock mechanism optimization. And after the conflict field simulation data is obtained, performing optimization of cache field line level lock granularity. Based on the simulation data, setting the row-level lock granularity for each conflict field by using a dynamic lock granularity adjustment algorithm. The algorithm flexibly adjusts granularity of the lock according to the frequency and the intensity of field conflict, and ensures that the lock operation under the high concurrency condition can not influence the overall performance of the system. By refining the row level lock, the range of lock conflicts can be reduced, and locking of the entire table or large blocks of data is avoided. The optimized row-level lock granularity ensures that the granularity of lock operation is small enough when the conflict field is accessed concurrently, so that lock competition is reduced, and the concurrency processing capacity of the system is improved. And finally obtaining conflict field line-level lock granularity optimization data, and describing the locking strategy of each field. And performing optimistic lock concurrency control processing on the conflict field positioning data based on the conflict field line-level lock granularity optimization data. In this step, an optimistic lock mechanism is employed to detect concurrency conflicts by recording the version number of each conflict field. The specific operation comprises setting a unique version number for each field, firstly reading the version number during read-write operation, then checking whether the version number changes during write-in operation, and allowing the write-in operation to be submitted if the version number does not change. The introduction of optimistic locks can reduce unnecessary lock operations, thereby improving concurrency performance. The process is based on a version control technology, and synchronously processes the multithreading read-write operation, so that the concurrency conflict is ensured to be effectively controlled. And finally, conflict field optimistic lock control data is generated, and a specific concurrency control mechanism and a field version number management method are provided. The conflict concurrency control logic is designed by combining conflict field optimistic lock control data and conflict field line level lock granularity optimization data. In this step, a set of concurrency control logic is designed by combining the row level lock with the optimistic lock by using a conflict control algorithm. The logic controls by dynamically selecting the use of an optimistic lock or a row-level lock based on the frequency of access to the conflict field and the strength of the conflict. In the case of lighter conflicts, optimistic locks are used preferentially for concurrency control, while in the case of heavier conflicts, row-level locks are used to ensure data consistency. By the hybrid control strategy, the read-write concurrency optimal control can be realized on the premise of ensuring the stability of the system. Finally, read-write conflict logic control data is obtained, and specific execution rules and flows of the control logic are described in detail.
Preferably, the present invention also provides a cloud computing-based intelligent information management system for executing the cloud computing-based intelligent information management method as described above, the cloud computing-based intelligent information management system comprising:
The load time sequence deviation analysis module is used for obtaining the log data of the enterprise database and the management authority of the enterprise database, carrying out load time sequence deviation analysis on the log data of the enterprise database and obtaining load fluctuation time sequence deviation data;
The asynchronous read-write cache processing module is used for carrying out table structure analysis on the enterprise database based on the enterprise database management authority to obtain a database table structure, carrying out table structure performance bottleneck analysis on the database table structure to obtain table structure performance bottleneck data, carrying out asynchronous read-write cache processing on the database table structure according to the table structure performance bottleneck data and the load fluctuation time sequence bias data to obtain an asynchronous read-write cache database;
The compatibility matching module is used for carrying out compatibility matching on the asynchronous read-write cache database based on a containerization technology in cloud computing to obtain an asynchronous read-write compatibility cache database;
And the execution docking module is used for docking the asynchronous read-write compatibility cache database into the enterprise database so as to execute intelligent information management.
The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.
The foregoing is only a specific embodiment of the invention to enable those skilled in the art to understand or practice the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202411613478.1A CN119127975B (en) | 2024-11-13 | 2024-11-13 | Intelligent information management method and system based on cloud computing |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202411613478.1A CN119127975B (en) | 2024-11-13 | 2024-11-13 | Intelligent information management method and system based on cloud computing |
Publications (2)
Publication Number | Publication Date |
---|---|
CN119127975A true CN119127975A (en) | 2024-12-13 |
CN119127975B CN119127975B (en) | 2025-02-07 |
Family
ID=93750367
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202411613478.1A Active CN119127975B (en) | 2024-11-13 | 2024-11-13 | Intelligent information management method and system based on cloud computing |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN119127975B (en) |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH06314299A (en) * | 1993-04-28 | 1994-11-08 | Hitachi Ltd | Data base control method |
US7822727B1 (en) * | 2004-07-02 | 2010-10-26 | Borland Software Corporation | System and methodology for performing read-only transactions in a shared cache |
US10331657B1 (en) * | 2015-09-28 | 2019-06-25 | Amazon Technologies, Inc. | Contention analysis for journal-based databases |
CN117951119A (en) * | 2024-02-21 | 2024-04-30 | 重庆邮电大学 | A database performance optimization method based on cloud computing |
CN118467465A (en) * | 2024-07-09 | 2024-08-09 | 青岛市胶州中心医院 | File information data management method based on digitization |
CN118708593A (en) * | 2024-07-10 | 2024-09-27 | 北京顺达尼科技有限公司 | A data processing method and system in a high-concurrency scenario of a data center |
CN118838889A (en) * | 2024-07-18 | 2024-10-25 | 辽宁工业大学 | Time sequence data optimized storage method |
CN118885464A (en) * | 2024-08-21 | 2024-11-01 | 上海米喜网络科技有限公司 | A classified cloud database management system and method |
-
2024
- 2024-11-13 CN CN202411613478.1A patent/CN119127975B/en active Active
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH06314299A (en) * | 1993-04-28 | 1994-11-08 | Hitachi Ltd | Data base control method |
US7822727B1 (en) * | 2004-07-02 | 2010-10-26 | Borland Software Corporation | System and methodology for performing read-only transactions in a shared cache |
US10331657B1 (en) * | 2015-09-28 | 2019-06-25 | Amazon Technologies, Inc. | Contention analysis for journal-based databases |
CN117951119A (en) * | 2024-02-21 | 2024-04-30 | 重庆邮电大学 | A database performance optimization method based on cloud computing |
CN118467465A (en) * | 2024-07-09 | 2024-08-09 | 青岛市胶州中心医院 | File information data management method based on digitization |
CN118708593A (en) * | 2024-07-10 | 2024-09-27 | 北京顺达尼科技有限公司 | A data processing method and system in a high-concurrency scenario of a data center |
CN118838889A (en) * | 2024-07-18 | 2024-10-25 | 辽宁工业大学 | Time sequence data optimized storage method |
CN118885464A (en) * | 2024-08-21 | 2024-11-01 | 上海米喜网络科技有限公司 | A classified cloud database management system and method |
Non-Patent Citations (2)
Title |
---|
江勇;苗宗利;王伟;段世凯;刘财政;支孟轩;: "基于变化数据捕获机制的分布式缓存一致性策略", 计算机系统应用, no. 12, 15 December 2016 (2016-12-15) * |
马晶;: "信息系统中数据库的性能优化", 科技信息, no. 14, 15 May 2010 (2010-05-15) * |
Also Published As
Publication number | Publication date |
---|---|
CN119127975B (en) | 2025-02-07 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11429584B2 (en) | Automatic determination of table distribution for multinode, distributed database systems | |
Ma et al. | Query-based workload forecasting for self-driving database management systems | |
Van Aken et al. | Automatic database management system tuning through large-scale machine learning | |
Kester et al. | Access path selection in main-memory optimized data systems: Should I scan or should I probe? | |
US9424282B2 (en) | Online reorganization of hybrid in-memory databases | |
US9031826B2 (en) | Method and apparatus for simulating operation in a data processing system | |
US20180218038A1 (en) | Database optimization based on forecasting hardware statistics using data mining techniques | |
Jindal et al. | Microlearner: A fine-grained learning optimizer for big data workloads at microsoft | |
Singhal et al. | Predicting SQL query execution time for large data volume | |
Zhang et al. | HTAP databases: A survey | |
Ouared et al. | DeepCM: Deep neural networks to improve accuracy prediction of database cost models | |
Li et al. | A resource-aware deep cost model for big data query processing | |
Zhou et al. | Febench: A benchmark for real-time relational data feature extraction | |
Lu et al. | On the auto-tuning of elastic-search based on machine learning | |
CN119127975B (en) | Intelligent information management method and system based on cloud computing | |
Gu et al. | Adaptive cache policy scheduling for big data applications on distributed tiered storage system | |
Li et al. | dCCPI-predictor: A state-aware approach for effectively predicting cross-core performance interference | |
Castro Fernandez et al. | Meta-dataflows: Efficient exploratory dataflow jobs | |
Guo et al. | XDataExplorer: a three-stage comprehensive self-tuning tool for Big Data platforms | |
Hurdelhey et al. | Workload-Driven Data Placement for Tierless In-Memory Database Systems | |
Wu et al. | Fast and Accurate Optimizer for Query Processing over Knowledge Graphs | |
Rao et al. | SODA: A semantics-aware optimization framework for data-intensive applications using hybrid program analysis | |
Wang et al. | Automatic single table storage structure selection for hybrid workload | |
Munir et al. | Intermediate results materialization selection and format for data-intensive flows | |
US12277116B2 (en) | Query plan cache shadow modes |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |