[go: up one dir, main page]

CN104573428B - A kind of method and system for improving server cluster resource availability - Google Patents

A kind of method and system for improving server cluster resource availability Download PDF

Info

Publication number
CN104573428B
CN104573428B CN201310475814.6A CN201310475814A CN104573428B CN 104573428 B CN104573428 B CN 104573428B CN 201310475814 A CN201310475814 A CN 201310475814A CN 104573428 B CN104573428 B CN 104573428B
Authority
CN
China
Prior art keywords
server
data block
write
data
authority
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
CN201310475814.6A
Other languages
Chinese (zh)
Other versions
CN104573428A (en
Inventor
璧靛己
赵强
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
FOUNDER BROADBAND NETWORK SERVICE CO LTD
Original Assignee
FOUNDER BROADBAND NETWORK SERVICE Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by FOUNDER BROADBAND NETWORK SERVICE Co Ltd filed Critical FOUNDER BROADBAND NETWORK SERVICE Co Ltd
Priority to CN201310475814.6A priority Critical patent/CN104573428B/en
Publication of CN104573428A publication Critical patent/CN104573428A/en
Application granted granted Critical
Publication of CN104573428B publication Critical patent/CN104573428B/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0602Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/062Securing storage systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0668Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The method and system of the present invention for improving server cluster resource availability, before server request carries out write operation to a certain data, control the server that there is the data block and monopolize write-in authority, other servers outside the server can not carry out write operation to the data block;The server carries out releasing the exclusive write-in authority to the data block after the completion of write-in behaviour to the data block.Therefore, when a certain server needs to carry out write operation to a certain data block, only the server possesses the authority to the data block operation, and the server will store the data block being written over immediately after having performed write operation, it has not been the data block of legacy version so when other servers carry out write operation to the data block again.Using the present invention the above-mentioned coordination system, can effectively avoid write data loss, it is ensured that the integrality and accuracy of data.

Description

A kind of method and system for improving server cluster resource availability
Technical field
The present invention relates to domain name the application of cluster server technical field, specifically one kind is related to multi-node cluster server The method and system of the data syn-chronization between data storage, caching and client and server cluster with server.
Background technology
Server cluster includes one group of separate server, and it shows as triangular web and with single system in a network The pattern of system is managed.Each server in server cluster can be regarded as a node.From client, clothes Just as a large-scale computer system, the application for running client needs thereon takes the Servers-all being engaged in device cluster Business.
In group system, cluster environment configuration file is not what is left concentratedly, but is all deposited on each server It is placed with environment profile copy.In group system, it is master server that can set a certain server, in theory only main clothes Business device can be modified to environment profile, and the environment profile after change can be synchronized into group system in real time In other servers on.But in group system normal operation, client can change ring on any node server Border configuration file copy is simultaneously modified by master server to environment profile, and this change can be synchronized to group system In other node servers on.
When client is modified by different node servers to environment profile, its course of work is as follows:
In moment T01:First client is modified by a certain data block in server A;
In moment T02:When the modification that server A is made is not stored on the disk into server A, the second client passes through Server B is modified to identical data block, because modification of the server A to the data block is not stored into disk also simultaneously And be not synchronized on other servers, therefore server B is still entered when changing the data block on the basis of original data block Capable;
In moment T03:The modification that server A is made store on the disk of server A as newest data block, and The modification is synchronized on other servers by master server;
In moment T04:The modification that server B is made store on the disk of server B as newest data block, and The modification is synchronized on other servers by master server.
Obviously, the modification operation that server B is made to data block can operate the modification that server A is made to the data block Covering, cause the operation that server A is carried out to the data block all to be lost, had influence on the integrality and correctness of data.
The content of the invention
The technical problems to be solved by the invention are due to the read operation of data in server cluster of the prior art Partial write loss of data caused by not coordinated with write operation has influence on the integrality and accuracy of data, so as to carry For a kind of method and system for improving server cluster resource availability.
In order to solve the above technical problems, the present invention is achieved by the following technical solutions:
The present invention provides a kind of method for improving server cluster resource availability, comprises the following steps:
Before server request carries out write operation to a certain data, control the server that there is the data block only Account for write-in authority, other servers outside the server can not carry out write operation to the data block;
The server releases the exclusive write-in authority to the data block after the completion of the write-in behaviour to the data block.
Further, the method for described raising server cluster resource availability, the control server is to institute The processing that stating data block has exclusive write-in authority includes:
It is confirmed whether there are other servers that there is exclusive write-in authority to the data block:
If so, after then server of the request with exclusive write-in authority discharges the exclusive write-in authority of the data block, if The fixed server has exclusive write-in authority to the data block;
If it is not, then directly setting the server has exclusive write-in authority to the data block.
Further, the method for described raising server cluster resource availability, also comprises the following steps:
When the server reads the data block, the data block is obtained in the data snapshot of current time while is recorded Read access time;
Before the data block is not stored on the disk of the server:
If other servers have carried out write operation to the data block, often carry out write-once operation and obtain once institute State the data snapshot of data block while record the write time;
If there is other servers to carry out read operation to the data block, during by the write-in minimum with read time interval Between or read access time corresponding to data snapshot send to request read operation server.
Further, the method for described raising server cluster resource availability, also comprises the following steps:
Minimum interval is set;
When occurring empty data block in data snapshot, then proceed as follows:
Between the read access time or write time and the adjacent data snapshot that judge the data snapshot where the empty data block Read access time or the interval of write time be greater than or equal to the minimum interval:
If so, then the empty data block is retained as valid data;
If it is not, then the empty data block is deleted as invalid data.
Further, the method for described raising server cluster resource availability, also comprises the following steps:
The caching of each server is merged, the data block stored in the caching of each server of acquisition Mirror image data;
When an a certain data block of server acquisition request is operated, if the mirror image data of the data block is stored In the caching of a certain server, then the server directly obtains the mirror image data of the data block from caching and operated.
Further, the method for described raising server cluster resource availability, also comprises the following steps:
Resource TOC list is established, the application clothes that each server can provide are recorded in the Resource TOC list Business;
Control the application service all same that each server can provide;
The service request of client is controlled averagely on each server.
The present invention also provides a kind of system for improving server cluster resource availability, including:
Write operation monitoring module, before server request carries out write operation to a certain data, control the server pair The data block, which has, monopolizes write-in authority, and other servers outside the server can not write to the data block Operation;
It is additionally operable to release after the completion of write-in behaviour of the server to the data block and the exclusive of the data block is write Enter authority.
Further, the system of described raising server cluster resource availability, the write operation monitoring module control The processing that the server has exclusive write-in authority to the data block includes:
It is confirmed whether there are other servers that there is exclusive write-in authority to the data block:
If so, after then server of the request with exclusive write-in authority discharges the exclusive write-in authority of the data block, if The fixed server has exclusive write-in authority to the data block;
If it is not, then directly setting the server has exclusive write-in authority to the data block.
Further, the system of described raising server cluster resource availability, the write operation monitoring module use DML processes are realized.
Further, the system of described raising server cluster resource availability, in addition to:
Read operation monitoring module, when reading the data block for the server, the data block is obtained when current Between data snapshot simultaneously record read access time;
Before the data block is not stored on the disk of the server:
If other servers have carried out write operation to the data block, often carry out write-once operation and obtain once institute State the data snapshot of data block while record the write time;
If there is other servers to carry out read operation to the data block, during by the write-in minimum with read time interval Between or read access time corresponding to data snapshot send to request read operation server.
Further, the system of described raising server cluster resource availability, the read operation monitoring module use CR processes are realized.
Further, the system of described raising server cluster resource availability, in addition to:
Time interval setup module, for setting minimum interval;
First ratio sentences module, during for empty data block occur in the data snapshot, proceeds as follows:
Judge the data snapshot where empty data block read access time or write time and adjacent data snapshot between reading The interval of time or write time is taken to be greater than or equal to the minimum interval:
If so, then empty data block is retained as valid data;
If it is not, then empty data block is deleted as invalid data.
Further, the system of described raising server cluster resource availability, in addition to:
Fusion Module is cached, for being merged to the caching of each server, obtains the caching of each server The mirror image data of middle stored data block;
Second ratio sentences module, for when an a certain data block of server acquisition request is operated, if the data block Mirror image data have stored in the caching of a certain server, then the server directly obtains the mirror of the data block from caching As data are operated.
Further, the system of described raising server cluster resource availability, the caching Fusion Module use LMON processes are realized.
Further, the system of described raising server cluster resource availability, in addition to:
Module is established in Resource TOC list, for establishing Resource TOC list, is recorded in the Resource TOC list each The application service that platform server can provide;
Buffer service module, the application service all same that each server of control can provide;
Queue service module, control the service request of client averagely on each server.
The above-mentioned technical proposal of the present invention has advantages below compared with prior art:
(1)The method and system of the present invention for improving server cluster resource availability, are asked to certain in server Before one data carry out write operation, control the server that there is exclusive write-in authority to the data block, its outside the server He can not carry out write operation by server to the data block.Therefore, when a certain server needs to write a certain data block During operation, only the server possesses the authority to the data block operation, and the server immediately will after having performed write operation The data block being written over is stored, so when other servers carry out write operation to the data block again, It is not the data block of legacy version.Using the present invention the above-mentioned coordination system, can effectively avoid write data loss, it is ensured that The integrality and accuracy of data.
(2)The method and system of the present invention for improving server cluster resource availability, described control server The processing to the data block with exclusive write-in authority includes:It is confirmed whether there are other servers that there is exclusive write to the data block Enter authority:If so, after then server of the request with exclusive write-in authority discharges the exclusive write-in authority of the data block, setting should Server has exclusive write-in authority to the data block;If it is not, then directly setting the server has exclusive write to the data block Enter authority.In such scheme, whether before server is rewritten to data block each time, all first confiring that has other servers There is exclusive write-in authority to the data block.In actual applications, when there is other servers that there is exclusive write-in to the data block During authority, the processing mode that can take has a variety of, such as write operation of other servers to the data block can be waited complete The write operation for carrying out data block after exclusive write-in authority again is automatically terminated after.And directly ask with exclusive write-in authority The exclusive write-in authority that server discharges the data block has higher data-handling efficiency.
(3)The method and system of the present invention for improving server cluster resource availability, also comprise the following steps:When When one server reads a certain data block, the data block is obtained in the data snapshot of current time while records read access time; Before the data block is not stored on the disk of the server:If other servers have carried out write-in behaviour to the data block Make, then often carry out write-once operation and obtain once the data snapshot of the data block while record the write time;If there are other clothes Device be engaged in data block progress read operation, then by corresponding to the write time minimum with read time interval or read access time Data snapshot is sent to corresponding server.Because the rewriting each time to data block can all obtain a new data block, and , it is necessary to the regular hour during data block is stored to the disk of server, if other within this period The data block is read in server request, can by this programme come ensure data block that it reads all be latest edition data Block.
(4)The method and system of the present invention for improving server cluster resource availability, also comprise the following steps:If Put minimum interval;When there is empty data block in the data snapshot, then proceed as follows:Where judging empty data block The interval of read access time or write time between the read access time of data snapshot or write time and adjacent data snapshot is big In or equal to the minimum interval:If so, then empty data block is retained as valid data;If it is not, then empty data block As invalid data, it is deleted.When chance failure occurs in server cluster, it is clear to may result in data block, that is, exists Data block is caused surprisingly to be reset during digital independent and storage.By this unexpected situation and it is strictly using the above method The actual conditions that clear operation has been carried out to the data block are distinguished, and obtained knot will be clearly operated caused by unexpected situation Fruit weeds out, and ensures the accuracy of data.
(5)The method and system of the present invention for improving server cluster resource availability, also comprise the following steps:It is right The caching of each server is merged, and obtains the mirror image data of the data block stored in the caching of each server; When an a certain data block of server acquisition request is operated, if the mirror image data of the data block has stored in a certain clothes It is engaged in the caching of device, then the server directly obtains the mirror image data of the data block from caching and operated.Due to from caching Time required for middle read block much shorter than the time of the read block directly from disk, so if in caching During the saved mirror image data of the data block of acquisition request, the mirror image data of data block is obtained directly from caching, for one The read operation of a little larger data blocks, this programme can increase substantially treatment effeciency.
(6)The method and system of the present invention for improving server cluster resource availability, also comprise the following steps:Build Vertical Resource TOC list, records the application service that each server can provide in the Resource TOC list;Control each The application service all same that platform server can provide;The service request of client is controlled averagely on each server.It is logical The service that the form of Resource TOC list will appreciate that in server cluster the state of each server and can provide is crossed, , can be according to the information in Resource TOC list by the request of client average mark as far as possible when there is client request service It is balanced to be fitted on proof load on each server.
Brief description of the drawings
In order that present disclosure is more likely to be clearly understood, below in conjunction with the accompanying drawings, the present invention is made further detailed Thin explanation, wherein,
Fig. 1 is the method flow diagram that one embodiment of the invention improves server cluster resource availability;
Fig. 2 is that one embodiment of the invention write operation performs method flow diagram;
Fig. 3 is that one embodiment of the invention read operation performs method flow diagram;
Fig. 4 is the schematic diagram of data snapshot chain in one embodiment of the present invention;
Embodiment
Embodiment 1
The present embodiment provides a kind of method for improving server cluster resource availability, as shown in figure 1, including following step Suddenly:
Before server request carries out write operation to a certain data, control the server that there is the data block only Account for write-in authority, other servers outside the server can not carry out write operation to the data block;
The server releases the exclusive write-in authority to the data block after the completion of the write-in behaviour to the data block.
Specific write operation process is as shown in Figure 2:
In moment T1:Server A request carries out write operation to a certain data block, asks have exclusive write to the data block Enter authority;
In moment T2:Server A carries out write operation to the data block.
In moment T3:Server B request carries out write operation to the data block, asks have exclusive write-in to the data block Authority;At this moment, server A has exclusive write-in authority to the data block, therefore server B can not be entered to the data block Row write operation.
In moment T4:After server A completes the write operation to the data block, the data block of latest edition is stored in clothes Also it is not stored in business device A caching on the disk of server A, exclusive write-in power of the now server A release to the data block Limit, and the data block of latest edition is transferred in the caching of server B from the caching of server A, and server B can be to Data block after performing write-once operation has exclusive write-in authority;Because server A has been released to the data The exclusive write-in authority of block, therefore the write operation that server A is carried out to the data block is stored, now server B obtains The exclusive write-in authority of the data block is carried out on the basis of the data block after performing write-once operation, Reserved write operation of the server A to the data block.
In moment T5:Server B carries out write operation to the data block.
Obviously the such scheme in the present embodiment is used, when a certain server needs to carry out write operation to a certain data block When, only the server possesses the authority to the data block operation, and the server immediately will storage after having performed write operation The data block being written over, so when other servers carry out write operation to the data block again, it is not The data block of legacy version.Using the above-mentioned coordination system of the present embodiment, it can effectively avoid writing the loss of data, it is ensured that number According to integrality and accuracy.
Here, it should be noted that in server cluster only have master server can to it is above-mentioned it is exclusive write-in authority carry out with Track management, and master server can select any one server in server cluster.
Each server can be adopted as in the present embodiment sets service processes DLM, DLM process to be mainly responsible for from data block The uniformity copy data of data block is transmitted in the caching of the server of the cache request of the place server data block, without Carry out disk write-in.Every server can at most possess 10 DLM processes, and the actual quantity of DLM processes can be according to service Message communicating flow between device changes, and client can also control the quantity of DLM processes manually using the interface specified. DML processes are all set on each server, but the DLM processes only on master server can be write to the exclusive of data block Enter authority to be managed, the DLM processes on other servers are the backups as the DML processes on master server.It is then above-mentioned The specific implementation of scheme is as follows:
In moment t1:Server A needs to modify to a certain data block, and server A reads the data from local disk Block, still, before reading, he must notify to prepare the intention so done, main service himself to the DLM processes on master server DLM process on behalf server A on device adds an exclusive lock to show that server A has to the data block to this data block Exclusive write-in authority, so as to the lock status of the tracking server A data blocks changed;
In moment t2:The same data block of server B desired modifications, before modification, he must be to the DLM on master server Process notifies oneself to prepare the intention for changing the data block.When the DLM processes on master server receive the request from server B When, the owner i.e. server A that he requires currently to lock discharges the lock, that is, discharges the exclusive write-in authority to the data block, therefore DLM processes ensure that server B obtains the latest edition of this data block, and are transmitted to him and write privilege.
In moment t3:Server B obtains this data block latest edition that being serviced device A was changed, and it is entered Row modification.
At any one time, only a server example possesses the current version of this data block, and only this server can To carrying out write operation in the data block, to be all saved and in need so as to ensure that the whole made to this data block is changed Will when write disk.The state of lock can be tracked for each data block for being read and being changed by the server in cluster, DLM processes are a databases being present in internal memory, and it contains all of the storage in each server in current cluster The relevant information of the lock of data block.
In addition, the present embodiment said write operates, including enter all operations of edlin to data block, such as change wherein Content, the new content of write-in or delete content therein etc..
In the present embodiment, the processing that the control server has exclusive write-in authority to the data block includes:
It is confirmed whether there are other servers that there is exclusive write-in authority to the data block:
If so, after then server of the request with exclusive write-in authority discharges the exclusive write-in authority of the data block, if The fixed server has exclusive write-in authority to the data block.
If it is not, then directly setting the server has exclusive write-in authority to the data block.
In such scheme, whether before server is rewritten to data block each time, all first confiring that has other services Device has exclusive write-in authority to the data block.In actual applications, when there is other servers that there is exclusive write to the data block When entering authority, the processing mode that can take has a variety of, such as can wait write operation of other servers to the data block After the completion of automatically terminate the write operation for carrying out data block after exclusive write-in authority again.And directly ask that there is exclusive write-in authority The server exclusive write-in authority that discharges the data block there is higher data-handling efficiency.
Embodiment 2
The method of raising server cluster resource availability in the present embodiment, on the basis of embodiment 1, in addition to figure Step shown in 3:
When the server reads the data block, the data block is obtained in the data snapshot of current time while is recorded Read access time.
Before the data block is not stored on the disk of the server:
If other servers have carried out write operation to the data block, often carry out write-once operation and obtain once institute State the data snapshot of data block while record the write time.
If there is other servers to carry out read operation to the data block, during by the write-in minimum with read time interval Between or read access time corresponding to data snapshot send to request read operation server.
The purpose of such scheme is to realize the write-in of concurrency and without prejudice to the reading data of concurrency.When reading data generation And data need not be added and write lock, it can be used in the present embodiment and mechanism is read using uniformity on each server in the cluster CR is realized to be kept reading uniformity in each server.Illustrate specific implementation with reference to figure 4:
For a certain data block, server A reads the data block at the Td moment, and the CR processes on master server obtain the number According to block moment Td data snapshot;During being stored to the data block, as long as having other servers to the data block Write operation is carried out, the data that can all obtain the data block that a data snapshot preserves the latest edition after write operation are fast According to as shown in figure 4, add has server to write same data block at tetra- time points of Tx1, Tx2, Tx3 and Tx4 Operation, then the CR processes on master server will obtain the data snapshot on this four time points, and be stored on local cache And it is synchronized in the caching of other servers.And in this course, add and also have other server wants to obtain the data Block, such as in time point Td1 and time point Td2, there is server request to read the data block, if it is directly obtained from disk The data block is taken, must be the data block of reset condition, and the data block has already been through write-once behaviour in time point Td1 Make, in time point Td2, the data block has already been through write operation three times, so if directly reading the initial data in disk Block, the loss of write operation can be caused.And the such scheme in the present embodiment is used, when reading the data block in time point Td1 When, what it was obtained is time point Tx1 data snapshot, and the data block of first time write operation has been saved in the data snapshot Mirror image data.Same reason, when reading the data block in time Td2, its obtain be time point Tx3 data it is fast According to having saved the mirror image data of the data block of write operation three times in the data snapshot.Therefore when the write-in for having concurrent type frog During the read operation of operation and concurrent type frog, it can't interact between both, and using the scheme in the present embodiment, The loss of data will not be caused, ensure that the integrality and accuracy of data.
In the present embodiment, also comprise the following steps:
Minimum interval is set.
When occurring empty data block in data snapshot, then proceed as follows:
Between the read access time or write time and the adjacent data snapshot that judge the data snapshot where the empty data block Read access time or the interval of write time be greater than or equal to the minimum interval:
If so, then the empty data block is retained as valid data.
If it is not, then the empty data block is deleted as invalid data.
During the processing of CR processes, a minimum interval is set, this gap length is specified by client.Sometimes, when When chance failure occurs in server cluster, it is clear to may result in data block, i.e., is led during digital independent and storage Data block is caused surprisingly to be reset.By this unexpected situation and it is strictly that clear operation has been carried out to the data block using the above method Actual conditions distinguished, obtained result will be clearly operated caused by unexpected situation and is weeded out, ensures the accurate of data Property.
Once this data block is eliminated, but the either read access time of data snapshot or write time and adjacent data The interval of read access time or write time between snapshot is more than or during minimum interval, then it is assumed that the data block is effective 's.Because time interval between two data snapshots is less than the situation that default minimum interval formula should not occur, one Caused by chance failure occurs in group system, in this case recognizing occurs when such case then thinks deletion action in denier For data invalid, it is deleted.
Embodiment 3
The method of raising server cluster resource availability in the present embodiment, on the basis of embodiment 1 or embodiment 2 On, also comprise the following steps:
The caching of each server is merged, the data block stored in the caching of each server of acquisition Mirror image data;
When an a certain data block of server acquisition request is operated, if the mirror image data of the data block is stored In the caching of a certain server, then the server directly obtains the mirror image data of the data block from caching and operated.
Such scheme is realized using LMON processes in the present embodiment, LMON processes are respectively provided with each server, still With identical described by front, the LMON processes only in main control server can realize the function of global buffer fusion.From A data block mirror image is obtained in one server buffer, its speed will be far faster than reading the data block from server disk. Therefore, under given conditions, caching fusion between server buffer transmit data block mirror image data, without again from Reading these data blocks in disk, when needing the data block on another server, caching fusion can take accordingly It is engaged in transmitting data block mirror image between device.Such scheme in the present embodiment, for the read operation of some larger data blocks, sheet Scheme can increase substantially treatment effeciency.
Embodiment 4
The present embodiment does following improvement on the basis of embodiment 1 to any embodiment of embodiment 3, and it also includes following step Suddenly:
Resource TOC list is established, the application clothes that each server can provide are recorded in the Resource TOC list Business;
Control the application service all same that each server can provide;
The service request of client is controlled averagely on each server.
Resource TOC list can be established in the present embodiment in shared storage, in synchronized clusters system between Servers-all Topology information, one pooling of resources database of all resource compositions in group system, the service that each server is run Resource TOC list is constituted using adding up.Meanwhile Resource TOC list is supervised using two kinds of system service mechanisms Control:One is buffer service, and another is queue service.Resource in group system is distributed in service according to its weight equality Between device, buffer service coordinates each server of cluster internal so that each server stores same application in internal memory Service or data, and can be shared between each server;Queue service is responsible for buffering please for the client of group system Ask, when there is multiple client request to come then, will can ask be distributed to server in cluster first, when whole servers are all gathered around When having load, user terminal request will enter service queue, and waiting, which has server free to issue the server later, is serviced. The request of client can be evenly distributed to each service as far as possible by this programme according to the information in Resource TOC list Proof load is balanced on device.
In addition, a kind of implementation as the present embodiment, in server cluster system, may be provided for meeting The process DIAG that server diagnosis in cluster needs.Group system need diagnose Servers-all running status, it is necessary to Diagnosis is interacted between multiple servers, therefore, process can be diagnosed with independent design DIAG.Draw with DIAG diagnosis process Entered DIAG frameworks, this framework without interference with or influence the normal operating of group system, if DIAG diagnosis process is dead, that Process monitoring device can restart a new DIAG diagnosis process, to continue its service.DIAG diagnosis process can also monitor The health status of group system, when a basic service process failures, the DIAG diagnosis processes in home server can Diagnose and use to capture group system state and other useful informations, after being provided with, then notify the DIAG on another server Diagnosis process captures similar information, and DIAG diagnosis process is responsible for monitoring local cluster database or group system Service Instance behaviour The active degree of work, and perform any necessary recovery operation when detecting that operation stops.Diagnosis process can use existing The foundation communication connection commonly used in technology, when establishing its communication connection between two servers, if two servers are located In normal operating conditions, then one of server can receive the signal that another server sends over and be responded, If a server, which sends a signal to another server, can not but receive the answer signal that another server returns, Think that it goes wrong, the server that group system can will appear from problem excludes from cluster, until the server recovers again Normally.
Embodiment 5
The present embodiment provides a kind of system for improving server cluster resource availability, as shown in figure 4, it includes:
Write operation monitoring module, before server request carries out write operation to a certain data, control the server pair The data block, which has, monopolizes write-in authority, and other servers outside the server can not write to the data block Operation;The exclusive write-in for being additionally operable to release to the data block after the completion of write-in behaviour of the server to the data block is weighed Limit.
In the such scheme of the present embodiment, when a certain server needs to carry out write operation to a certain data block, only The server possesses the authority to the data block operation, and the server has immediately been changed storage after having performed write operation The data block write, it has not been legacy version so when other servers carry out write operation to the data block again Data block.Using the present invention the above-mentioned coordination system, can effectively avoid write data loss, it is ensured that the integrality of data And accuracy.
In the present embodiment, the write operation monitoring module controls the server that there is exclusive write-in to weigh to the data block The processing of limit includes:
It is confirmed whether there are other servers that there is exclusive write-in authority to the data block:
If so, after then server of the request with exclusive write-in authority discharges the exclusive write-in authority of the data block, if The fixed server has exclusive write-in authority to the data block.
If it is not, then directly setting the server has exclusive write-in authority to the data block.
In the present embodiment, the write operation monitoring module is realized using DML processes.
Each server can be adopted as in the present embodiment sets service processes DLM, DLM process to be mainly responsible for from data block The uniformity copy data of data block is transmitted in the caching of the server of the cache request of the place server data block, without Carry out disk write-in.Every server can at most possess 10 DLM processes, and the actual quantity of DLM processes can be according to service Message communicating flow between device changes, and client can also control the quantity of DLM processes manually using the interface specified. DML processes are all set on each server, but the DLM processes only on master server can be write to the exclusive of data block Enter authority to be managed, the DLM processes on other servers are the backups as the DML processes on master server.It is then above-mentioned The specific implementation of scheme is as follows:
In moment t1:Server A needs to modify to a certain data block, and server A reads the data from local disk Block, still, before reading, he must notify to prepare the intention so done, main service himself to the DLM processes on master server DLM process on behalf server A on device adds an exclusive lock to show that server A has to the data block to this data block Exclusive write-in authority, so as to the lock status of the tracking server A data blocks changed;
In moment t2:The same data block of server B desired modifications, before modification, he must be to the DLM on master server Process notifies oneself to prepare the intention for changing the data block.When the DLM processes on master server receive the request from server B When, the owner i.e. server A that he requires currently to lock discharges the lock, that is, discharges the exclusive write-in authority to the data block, therefore DLM processes ensure that server B obtains the latest edition of this data block, and are transmitted to him and write privilege.
In moment t3:Server B obtains this data block latest edition that being serviced device A was changed, and it is entered Row modification.
At any one time, only a server example possesses the current version of this data block, and only this server can To carrying out write operation in the data block, to be all saved and in need so as to ensure that the whole made to this data block is changed Will when write disk.The state of lock can be tracked for each data block for being read and being changed by the server in cluster, DLM processes are a databases being present in internal memory, and it contains all of the storage in each server in current cluster The relevant information of the lock of data block.
In such scheme, whether before server is rewritten to data block each time, all first confiring that has other services Device has exclusive write-in authority to the data block.In actual applications, when there is other servers that there is exclusive write to the data block When entering authority, the processing mode that can take has a variety of, such as can wait write operation of other servers to the data block After the completion of automatically terminate the write operation for carrying out data block after exclusive write-in authority again.And directly ask that there is exclusive write-in authority The server exclusive write-in authority that discharges the data block there is higher data-handling efficiency.
Embodiment 6
The system of raising server cluster resource availability in the present embodiment, on the basis of embodiment 5, in addition to:
Read operation monitoring module, when reading the data block for the server, the data block is obtained when current Between data snapshot simultaneously record read access time.
Before the data block is not stored on the disk of the server:
If other servers have carried out write operation to the data block, often carry out write-once operation and obtain once institute State the data snapshot of data block while record the write time.
If there is other servers to carry out read operation to the data block, during by the write-in minimum with read time interval Between or read access time corresponding to data snapshot send to request read operation server.
In the present embodiment, the read operation monitoring module is realized using CR processes.
The purpose of such scheme is to realize the write-in of concurrency and without prejudice to the reading data of concurrency.When reading data generation And data need not be added and write lock, it can be used in the present embodiment and mechanism is read using uniformity on each server in the cluster CR is realized to be kept reading uniformity in each server.
Said system in the present embodiment also includes:
Time interval setup module, for setting minimum interval;
First ratio sentences module, during for empty data block occur in the data snapshot, proceeds as follows:
Judge the data snapshot where empty data block read access time or write time and adjacent data snapshot between reading The interval of time or write time is taken to be greater than or equal to the minimum interval:
If so, then empty data block is retained as valid data;
If it is not, then empty data block is deleted as invalid data.
During the processing of CR processes, a minimum interval is set, this gap length is specified by client.Sometimes, when When chance failure occurs in server cluster, it is clear to may result in data block, i.e., is led during digital independent and storage Data block is caused surprisingly to be reset.By this unexpected situation and it is strictly that clear operation has been carried out to the data block using the above method Actual conditions distinguished, obtained result will be clearly operated caused by unexpected situation and is weeded out, ensures the accurate of data Property.
Once this data block is eliminated, but the either read access time of data snapshot or write time and adjacent data The interval of read access time or write time between snapshot is more than or during minimum interval, then it is assumed that the data block is effective 's.Because time interval between two data snapshots is less than the situation that default minimum interval formula should not occur, one Caused by chance failure occurs in group system, in this case recognizing occurs when such case then thinks deletion action in denier For data invalid, it is deleted.
Embodiment 7
The system of raising server cluster resource availability described in the present embodiment, on the basis of embodiment 5 or embodiment 6 On, in addition to:
Fusion Module is cached, for being merged to the caching of each server, obtains the caching of each server The mirror image data of middle stored data block;
Second ratio sentences module, for when an a certain data block of server acquisition request is operated, if the data block Mirror image data have stored in the caching of a certain server, then the server directly obtains the mirror of the data block from caching As data are operated.
Wherein, the caching Fusion Module is realized using LMON processes.
It is respectively provided with LMON processes in each server, but with identical described by front, only in main control server LMON processes can realize the function of global buffer fusion.A data block mirror image is obtained from a server buffer, its Speed will be far faster than reading the data block from server disk.Therefore, under given conditions, caching fusion is in server buffer Between transmit the mirror image data of data block, without again from these data blocks of the reading in disk, when needing another clothes Be engaged in device on data block when, caching fusion data block mirror image can be transmitted between corresponding server.It is upper in the present embodiment Scheme is stated, for the read operation of some larger data blocks, this programme can increase substantially treatment effeciency.
Embodiment 8
The present embodiment on the basis of embodiment 5 to any embodiment of embodiment 7, in addition to:
Module is established in Resource TOC list, for establishing Resource TOC list, is recorded in the Resource TOC list each The application service that platform server can provide;
Buffer service module, the application service all same that each server of control can provide.
Queue service module, control the service request of client averagely on each server.
Resource TOC list can be established in the present embodiment in shared storage, in synchronized clusters system between Servers-all Topology information, one pooling of resources database of all resource compositions in group system, the service that each server is run Resource TOC list is constituted using adding up.Meanwhile Resource TOC list is supervised using two kinds of system service mechanisms Control:One is buffer service, and another is queue service.Resource in group system is distributed in service according to its weight equality Between device, buffer service coordinates each server of cluster internal so that each server stores same application in internal memory Service or data, and can be shared between each server;Queue service is responsible for buffering please for the client of group system Ask, when there is multiple client request to come then, will can ask be distributed to server in cluster first, when whole servers are all gathered around When having load, user terminal request will enter service queue, and waiting, which has server free to issue the server later, is serviced. The request of client can be evenly distributed to each service as far as possible by this programme according to the information in Resource TOC list Proof load is balanced on device.
In addition, a kind of implementation as the present embodiment, in server cluster system, may be provided for meeting The process DIAG that server diagnosis in cluster needs.Group system need diagnose Servers-all running status, it is necessary to Diagnosis is interacted between multiple servers, therefore, process can be diagnosed with independent design DIAG.Draw with DIAG diagnosis process Entered DIAG frameworks, this framework without interference with or influence the normal operating of group system, if DIAG diagnosis process is dead, that Process monitoring device can restart a new DIAG diagnosis process, to continue its service.DIAG diagnosis process can also monitor The health status of group system, when a basic service process failures, the DIAG diagnosis processes in home server can Diagnose and use to capture group system state and other useful informations, after being provided with, then notify the DIAG on another server Diagnosis process captures similar information, and DIAG diagnosis process is responsible for monitoring local cluster database or group system Service Instance behaviour The active degree of work, and perform any necessary recovery operation when detecting that operation stops.Diagnosis process can use existing The foundation communication connection commonly used in technology, when establishing its communication connection between two servers, if two servers are located In normal operating conditions, then one of server can receive the signal that another server sends over and be responded, If a server, which sends a signal to another server, can not but receive the answer signal that another server returns, Think that it goes wrong, the server that group system can will appear from problem excludes from cluster, until the server recovers again Normally.
The present invention is the flow with reference to method according to embodiments of the present invention, equipment (system) and computer program product Figure and/or block diagram describe.It should be understood that can be by every first-class in computer program instructions implementation process figure and/or block diagram Journey and/or the flow in square frame and flow chart and/or block diagram and/or the combination of square frame.These computer programs can be provided The processors of all-purpose computer, special-purpose computer, Embedded Processor or other programmable data processing devices is instructed to produce A raw machine so that produced by the instruction of computer or the computing device of other programmable data processing devices for real The device for the function of being specified in present one flow of flow chart or one square frame of multiple flows and/or block diagram or multiple square frames. These computer program instructions, which may be alternatively stored in, can guide computer or other programmable data processing devices work in a specific way In the computer-readable memory of work so that the instruction being stored in the computer-readable memory, which produces, includes command device Manufacture, the command device are realized in one flow of flow chart or multiple flows and/or one square frame of block diagram or multiple square frames In the function specified.These computer program instructions can be also loaded into computer or other programmable data processing devices, made Obtain and series of operation steps is performed on computer or other programmable devices to produce computer implemented processing, so as to count The instruction performed on calculation machine or other programmable devices is provided for realizing in one flow of flow chart or multiple flows and/or side The step of function of being specified in one square frame of block diagram or multiple square frames.Although preferred embodiments of the present invention have been described, but originally Technical staff in field once knows basic creative concept, then other change can be made to these embodiments and is repaiied Change.Obviously, those skilled in the art can to the present invention carry out it is various change and modification without departing from the present invention spirit and Scope.So, if these modifications and variations of the present invention belong within the scope of the claims in the present invention and its equivalent technologies, Then the present invention is also intended to comprising including these changes and modification.

Claims (11)

  1. A kind of 1. method for improving server cluster resource availability, it is characterised in that comprise the following steps:
    Before server request carries out write operation to a certain data, control the server that there is exclusive write to the data block Enter authority, other servers outside the server can not carry out write operation to the data block;The server is to institute Data block is stated to carry out releasing the exclusive write-in authority to the data block after the completion of write-in behaviour;
    When the server reads the data block, the data block is obtained in the data snapshot of current time while records reading Time;Before the data block is not stored on the disk of the server:If other servers enter to the data block Write operation is gone, then when often carrying out write-once operation and obtaining once the data snapshot of the data block and record write-in simultaneously Between;If there is other servers to carry out read operation to the data block, by the write time minimum with read time interval or Data snapshot corresponding to read access time is sent to the server of request read operation;
    Minimum interval is set;When occurring empty data block in data snapshot, then proceed as follows:Judge the empty data block Read access time between the read access time of the data snapshot at place or write time and adjacent data snapshot or between the write time Every whether being more than or equal to the minimum interval:If so, then the empty data block is retained as valid data;If No, then the empty data block is deleted as invalid data.
  2. 2. the method according to claim 1 for improving server cluster resource availability, it is characterised in that the control institute The processing that stating server has exclusive write-in authority to the data block includes:
    It is confirmed whether there are other servers that there is exclusive write-in authority to the data block:
    If so, after then server of the request with exclusive write-in authority discharges the exclusive write-in authority of the data block, institute is set Stating server has exclusive write-in authority to the data block;
    If it is not, then directly setting the server has exclusive write-in authority to the data block.
  3. 3. the method according to claim 1 for improving server cluster resource availability, it is characterised in that also include as follows Step:
    The caching of each server is merged, obtains the mirror image of the data block stored in the caching of each server Data;
    When an a certain data block of server acquisition request is operated, if the mirror image data of the data block has stored in certain In the caching of one server, then the server directly obtains the mirror image data of the data block from caching and operated.
  4. 4. according to the method for any described raising server cluster resource availabilities of claim 1-3, it is characterised in that also wrap Include following steps:
    Resource TOC list is established, the application service that each server can provide is recorded in the Resource TOC list;
    Control the application service all same that each server can provide;
    The service request of client is controlled averagely on each server.
  5. A kind of 5. system for improving server cluster resource availability, it is characterised in that including:
    Write operation monitoring module, before server request carries out write operation to a certain data, the server is controlled to described Data block, which has, monopolizes write-in authority, and other servers outside the server can not carry out write-in behaviour to the data block Make;It is additionally operable to release the exclusive write-in power to the data block after the completion of the server carries out the data block write-in behaviour Limit;
    Read operation monitoring module, when reading the data block for the server, the data block is obtained in current time Data snapshot records read access time simultaneously;Before the data block is not stored on the disk of the server:If other Server has carried out write operation to the data block, then often carries out the data that write-once operation obtains once the data block Snapshot records the write time simultaneously;, will be between read time if there is other servers to carry out read operation to the data block Sent every the data snapshot corresponding to the write time of minimum or read access time to the server of request read operation;
    Time interval setup module, for setting minimum interval;
    First ratio sentences module, during for empty data block occur in the data snapshot, proceeds as follows:Judge empty data block Read access time between the read access time of the data snapshot at place or write time and adjacent data snapshot or between the write time Every whether being more than or equal to the minimum interval:If so, then empty data block is retained as valid data;If it is not, then Empty data block is deleted as invalid data.
  6. 6. the system according to claim 5 for improving server cluster resource availability, it is characterised in that the write operation Monitoring module controls the server that there is the processing of exclusive write-in authority to include to the data block:
    It is confirmed whether there are other servers that there is exclusive write-in authority to the data block:
    If so, after then server of the request with exclusive write-in authority discharges the exclusive write-in authority of the data block, institute is set Stating server has exclusive write-in authority to the data block;
    If it is not, then directly setting the server has exclusive write-in authority to the data block.
  7. 7. the system according to claim 6 for improving server cluster resource availability, it is characterised in that the write operation Monitoring module is realized using DML processes.
  8. 8. the system according to claim 7 for improving server cluster resource availability, it is characterised in that the read operation Monitoring module is realized using CR processes.
  9. 9. according to the system of any described raising server cluster resource availabilities of claim 6-8, it is characterised in that also wrap Include:
    Fusion Module is cached, for being merged to the caching of each server, obtains institute in the caching of each server The mirror image data of the data block of storage;
    Second ratio sentences module, for when an a certain data block of server acquisition request is operated, if the mirror of the data block Had stored in as data in the caching of a certain server, then the server directly obtains the mirror image number of the data block from caching According to being operated.
  10. 10. the system according to claim 9 for improving server cluster resource availability, it is characterised in that the caching Fusion Module is realized using LMON processes.
  11. 11. the system according to claim 10 for improving server cluster resource availability, it is characterised in that also include:
    Module is established in Resource TOC list, and for establishing Resource TOC list, each clothes are recorded in the Resource TOC list The application service that business device can provide;
    Buffer service module, the application service all same that each server of control can provide;
    Queue service module, control the service request of client averagely on each server.
CN201310475814.6A 2013-10-12 2013-10-12 A kind of method and system for improving server cluster resource availability Expired - Fee Related CN104573428B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201310475814.6A CN104573428B (en) 2013-10-12 2013-10-12 A kind of method and system for improving server cluster resource availability

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201310475814.6A CN104573428B (en) 2013-10-12 2013-10-12 A kind of method and system for improving server cluster resource availability

Publications (2)

Publication Number Publication Date
CN104573428A CN104573428A (en) 2015-04-29
CN104573428B true CN104573428B (en) 2018-02-13

Family

ID=53089471

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201310475814.6A Expired - Fee Related CN104573428B (en) 2013-10-12 2013-10-12 A kind of method and system for improving server cluster resource availability

Country Status (1)

Country Link
CN (1) CN104573428B (en)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107135240B (en) * 2016-02-29 2020-12-15 华为技术有限公司 Method, device and system for distributing commands in distributed system
CN107066877B (en) * 2017-03-31 2019-12-31 武汉票据交易中心有限公司 Permission changing method and system for transaction system
CN108153492B (en) * 2017-12-22 2021-09-14 联想(北京)有限公司 Data processing method and system and electronic equipment
CN109992566A (en) * 2019-04-12 2019-07-09 苏州浪潮智能科技有限公司 A kind of file access method, device, equipment and readable storage medium storing program for executing
CN110442372A (en) * 2019-08-12 2019-11-12 安徽赛福贝特信息技术有限公司 A kind of preservation management system for software development
CN112511359B (en) * 2021-02-04 2021-05-07 上海爱可生信息技术股份有限公司 Method, system and computer readable storage medium for configuration change in service system
KR20230163041A (en) * 2022-05-23 2023-11-30 주식회사 티맥스티베로 Method and device for managing the index performed in active-active database cluster environment

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1786918A (en) * 2004-12-10 2006-06-14 惠普开发有限公司 Distributed lock
CN101090401A (en) * 2007-05-25 2007-12-19 金蝶软件(中国)有限公司 Data buffer store method and system at duster environment
CN101149755A (en) * 2007-10-25 2008-03-26 中国科学院计算技术研究所 A file writing system and method for a distributed file system
US7552122B1 (en) * 2004-06-01 2009-06-23 Sanbolic, Inc. Methods and apparatus facilitating access to storage among multiple computers
CN102088484A (en) * 2010-12-20 2011-06-08 南京中兴新软件有限责任公司 Write lock method of distributed file system and a system thereof
CN102355473A (en) * 2011-06-28 2012-02-15 用友软件股份有限公司 Locking control system in distributed computing environment and method
CN102834822A (en) * 2010-04-12 2012-12-19 微软公司 Fast-full backup of clusters shared by virtual machines

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8046550B2 (en) * 2008-07-14 2011-10-25 Quest Software, Inc. Systems and methods for performing backup operations of virtual machine files

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7552122B1 (en) * 2004-06-01 2009-06-23 Sanbolic, Inc. Methods and apparatus facilitating access to storage among multiple computers
CN1786918A (en) * 2004-12-10 2006-06-14 惠普开发有限公司 Distributed lock
CN101090401A (en) * 2007-05-25 2007-12-19 金蝶软件(中国)有限公司 Data buffer store method and system at duster environment
CN101149755A (en) * 2007-10-25 2008-03-26 中国科学院计算技术研究所 A file writing system and method for a distributed file system
CN102834822A (en) * 2010-04-12 2012-12-19 微软公司 Fast-full backup of clusters shared by virtual machines
CN102088484A (en) * 2010-12-20 2011-06-08 南京中兴新软件有限责任公司 Write lock method of distributed file system and a system thereof
CN102355473A (en) * 2011-06-28 2012-02-15 用友软件股份有限公司 Locking control system in distributed computing environment and method

Also Published As

Publication number Publication date
CN104573428A (en) 2015-04-29

Similar Documents

Publication Publication Date Title
CN104573428B (en) A kind of method and system for improving server cluster resource availability
CN105959151B (en) A kind of Stream Processing system and method for High Availabitity
CN101370030B (en) Resource load stabilization method based on contents duplication
CN1996847B (en) Image and Multimedia Data Communication and Storage System Based on Collaborative Grid
CN111124277B (en) Deep learning data set caching method, system, terminal and storage medium
CN109857445A (en) Storage system and control software layout method
US20230020330A1 (en) Systems and methods for scalable database hosting data of multiple database tenants
CN110990432B (en) A device and method for synchronizing distributed cache clusters across computer rooms
CN106341454A (en) Across-room multiple-active distributed database management system and across-room multiple-active distributed database management method
CN110045912A (en) Data processing method and device
CN103647668A (en) Host group decision system in high availability cluster and switching method for host group decision system
CN102929769B (en) Virtual machine internal-data acquisition method based on agency service
CN110427299A (en) Log processing method, relevant device and the system of micro services system application
CN103634411B (en) A kind of marketing data real time broadcasting system and method with state consistency
CN111324667B (en) Data synchronization method and device, electronic equipment and storage medium
CN112565415A (en) Cross-region resource management system and method based on cloud edge cooperation
CN105592139B (en) A kind of the HA implementation method and device of distributed file system management platform
CN111787055A (en) A Redis-based, transaction-oriented and multi-data center data distribution method and system
CN104077199A (en) Shared disk based high availability cluster isolation method and system
CN109992373A (en) Resource regulating method, approaches to IM and device and task deployment system
CN110099084A (en) A kind of method, system and computer-readable medium guaranteeing storage service availability
CN109783468A (en) Database switching method and system, medium and computer system
CN110348826A (en) Strange land disaster recovery method, system, equipment and readable storage medium storing program for executing mostly living
CN104852814A (en) Intelligent integrated emergency system and emergency method thereof
CN115562676B (en) Triggering method of graph calculation engine

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CP01 Change in the name or title of a patent holder

Address after: 100088 Beijing City, Haidian District Institute of Road No. 15, North Building B block 5 layer

Patentee after: FOUNDER BROADBAND NETWORK SERVICE Co.,Ltd.

Address before: 100088 Beijing City, Haidian District Institute of Road No. 15, North Building B block 5 layer

Patentee before: FOUNDER BROADBAND NETWORK SERVICE Co.,Ltd.

CP01 Change in the name or title of a patent holder
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20180213

Termination date: 20211012

CF01 Termination of patent right due to non-payment of annual fee