CN102479187B - GBK character inquiry system based on even-odd check and its implementation - Google Patents

Abstract

The invention discloses a GBK character query system based on a parity check, which includes a character basic query module, a GBK parity check module and a Chinese character boundary verification module; the character basic query module is used for reading GBK encoded strings and query characters The string is used for basic query, find the position that satisfies the query string in the GBK coded string, and record the start position and end position; the GBK parity check module is used to perform code verification on the basic query results obtained by the character basic query module , using a forward-backward bidirectional parity check method; the Chinese character boundary verification module is used to judge whether the start position and the end position are on the boundary of GBK double-byte Chinese characters according to the parity value check result of the GBK parity check module. In addition, the invention also discloses the realization method of the system. The invention can search and determine the boundaries of Chinese characters, thereby successfully solving the problem of query mismatch easily caused by the intersection of high and low byte codes of Chinese characters.

Description

GBK Character Query System Based on Parity Check and Its Realization Method

technical field

The present invention relates to a Chinese character query system, in particular to a parity-based GBK character query system; moreover, the invention also relates to a method for realizing the parity-based GBK character query system.

Background technique

1. Chinese character boundary problem of GBK encoding

GBK encoding is one of the most common methods for representing Chinese characters in computers. Its full name is "Chinese Character Internal Code Extension Specification" (GBK), the English name is Chinese Internal Code Specification, formulated by the National Information Technology Standardization Technical Committee of the People's Republic of China on December 1, 1995, the Standardization Department of the State Bureau of Technical Supervision and the Ministry of Electronics Industry Jointly with the Quality Supervision Department on December 15, 1995, in the form of Technical Supervision Letter No. 1995 229, it was determined as a technical specification guidance document, released and implemented. GBK encoding is an internal code extension specification based on the GB2312-80 standard, using a double-byte encoding scheme, and its encoding range is from 8140 to FEFE (excluding xx7F), with a total of 23,940 code points and a total of 21,003 Chinese characters, fully compatible The GB2312-80 standard supports all Chinese, Japanese and Korean Chinese characters in the international standard ISO/IEC10646-1 and the national standard GB13000-1, and includes all Chinese characters in the BIG5 encoding. The high byte range of GBK encoding is 0x81-0xFE, and the low byte range is 0x40-7E and 0x80-0xFE.

GBK Chinese character encoding may encounter problems when determining the boundaries of Chinese characters, mainly in:

1. In the GBK encoding, part of the encoding overlaps with the ASC-II encoding, which leads to the problem of false hits in ASC character search;

2. Double-byte encoding method, without flag bits, it is difficult to determine the position of the start byte and end byte of a single Chinese character, resulting in the inability to determine the boundary of Chinese characters when querying a single Chinese character.

Therefore, a self-matching system of GBK encoding is needed, which can search and determine the boundaries of Chinese characters, so as to successfully solve the above-mentioned query mismatching problem.

2. The problem caused by the unclear boundary of GBK Chinese characters

In the GBK encoding system, Chinese characters adopt a double-byte encoding method, that is, each Chinese character is represented by 2 bytes. For example, the sentence "Jin Huaicheng Water Culture Collection" is a string in the eyes of a computer, expressed in hexadecimal as: "BD F9 BB B3 89 40 CB AE CE C4 BB AF CE C4 BC AF".

Due to the lack of flags in the GBK code, it is difficult for the computer to judge the boundaries of Chinese characters when judging the string. The problem caused by this is that when searching for characters, due to the uncertain boundaries of Chinese characters, it leads to the problem of character matching misplacement.

For example, in the above character string, the GBK encoding of the Chinese character "堾" is [89,40]. The code 40 also represents the ASC symbol "@". Therefore, when the symbol '@' is used, it will be found that this symbol appears in the character string "Jin Huaicheng Water Culture Collection", which is inconsistent with the actual situation.

In addition to the problem of ASC character lookup, there will also be problems when looking up Chinese characters. This is because the boundary of GBK Chinese character encoding is not clear, which will also lead to the problem of Chinese character misplacement.

For example, the Chinese character string "Health Education for College Students" corresponds to the GBK code "B4 F3 D1 A7 C9FA BD A1 BF B5 BD CC D3 FD". If you search for the Chinese character "笱", you will find that it exists. Because the encoding of "笱" is [F3, D1], which happens to be the combination of the low byte F3 of the Chinese character "big" and the high byte D1 of the Chinese character "xue", which leads to this error.

Contents of the invention

The technical problem to be solved by the present invention is to provide a GBK character query system based on a parity check, which can search and determine the boundaries of Chinese characters, thereby successfully solving the query mismatch problem easily caused by the intersection of high and low byte codes of Chinese characters. For this reason, the present invention also provides the realization method of the GBK character query system based on the parity check.

In order to solve the above-mentioned technical problems, the present invention provides a GBK character query system based on parity check, which system includes three modules: character basic query module, GBK parity check module and Chinese character boundary verification module;

The character basic query module is used to perform basic query on the read GBK coded string and query string, find the position in the GBK coded string that satisfies the occurrence of the query string, and record its start position and end position;

The GBK parity check module is used to perform encoding verification on the basic query result obtained by the character basic query module. First, the bytes are detected forward sequentially along the starting position to judge the parity value of the step counter count, and then along the end position to Then detect the bytes in order to judge the parity value of the step counter count;

The Chinese character boundary verification module is used to judge whether the start position and the end position are on the boundary of GBK double-byte Chinese characters according to the parity value check result of the GBK parity check module.

The query mode of the character basic query module is to scan the GBK coded string from left to right, find the position that meets the query string, and record the starting position as headpos_k and the ending position as tailpos_k, where the number k represents the current The result matched by the kth basic query in the GBK-encoded string.

In the GBK parity check module, the bytes are sequentially detected forward along the starting position to determine the parity value of the step counter count. Specifically, the following method is used: first, the step counter count is initially set to 0; then, forward along the headpos_k , detect byte T_i successively, wherein i=headpos_k-1, headpos_k-2,...,0; Then, calculate whether T_i&0x80 is 0, and carry out processing logic according to the numerical value of T_i&0x80 operation; Finally judge the parity value of step counter count, If count is an odd number, record head_valid=0; if count is even, record head_valid=1, where the symbol & represents a bitwise Boolean AND operation, and head_valid represents the Boolean value of the starting position of the query string.

In the GBK parity check module, the bytes are sequentially detected backward along the termination position to determine the parity value of the step counter count. Specifically, the following method is used: first, the step counter count is initially set to 0; then, backward along tailpos_k, Detect byte T_i sequentially, where i=tailpos_k+1, tailpos_k+2,...; then, calculate whether T_i&0x80 is 0, and perform processing logic according to the value of T_i&0x80 operation; finally judge the parity value of the step counter count, if count is If it is an odd number, record tail_valid=0; if the count is even, record tail_valid=1, where the symbol & represents a bitwise Boolean AND operation, and tail_valid represents the Boolean value at the end position of the query string.

The verification method of described Chinese character boundary verification module is specifically: if head_valid=1 and tail_valid=1, then the matching result of current GBK character is valid, and the matching information of this query gained: headpos_k, tailpos_k is recorded; If head_valid=0 or tail_valid=0, the matching result of the current GBK character is invalid, ignore the matching result this time, and set k=k+1, continue to return to the GBK parity check module, and re-do the verification of the next query position until k is the character basis The last matching result of the query module ends the query and outputs the query result.

In addition, the present invention also provides a method for realizing a GBK character query system based on a parity check, including the following steps:

(1) Read the string encoded by GBK, import it into a given string, and mark it with text_str;

(2) Read the query string, marked as query_str;

(3) Perform basic query on query_str and text_str by scanning text_str from left to right to find the position where query_str appears, and record the starting position as headpos_k and the ending position as tailpos_k, where the number k represents the position in the current string text_str The result matched by the kth basic query in ;

(4) Perform encoding verification after obtaining the basic query results headpos_k and tailpos_k in step (3), specifically: Step A: Detect byte T_i sequentially forward along headpos_k to determine the parity value of the step counter count, where i =headpos_k-1, headpos_k-2,...,0; Step B: detect the byte T_i sequentially backward along tailpos_k to judge the parity value of the step counter count, wherein i=tailpos_k+1, tailpos_k+2,...;

(5) judge whether headpos_i and tailpos_i are on the boundary of GBK double-byte Chinese characters according to the parity check result of step (4);

(6) Output GBK character query result.

In step (4), the step A is specifically: 1) initially setting the step counter count to 0; 2) moving forward along headpos_k, and sequentially detecting bytes T_i, wherein, i=headpos1-1, headpos1-2 ,...,0; 3) Calculate whether T_i&0x80 is 0; 4) Perform the following processing logic according to the calculated value of T_i&0x80: if T_i&0x80 is equal to 0, record head_valid=1, and terminate the subsequent process of this step, and directly enter step B; if T_i&0x80 If it is not equal to 0, then set the counter count=count+1, and then perform the following two processing sub-logic according to the value of T_i: if i=0, then terminate this step, and output the value of count; if i>0, then set T_i Set it as T_i-1, and re-execute the operation of the above step A, that is, continue to detect forward along the character string; 5) judge the parity value of the step counter count, if count is an odd number, then record head_valid=0; if count is If it is an even number, record head_valid=1, where the symbol & represents a bitwise Boolean AND operation, and head_valid represents the Boolean value of the starting position of the query string.

In step (4), the step B is specifically: 1) initially setting the step counter count to 0; 2) detecting the byte T_i sequentially along tailpos_k, wherein i=tailpos+1, tailpos1+2, ...; 3) Calculate whether T_i&0x80 is 0; 4) Perform the following processing logic according to the value of T_i&0x80 operation: if T_i&0x80 is equal to 0, then record tail_valid=1, and terminate the follow-up process of this step; if T_i&0x80 is not equal to 0, then count the counter =count+1, then according to the value of T_i, carry out the following two processing sub-logic: if the length of i is the same as that of the current character string text_str, then terminate this step, and output the value of count; if i<the length of the current character string text_str, Then set T_i as T_i+1, and re-execute the operation of the above step B, that is, continue to detect backward along the character string; 5) judge the parity value of the step counter count, if count is an odd number, then record tail_valid=0; If count is an even number, record tail_valid=1, where the symbol & represents a bitwise Boolean AND operation, and tail_valid represents the Boolean value of the end position of the query string.

Step (5) is specifically: if head_valid=1 and tail_valid=1, then the matching result of the current GBK character is valid, and the matching information obtained by the query: headpos_k, tailpos_k is recorded; if head_valid=0 or tail_valid=0 , then the matching result of the current GBK character is invalid, ignore the matching result this time, and set k=k+1, continue to return to step (4), and perform the verification work of the next query position again.

Step (6) is specifically: if k is the last matching result of step (3), then end all the verification work of this query, and output all headpos_k and tailpos_k character query results that are successfully matched; if k is not the step ( 3) For the last matching result, set k=k+1, continue to return to step (4), and perform the verification work of the next query position again.

The beneficial effect of the present invention is: the present invention is aimed at the character string of GBK code, when performing character query, because the Chinese character high and low byte code intersection easily causes the wrong matching problem, proposes a kind of self-matching GBK character based on parity check Query system, the system is composed of three parts: basic character query module, GBK parity check module, and Chinese character boundary verification module. Using the characteristics of double-byte encoding of Chinese characters, the method of forward and backward bidirectional parity check is used to match characters. The results are verified one by one. The practice of the system has proved that it can handle the problem of GBK character query very well. It can not only eliminate the result of wrong matching of high and low bytes, but also retain the result of legal matching.

Description of drawings

Fig. 1 is a schematic diagram of the module structure and flow chart of the system of the present invention.

detailed description

As shown in Fig. 1, the present invention proposes a self-matching GBK character query system based on parity check, which includes three modules: character basic query module, GBK parity check module, and Chinese character boundary verification module. Through the mutual connection of these three modules, the character matching misplacement problem encountered during character string search in GBK encoding can be well handled.

Give an embodiment below to specify the implementation method of this GBK character query system based on parity, which includes the following steps (seeing Fig. 1):

1. The system first reads the GBK-encoded string (that is, reads the GBK matching text) and imports it into a given string, marked with text_str. And take byte (Byte) as the unit, name each byte of the string in turn: T_0, T_1, T_2, T_3, T_4...

2. Read the query string, marked here as query_str, also marked in bytes (Byte), named: Q_0, Q_1, Q_2, Q_3, Q_4,….

3. Character basic query module:

Perform basic query on query_str and text_str by scanning text_str sequentially from left to right to find the position where query_str appears, and record the starting position as headpos_k and the ending position as tailpos_k (where the number k represents the kth position in the current string text_str The results matched by the second basic query, that is, headpos_1 and tailpos_1 for the first time, headpos_2 and tailpos_2 for the second time, and so on), that is, the strings from T_headpos_k to T_tailpo_k in text_str are exactly the same as query_str.

4. GBK parity module:

After the module obtains the basic query results headpos_k and tailpos_k obtained by the character basic query module, it performs code verification. The specific processing flow is:

Step A:

1) Set the step counter count to 0 initially;

2) forward along headpos_k, sequentially detect byte T_i (i=headpos1-1, headpos1-2,...,0);

3) Calculate whether T_i&0x80 is 0 (the reason why the operation parameter is 0x80 here is that, according to the rules of GBK encoding, the encoding range of the high byte is greater than or equal to 0x80, which is a sign that the high byte of legal Chinese characters begins). Wherein, the symbol & represents a bitwise Boolean "AND" operation, and the operation (And) rule is: 0&0=0, 0&1=0, 1&0=0, 1&1=1; the bitwise AND operation is to expand the byte in binary, And calculate each bit, for example: if T_i=0x9E, the binary representation is 10011110; and the binary expansion of 0x80 is 10000000. At this time: T_i&0x80＝10011110&10000000＝10000000;

4) According to the value calculated by T_i&0x80, perform the following processing logic:

- If it is equal to 0, record head_valid=1; and terminate the follow-up process of this step, and directly enter the following step B;

- If it is not equal to 0, set the counter count=count+1, and then perform the following two processing sub-logic according to the value of T_i:

If i=0 (meaning that the head of the character string has been reached), the step is terminated and the count value is output;

If i>0, then set T_i as T_i-1, and re-execute the operation of the above step A, that is, continue to detect forward along the string;

5) After the above processing in step A is completed normally, judge the parity value of the step counter count:

- if count is odd, record head_valid=0;

- If count is even, record head_valid=1.

Step B:

1) Set the step counter count to 0 initially;

2) Backward along tailpos_k, sequentially detect bytes T_i (i=tailpos+1, tailpos1+2,...);

3) Calculate whether T_i&0x80 is 0 (the method is the same as step A);

4) According to the value calculated by T_i&0x80, perform the following processing logic:

- If it is equal to 0, record tail_valid=1, and terminate the subsequent process of this step;

- If it is not equal to 0, set the counter count=count+1, and then perform the following two processing sub-logic according to the value of T_i:

If i=text_str.length (representing that the length of i and current character string text_str is the same, promptly has reached the end of character string), then terminate this step, and output count numerical value;

If i<text_str.length, set T_i to T_i+1, and re-execute the operation of the above steps, that is, continue to detect backwards along the string;

5) After the above processing in step B is completed normally, judge the parity value of the step counter count:

- if count is odd, record tail_valid=0;

- If count is even, record tail_valid=1.

5. Chinese character boundary verification module:

Input the result of the GBK parity check module into the Chinese character boundary verification module for processing. According to the values of head_valid and tail_valid, it is judged whether headpos_i and tailpos_i are on the boundary of GBK double-byte Chinese characters. The processing logic is:

- If head_valid=1 and tail_valid=1, it is considered that the matching result of the current GBK character is valid, and the matching information obtained by this query: headpos_k, tailpos_k is recorded;

- If head_valid=0 or tail_valid=0, consider the matching result of the current GBK character to be invalid, ignore the matching result this time, and set k=k+1, continue to return to the GBK parity check module, and re-do the verification of the next query position Work;

6. Output GBK character query results

After the above judgment is completed, the verification of the subsequent query position of the string is performed in a loop, and the processing logic is as follows:

- If k is already the last matching result of the character basic query module, all verification work of this query is ended, and all headpos_k and tailpos_k information (that is, GBK character query results) that are successfully matched are output.

- If k is not the last matching result of the character basic query module, then set k=k+1, continue to return to the GBK parity check module, and re-do the verification of the next query position.

A GBK character query system based on parity check proposed by the present invention, the system proposes a set of query for GBK-coded character strings. The system consists of three parts: character basic query module, GBK parity check module, and Chinese character boundary verification module. Using the characteristics of Chinese character double-byte encoding, the method of forward and backward bidirectional parity check is used to check the character matching results. Verified one by one, the practice of the system has proved that it can handle the GBK character query very well, it can not only eliminate the results of high and low byte mismatches, but also retain the legal matching results.

Claims (10)

1. a GBK character query system based on parity check, is characterized in that, the system comprises three modules: character basic query module, GBK parity check module and Chinese character boundary verification module; The character basic query module is used to perform basic query on the read GBK coded string and query string, find the position in the GBK coded string that satisfies the occurrence of the query string, and record its start position and end position; The GBK parity check module is used to perform encoding verification on the basic query result obtained by the character basic query module. First, the bytes are detected forward sequentially along the starting position to judge the parity value of the step counter count, and then along the end position to Then detect the bytes in order to judge the parity value of the step counter count; The Chinese character boundary verification module is used to judge whether the start position and the end position are on the boundary of GBK double-byte Chinese characters according to the parity value check result of the GBK parity check module. 2. the GBK character query system based on parity as claimed in claim 1, is characterized in that, the query mode of described character basic query module is to scan GBK coded character string successively from left to right, finds and meets query character string to occur The starting position is headpos_k and the ending position is tailpos_k, where the number k represents the result matched by the kth basic query in the current GBK encoded string. 3. the GBK character query system based on parity as claimed in claim 2, is characterized in that, in described GBK parity module, detect byte forward successively along starting position to judge the parity value of step counter count Specifically, the following method is adopted: first, the step counter count is initially set to 0; then, along the headpos_k forward, sequentially detect the byte T_i, where i=headpos_k-1, headpos_k-2,...,0; then, calculate whether T_i&0x80 It is 0, and the processing logic is performed according to the numerical value of T_i&0x80 operation; finally judge the parity value of the step counter count, if the count is odd, record head_valid=0; if the count is even, record head_valid=1, where the symbol & means press The bit performs Boolean AND operation, and head_valid represents the Boolean value of the starting position of the query string. 4. the GBK character query system based on parity as claimed in claim 2, is characterized in that, in the described GBK parity module, detect byte successively along end position to judge the specific parity value of step length counter count The following method is adopted: first, the step counter count is initially set to 0; then, along the tailpos_k backward, sequentially detect the byte T_i, wherein i=tailpos_k+1, tailpos_k+2,...; then, calculate whether T_i&0x80 is 0, And perform processing logic according to the numerical value of T_i&0x80 operation; finally judge the parity value of the step counter count, if count is an odd number, then record tail_valid=0; AND operation, tail_valid Boolean value indicating where the query string terminates. 5. the GBK character query system based on parity as claimed in claim 3 or 4, is characterized in that, the verification method of described Chinese character boundary verification module is specifically: if head_valid=1 and tail_valid=1, then current GBK character The matching result of the current GBK character is valid, and the matching information obtained from this query: headpos_k, tailpos_k is recorded; if head_valid=0 or tail_valid=0, the matching result of the current GBK character is invalid, the matching result of this time is ignored, and k=k+ 1. Continue to return to the GBK parity check module, and re-do the verification of the next query position until k is the last matching result of the character basic query module before ending the query and outputting the query result. 6. an implementation method based on the GBK character query system of parity check, it is characterized in that, comprising the steps: (1) Read the string encoded by GBK, import it into a given string, and mark it with text_str; (2) Read the query string, marked as query_str; (3) Perform basic query on query_str and text_str by scanning text_str from left to right to find the position where query_str appears, and record the starting position as headpos_k and the ending position as tailpos_k, where the number k represents the position in the current string text_str The result matched by the kth basic query in ; (4) Perform encoding verification after obtaining the basic query results headpos_k and tailpos_k in step (3), specifically: Step A: Detect byte T_i sequentially forward along headpos_k to determine the parity value of the step counter count, where i =headpos_k-1, headpos_k-2,...,0; Step B: detect the byte T_i sequentially backward along tailpos_k to judge the parity value of the step counter count, wherein i=tailpos_k+1, tailpos_k+2,...; (5) judge whether headpos_i and tailpos_i are on the boundary of GBK double-byte Chinese characters according to the parity check result of step (4); (6) Output GBK character query result. 7. the implementation method of the GBK character query system based on parity as claimed in claim 6, is characterized in that, in step (4), described step A is specifically: 1) step length counter count is initially set to 0; 2) Move forward along headpos_k, and detect byte T_i sequentially, wherein, i=headpos1-1, headpos1-2,...,0; 3) Calculate whether T_i&0x80 is 0; 4) Perform the following processing logic according to the calculated value of T_i&0x80 : If T_i&0x80 is equal to 0, then record head_valid=1, and terminate the subsequent process of this step, and directly enter step B; if T_i&0x80 is not equal to 0, then set the counter count=count+1, and then perform the following two processes according to the value of T_i Sub-logic: If i=0, then terminate this step and output the count value; if i>0, then set T_i to T_i-1, and re-execute the operation of the above step A, that is, continue to detect forward along the string ; 5) judge the parity value of the step counter count, if count is an odd number, then record head_valid=0; if count is an even number, then record head_valid=1, wherein the symbol & represents a bitwise Boolean AND operation, and head_valid represents a query string A boolean value for the starting position. 8. the realization method based on the GBK character query system of parity as claimed in claim 6 is characterized in that, in step (4), described step B is specifically: 1) the step length counter count is initially set to 0; 2) Backwards along tailpos_k, sequentially detect byte T_i, wherein i=tailpos+1, tailpos1+2,...; 3) Calculate whether T_i&0x80 is 0; 4) Carry out the following processing logic according to the numerical value of T_i&0x80 operation: if T_i&0x80 Equal to 0, then record tail_valid=1, and terminate the subsequent process of this step; if T_i&0x80 is not equal to 0, then set the counter count=count+1, and then perform the following two processing sub-logic according to the value of T_i: if i and the current character If the length of the string text_str is the same, this step is terminated and the value of count is output; if i<the length of the current string text_str, then T_i is set to T_i+1, and the operation of the above step B is re-executed, that is, continue along this string 5) judge the parity value of the step counter count, if count is an odd number, then record tail_valid=0; if count is an even number, then record tail_valid=1, wherein the symbol & represents a bitwise Boolean AND operation, and tail_valid represents A boolean value for where the query string ends. 9. the implementation method of the GBK character query system based on parity as claimed in claim 6, is characterized in that, step (5) is specifically: if head_valid=1 and tail_valid=1, then the matching result of current GBK character Valid, record the matching information obtained from this query: headpos_k, tailpos_k; if head_valid=0 or tail_valid=0, the matching result of the current GBK character is invalid, ignore this matching result, and set k=k+1, Continue to return to step (4), and perform the verification work of the next query position again. 10. the realization method based on the GBK character query system of parity as claimed in claim 6 is characterized in that, step (6) is specifically: if k has been the last matching result of step (3), then end this For all the verification work of the second query, all headpos_k and tailpos_k character query results that are successfully matched are output; if k is not the last matching result of step (3), set k=k+1, continue to return to step (4), and proceed again Validation job for next query location.