KR101656077B1 - System and method for time base partitioning using implicit time column value - Google Patents

Abstract

The present invention relates to a time-based partitioning system and method using an implicit time column value, wherein a time-based partitioning system according to an embodiment is a partition- And a record recorder for recording the time column value in the input record to the generated partition.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to time-based partitioning systems and methods using implicit time-

The present invention relates to the technical idea of partitioning data using implicit time column values to quickly and accurately retrieve records of a desired time range in an entire data set.

The time series data is generated in chronological order, and all records have a time column value. Therefore, time series data can be searched by using the time value set in each record as a conditional statement.

In the case of retrieving data by setting a time value as a conditional, dividing and storing the data file with respect to the time value is very effective as a method for reducing the retrieval time.

If these functions are implemented by using B + Tree index and partitioning function in existing DBMS, it is troublesome to manually generate partitions when storing data in real time. Also, as the number of data using the index increases, it becomes difficult to process data input such as sensor data due to an increase in the processing cost due to the update of the index.

Nevertheless, we use data partitioning and B + Tree indexing.

When B + Tree index is used, the index is used for the time column value of the record, which is very good, but the execution time for holding the index at the time of inputting and deleting data is long.

On the other hand, when the partitioning function is used, a range partition is created to use a search in a partition based on a time value. However, there is no problem in partitioning past data by using range partition. However, when data is inputted in real time, there is a possibility that a problem occurs because additional time partition must be created every time the use time progresses Fairly high.

Korean Patent Registration No. 10-0865481 Korean Patent Registration No. 10-0989904

The present invention automatically creates partitions for recording time series data so that high-speed record input is possible even if a large amount of time series data is inserted.

The present invention enables a high-speed search operation on time series data by assigning a conditional phrase for searching for each partition even when a record is inputted at a high speed.

The present invention enables high-speed input / output through time-based partitioning, thereby enabling real-time input / output of records.

The time-based partitioning system according to an embodiment of the present invention includes a partition generation unit for generating a partition when a record input in a time-series manner satisfies a preset condition, and a partition generation unit for reflecting a time column value in the input record, And a record record section for recording the record.

The time-based partitioning system according to an embodiment further includes a partition local index manager for updating the time column value of the recorded records to a partition local index when records corresponding to the predetermined condition are recorded in the created partition do.

In a case where a record corresponding to a preset number is recorded in the current partition, the partition creating unit according to an embodiment requests the record recording unit to stop record recording on the current partition, and creates a new partition.

The record recorder according to an embodiment records, in the new partition, records from which the record is stopped when the new partition is created.

In the case where records of a predetermined capacity are recorded in the current partition, the partition creation unit requests the record recording unit to stop recording the record of the current partition, and creates a new partition.

The partition generator according to an embodiment creates the new partition so that the last partition local index generated corresponding to the current partition and the first partition local index generated corresponding to the new partition are serially connected in a time series manner.

The partition generating unit may generate a partition index reflecting an interval separated by a first record input time recorded in the created partition and a last record input time recorded in the created partition, .

The record recorder according to an embodiment sets time information at the time of recording the inputted record as the time column value.

The time-based partitioning method according to an exemplary embodiment of the present invention includes a step of creating a partition when a record input in a time-series manner satisfies a predetermined condition, a step of reflecting a time column value in the input record, and recording the time- And updating the time column value of the recorded records to a partition local index when records corresponding to the predetermined condition are recorded in the created partition.

The step of creating the partition according to an embodiment of the present invention includes the steps of: when a record corresponding to a predetermined number is recorded in the current partition, requesting the record recording unit to stop recording the record of the current partition; .

The step of writing to the created partition according to an embodiment includes recording the records in which the record recording has been stopped to the new partition when the new partition is created.

The step of creating the partitions according to an embodiment of the present invention includes the steps of: requesting the record recording unit to stop record recording of the current partition when records of predetermined capacity are recorded in the current partition; .

The creating of the partition according to an exemplary embodiment may include creating the new partition so that the last partition local index generated corresponding to the current partition and the first partition local index generated corresponding to the new partition are serially connected in a time- .

The step of generating the partition according to an exemplary embodiment may include generating a partition index reflecting an interval separated by a first record input time recorded in the created partition and a last record input time recorded in the created partition, And matching the partition to be created.

The recording step may include setting time information at the time of recording the input record as the time column value.

A program according to an exemplary embodiment of the present invention includes a command set for generating a partition when a record input in a time series is satisfied with a preset condition, a command set for recording a time column value in the input record to record in the created partition, And a set of instructions for updating a time column value of the recorded records to a partition local index when records corresponding to the predetermined condition are recorded in the created partition.

According to the embodiments, partitions for recording time-series data are automatically generated and high-speed records can be input even when a large amount of time series data is inserted.

According to the present invention, even when a record is inputted at a high speed, automatic sorting of input time is performed for each record, thereby enabling high-speed search operation on time series data.

According to the present invention, since high-speed input / output is possible through time-based partitioning, real-time input / output of records is possible.

1 is a diagram illustrating a time-based partitioning system in accordance with one embodiment.
2 is a diagram illustrating an overall structure using a time-based partitioning system.
3 is a diagram illustrating a time-based partitioning method according to one embodiment.
Figure 4 is an illustration of an embodiment of 'reading' data in a partition created through a time-based partitioning system according to an embodiment.
5 is a diagram illustrating an embodiment of 'writing' data in a partition created through a time-based partitioning system according to an embodiment.

It is to be understood that the specific structural or functional descriptions of embodiments of the present invention disclosed herein are presented for the purpose of describing embodiments only in accordance with the concepts of the present invention, May be embodied in various forms and are not limited to the embodiments described herein.

Embodiments in accordance with the concepts of the present invention are capable of various modifications and may take various forms, so that the embodiments are illustrated in the drawings and described in detail herein. It should be understood, however, that there is no intention to limit the embodiments according to the concepts of the present invention to the particular forms disclosed, but includes modifications, equivalents, or alternatives falling within the spirit and scope of the present invention.

The terms first, second, or the like may be used to describe various elements, but the elements should not be limited by the terms. The terms may be named for the purpose of distinguishing one element from another, for example without departing from the scope of the right according to the concept of the present invention, the first element being referred to as the second element, Similarly, the second component may also be referred to as the first component.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between. Expressions that describe the relationship between components, for example, "between" and "immediately" or "directly adjacent to" should be interpreted as well.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises ", or" having ", and the like, are used to specify one or more of the features, numbers, steps, operations, elements, But do not preclude the presence or addition of steps, operations, elements, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the meaning of the context in the relevant art and, unless explicitly defined herein, are to be interpreted as ideal or overly formal Do not.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these embodiments. Like reference symbols in the drawings denote like elements.

1 is a diagram illustrating a time-based partitioning system 100 in accordance with one embodiment.

The time-based partitioning system 100 according to an exemplary embodiment of the present invention automatically generates partitions for recording time-series data, thereby enabling high-speed record input even when a large amount of time series data is inserted.

In addition, even when a record is inputted at high speed, a conditional phrase for searching for each partition is given, so that a high-speed search operation on time series data is possible.

In addition, time-based partitioning allows high-speed I / O, enabling real-time input / output of records.

To this end, the time-based partitioning system 100 according to an embodiment includes a partition generation unit 110 and a record recording unit 120. [

First, the partition generation unit 110 generates a partition when a record input in a time series corresponds to a predetermined condition.

For example, the partition generating unit 110 may generate a new partition when the number of records input to the current partition exceeds a predetermined number. At this time, a certain number of records including an exceeded record may be recorded in a newly created partition thereafter.

Therefore, the number of records recorded in the generated partitions may all be the same.

As another example, the partition generation unit 110 may set different numbers of records to be recorded in each of the partitions. For example, by changing the number of records according to a predetermined rule every time a new partition is created, the number of records to be recorded for each partition can be set differently.

More specifically, each time a partition is newly created, the number of records can be increased by a certain number. In this case, by checking the number of records recorded in an initially created partition and the number of records recorded in a specific partition, it is possible to determine the number of partitions corresponding to a specific partition.

Meanwhile, the partition generating unit 110 may generate a new partition when the total capacity of records input to the current partition exceeds a predetermined capacity. At this time, the partition generation unit 110 may include a record exceeding the capacity, and may record a certain number of records in the newly created partition thereafter.

Therefore, the total capacity of the records recorded in the generated partitions may all be the same.

As another example, the partition generation unit 110 may set the total capacity of records recorded in each of the partitions to be different from each other. For example, when the partition is newly created, the total capacity of the records to be recorded can be set differently by increasing or decreasing the total capacity of the allowed records according to a predetermined rule.

The number of records and the total capacity set in this manner can be variously changed to values that can increase the system efficiency in consideration of the system characteristics by the experimental values.

Next, the record recording unit 120 may write the time column value into the generated partition by reflecting the time column value.

The time column value includes time information at the moment when the record is input.

That is, the record recorder 120 may calculate the time information on the input time of the record as a time column value, and record the time information on the partition created together with the record.

The partition generation unit 110 monitors the record of the record, and creates the partition when the input record meets predetermined conditions. Accordingly, the partition generating unit 110 can monitor the input record if it does not meet predetermined conditions. In this case, the partition generating unit 110 requests the record recording unit 120 to stop recording the record.

For example, if a record corresponding to a preset number is recorded in the current partition, the record recording unit 120 may be requested to stop recording the record of the current partition from the partition generation unit 110. [

In this case, the record recorder 120 can stop recording the record in the current partition and resume recording the record in the newly created partition.

That is, when a new partition is created, the record recording unit 120 can record the records in which the record recording is stopped in the new partition. Similarly, the record recording unit 120 may continue recording records in the partition until receiving a recording stop request from the partition generating unit 110. [

As another example, if a record corresponding to a preset capacity is recorded in the current partition, the record recording unit 120 may be requested to stop recording the record of the current partition from the partition generation unit 110. [ In addition, when a new partition is created, the record recording unit 120 can record the records in which the record recording has been stopped to the new partition. Similarly, the record recording unit 120 may continue recording records in the partition until receiving a recording stop request from the partition generating unit 110. [

Thus, the time series data can be recorded in the partitions created by setting the number or the total capacity.

As a result, the records recorded in each partition can implement time series data, and each of the records can be recorded in a time-series manner in the partition since the time column values are reflected before being recorded in each partition. That is, even if the time series data is divided and recorded for each partition, it is possible to recover the entire time series data.

The time-based partitioning system according to an embodiment may further include a partition local index management unit 130. [

The partition local index management unit 130 updates the time column value of the recorded records to a partition local index when records corresponding to predetermined conditions are recorded in the created partition.

That is, when the record of the record in the current partition is completed and a new partition is created, the partition local index management unit 130 updates the time column of each record in the current partition in which the record recording is completed to the partition local index.

The partition local index management unit 130 assigns a partition local index to each record so that the partition can be identified in the partition for efficient search and management in each partition.

The partition local index may be an interval reflecting the input time of the record, and may include an identifier (e.g., numerals, alphabets, special characters, and combinations thereof, etc.) .

The partition generating unit 110 may generate a new partition so that the last partition local index generated corresponding to the current partition and the first partition local index generated corresponding to the new partition are serially connected in a time series manner.

For example, a partitioned local index may be a value that is uniquely identified for each partition, and each partition local index may also be created in a chronologically sequential manner to indicate that each partition is chronologically sequential. As a specific example, if the partition local index assigned to the last record in the first partition is 100, the partition local index assigned to the first record in the second partition may be 101.

The partition generation unit 110 according to an embodiment generates a partition index reflecting an interval distinguished by the first record input time recorded in the created partition and the last record input time recorded in the created partition, Can be matched. Each partition also needs to be identified, and the partition index can be given as information that can uniquely identify each partition. For example, a partition index may be an interval that reflects the input time of a record, and may include identifiers (e.g., numbers, alphabets, special characters, and combinations thereof) .

2 is a diagram illustrating an overall system 200 architecture using a time-based partitioning system.

Referring to FIG. 2, when new data 201 is input, the new data 201 can be recorded in the current partition until a predetermined condition is satisfied.

For example, if the predetermined condition is the number of records, the number of records included in the new data 201 may be recorded in the current partition 202 until the number of records exceeds a predetermined number.

If the number of records included in the new data 201 exceeds a predetermined number, the current partition 202 is shifted to the old partition in the timeline, and a new partition Can be generated.

Likewise, new data is written to the new partition until it exceeds the preset number.

A partition local index 203 may be assigned to each partition. For example, at a point when a new partition is created after a predetermined number is recorded in a partition, a partition local index may be assigned to records recorded in a partition moved to a past timeline. At this time, each record is matched with the column value reflecting the input time, and it can be updated to the partition local index instead of the column value.

The entire system 200 architecture using a time-based partitioning system allows fast and large data retrieval in real time.

Upon receiving a time-based query 205 from an application, the entire system 200 may preferentially quickly search the partition 204 corresponding to the time-based query 205. Next, a search for a specific record in the corresponding partition 204 or a range of a specific record in the corresponding partition 204 is performed, thereby reducing unnecessary searches and enabling quick search in real time.

The entire system 200 may retrieve (206) the search results from the partition 204.

3 is a diagram illustrating a time-based partitioning method according to one embodiment.

The time-based partitioning method according to an exemplary embodiment of the present invention creates a current partition (step 301) when a predetermined condition is satisfied. Next, the time-based partitioning method writes the time column value into the partition created by reflecting the time column value in the input record (step 302).

The time column value is time information of the moment when each record is input, and is recorded in matching with each record. Thus, time series data can be recovered in a timely manner even if it is divided into records and written to different partitions.

Next, the time-based partitioning method may determine whether records corresponding to preset conditions are recorded in the current partition while monitoring the record record (step 303).

For example, consider the case where a condition is set so that 10,000 records can be written to the current partition. In this case, the time-based partitioning method can determine whether 10,000 records have been recorded in the current partition through step 303. [

On the other hand, it is possible to consider a case where a condition is set so that a record of 1 terabyte can be recorded in the current partition. In this case, the time-based partitioning method can determine whether a record of one terabyte has been recorded in the current partition through step 303. [

If it is determined in step 303 that records corresponding to conditions preset in the current partition are recorded, that is, if the record to be input corresponds to the 10,001st record or a record exceeding 1 terabyte is recorded in the current partition, The time-based partitioning method may create a new partition via step 304.

Next, the time-based partitioning method may update the time column value assigned to each record with the partitioned local index for the recorded partition (step 305). For example, the time-based partitioning method may replace the time column value with the partition local index, and may add the partition local index without leaving the time column value.

That is, step 305 is to create an index of the column value in the partition for the indexed column. The index update operation for the newly inputted record is reflected only in the current partition, and is not changed in the previous partition, Do not.

Figure 4 is an illustration of an embodiment of 'reading' data in a partition created through a time-based partitioning system according to an embodiment.

Referring to FIG. 4, the server receives a query including a time condition from a client application (step 401). For time series data, it is possible to search by using the time value set in each record as a conditional.

Next, the server selects a partition containing the corresponding time condition (step 402). At this time, the server can retrieve the partition index corresponding to the conditional index. For this, the partition index may include an interval reflecting the input time according to the record input. That is, the server can check the interval of the partition index and retrieve the partition index corresponding to the inputted time condition.

The server further searches the records corresponding to the input query using the partition local index (step 403). That is, the server can further refine the query and retrieve specific records within the partition.

Next, the server opens only the partitioned data file, reads the desired record, and transmits it to the application (step 404).

5 is a diagram illustrating an embodiment of 'writing' data in a partition created through a time-based partitioning system according to an embodiment.

The client application transmits data to be input to the server (step 501).

The server adds the current time of the server to the input data to set the value of the _ARRIVAL_TIME column (step 502). The _ARRIVAL_TIME column value may include time information corresponding to the input time of the record, and all input records may be set to the _ARRIVAL_TIME column value.

Next, the server can determine whether the number of input records for the current partition has been exceeded (step 503). That is, the server can check that the maximum number of input records set in the partition has been exceeded before writing data to the last partition.

If the server has exceeded the maximum number of input records as a result of the determination in step 503, the server creates a new data partition and records it in the file (step 504).

On the other hand, if it is determined in step 503 that the maximum number of input records is not exceeded, the server writes the data in the last partition (step 505).

For partition local index management, the administrator can reflect the record in the data file after the last reflected record in the index exists in the index. Also, if the maximum number of records for the local partition index has been exceeded, you can create a new index and enter the data into a new index file.

As a result, according to the present invention, partitions for recording time-series data are automatically generated and high-speed records can be input even if a large amount of time series data is inserted. In addition, even when a record is input at a high speed, by providing a conditional phrase for searching each partition, it is possible to perform a high-speed search operation on time series data and to perform high-speed input / output through time- Real-time input / output is possible.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (10)

CLAIMS What is claimed is: 1. A time-based partitioning system implemented non-temporally by a computing terminal,
A partition generating unit for generating a partition when a record to be input in a time-series manner meets predetermined conditions; And
A recorder for recording the time column value in the input record to the generated partition,
/ RTI >
Wherein the record recorder calculates time information on the input time of the record as a time column value and records the time information on a partition created with the record.
The method according to claim 1,
A partition local index manager for updating a time column value of the recorded records to a partition local index when records corresponding to the predetermined condition are recorded in the created partition,
Wherein the time-based partitioning system further comprises:
The method according to claim 1,
Wherein the partition generating unit comprises:
Requesting the record recording section to stop recording the record for the current partition and generating a new partition when a record corresponding to a preset number is recorded in the current partition.
The method of claim 3,
The record-
And wherein when the new partition is created, the record is recorded in the new partition from the stopped records.
The method according to claim 1,
Wherein the partition generating unit comprises:
Requesting the record recording section to stop recording the record for the current partition and creating a new partition if records of a preset capacity are recorded in the current partition.
The method of claim 3,
Wherein the partition generating unit comprises:
Wherein the new partition is created so that the last partition local index generated corresponding to the current partition and the first partition local index generated corresponding to the new partition are chronologically continuous.
The method according to claim 1,
Wherein the partition generating unit comprises:
Wherein a partition index reflecting an interval separated by a first record input time recorded in the created partition and a last record input time recorded in the created partition is generated and matched to the generated partition.
The method according to claim 1,
The record-
And sets time information at a time of recording the input record as the time column value.
CLAIMS 1. A time-based partitioning method implemented non-temporally by a computing terminal,
Generating a partition when a record to be input in a time-series manner meets predetermined conditions;
Reflecting the time column value in the input record and recording the time column value in the created partition; And
Updating the time column value of the recorded records to a partition local index when records corresponding to the predetermined condition are recorded in the created partition
/ RTI >
Wherein the time column value includes time information at a moment when a record is input.
10. The method of claim 9,
Wherein the step of creating the partition further comprises:
Requesting to stop recording the record for the current partition and creating a new partition if a record corresponding to the preset number is recorded in the current partition.

Images