CN1968276A - Dynamic optimization method for data transmission between services in service oriented architecture environment - Google Patents

Abstract

The invention belongs to the technical field of data transmission between services under a service-oriented architecture environment, and is characterized in that, a data transmission optimization adapter is added to the server providing the service, and a local service includes data and the address of the destination service For the data transmission request, after the relevant conditions including the size of the transmitted data transmission are estimated, the test packet is sent to the server providing the service where the target service is located or the data server of the network where the corresponding protocol is located for testing, and then The corresponding transmission protocol is specifically selected from the bandwidth and delay information obtained from the response. The method can support protocols that can transmit data on heterogeneous platforms, including Simple Object Access Protocol, File Transfer Protocol, and Grid File Transfer Protocol, and overcomes the inflexibility of a single transfer protocol and low performance under certain conditions. shortcoming.

Description

A Dynamic Optimization Method of Data Transmission Between Services in Service-Oriented Architecture Environment

technical field

The invention is a data transmission method between services under a service-oriented architecture environment, in particular a dynamic self-adaptive optimization data transmission method for Web services, grid services and other services.

Background technique

Service-Oriented Architecture (SOA) is a coarse-grained, loosely coupled software architecture. Under the service-oriented architecture, all elements in the environment are abstracted as services, and services communicate through interfaces defined based on standards. However, a single service cannot complete complex tasks, so we need multiple services to perform a task collaboratively. In this process, how to handle communication and data transmission between services becomes a key issue.

At present, in the service-oriented architecture environment, the data transmission method between services usually only supports a single protocol, such as Simple Object Access Protocol (SOAP), File Transfer Protocol (FTP) or Grid File Transfer Protocol (GridFTP) one of them. The characteristics of these agreements are as follows:

● Simple Object Access Protocol is a lightweight protocol based on Extensible Markup Language (XML), a protocol for heterogeneous software objects to communicate on the Internet. Its introduction enables Web services to meet the requirements of interoperability, universal applicability and lower barriers to entry. Technically speaking, Simple Object Access Protocol is based on Extensible Markup Language Remote Method Call (RPC) protocol.

●The file transfer protocol is a widely used protocol, which enables us to conveniently transfer files on the network, and provides login, directory query, file operation and other session control functions.

●The grid file transfer protocol is fully compatible with the file transfer protocol, and at the same time provides the following features for grid data transfer on the basis of the file transfer protocol:

◆Grid security infrastructure GSI (Grid Security Infrastructure) and Kerberos authentication support. Provide a complete encryption mechanism to ensure security.

◆Third party Control of Data Transfer supports managing large datasets for large distributed communities. It enables third parties to control transfers between storage servers.

◆Parallel Data Transfer. Using multiple TCP streams improves bandwidth compared to using a single TCP stream, and parallel data transmission is supported by FTP command extensions and data channel extensions.

◆Block data transmission. Ability to split data across multiple servers, increasing aggregate bandwidth. GridFTP supports chunked data transfers through extensions defined in the Grid Forum draft.

◆Partial File Transfer. Unlike standard FTP, which requires the application to transfer the entire file, the GridFTP command supports transferring certain areas of the file.

◆Reliable Data Transfer and Restartable Data Transfer. The failover method can handle transient network failures and server failures, while restarting failed deliveries.

◆Automatic negotiation of TCP buffer/window size. Support for obtaining the maximum TCP/IP bandwidth.

These protocols have their own advantages and disadvantages. In terms of advantages: simple object access protocol realizes memory-to-memory through the network, and does not require external storage; file transfer protocol is simple to implement, mature technology, wide adaptability; grid file transfer protocol has high reliability, high security, Third-party transmission and other characteristics.

In terms of disadvantages: Simple Object Access Protocol is based on XML, and it takes a lot of time to generate and parse messages. Especially for a large amount of data, it is not feasible to use Simple Object Access Protocol to transmit data. In addition, the protocol is not suitable for complex data types and objects. The transmission support is not good; the use of the file transfer protocol needs to save the data into a file before transferring, which is time-consuming and laborious for small data; the grid file transfer protocol also needs to transfer the data to the external storage before transferring, and the protocol is more complicated , the security and authentication overhead is large, and it is not very convenient for users to use.

Only by giving full play to the advantages of various protocols and overcoming their respective shortcomings can we better complete the service-oriented architecture system.

Contents of the invention

The purpose of the present invention is to propose a service that is distributed in local area networks, wide area networks, and the Internet, based on services that can target heterogeneous platforms including Simple Object Access Protocol SOAP, File Transfer Protocol FTP, and Grid File Transfer Protocol GridFTP. The protocol for data transmission between them, and the method of dynamic optimization for data transmission through the network.

The present invention is characterized by

Based on the following service-oriented architecture SOA, the following steps are carried out sequentially:

Step 1. Initialize

The operating system, service-oriented architecture SOA infrastructure software, at least one service running on the infrastructure, and data transmission dynamic optimization adapter are running on the server providing the service; the servers providing the service are interconnected by the network ;

The data server is provided with: server support software, the function of the data server is that, in addition to transferring data between the servers providing services, in the data transmission process, it is used to include file transfer protocol FTP, grid file transfer When protocols such as the protocol GridFTP need to carry out data transfer through external storage, data transfer between services is performed, and between data servers, between the data server and the server providing the service, are connected through the network;

The data transmission optimization adapter determines the specific transmission protocol and method according to the data transmission request including the data and the address of the destination service sent by a local service, and determines the specific transmission protocol and method according to the address of the given destination service. transfer data;

Step 2. A service on the local service-providing server initiates a data transmission, and sends the data and the address of the destination service to the local data transmission optimization adapter;

Step 3. The data transmission optimization adapter described in step 2 analyzes the address of the destination service, and searches in the cache whether the data has been transmitted to the server providing the service where the destination service is located recently; if so, obtains the optimized data transmission table that has been cached, Select the transport protocol on the cache, and proceed to step 5; if not, proceed to step 4;

Step 4. According to the judgment of step 3, if not, the data transmission optimization adapter described in step 2 estimates a test packet according to relevant conditions including the size of the data to be transmitted, performance requirements, security requirements, and sends to the destination The data transmission optimization adapter on the service-providing server where the service is located is sent for testing, and then the data obtained through the response judgment is related to the service-providing server where the target service is located or information including network bandwidth and delay between the data server and the data server. Choose a protocol to proceed to the next step;

Step 5. The local data transmission optimization adapter performs the following different operations according to different protocols:

If the SOAP protocol is selected, the data is directly sent to the data transmission optimization adapter in the server providing the service where the destination service is located through the network after SOAP encoding, and the data transmission optimization adapter in the server providing the service where the destination service is located is then transferred to the destination server;

If the FTP protocol is selected, the data is uploaded to the FTP data server through the network, and at the same time the data transmission optimization adapter in the server providing the service where the destination service is notified is passed through the network by the data transmission optimization adapter through the network. Obtain data from the server and send it to the destination service;

If the GridFTP protocol is selected, the data is uploaded to the GridFTP data server through the network, and the data transmission optimization adapter in the server providing the service where the destination service is notified is passed through the network at the same time. The data is obtained from the server and sent to the destination service.

The invention proposes a dynamic optimization method for data transmission between services in an SOA environment, with the purpose of enabling efficient and reliable data transmission between services. The dynamic optimization method provides support for a variety of data transmission protocols, including simple object access protocol, file transfer protocol, grid file transfer protocol, etc., and the method also provides the ability to support data transmission protocols that may appear in the future. It is precisely because the data transmission dynamic optimization method supports a variety of data transmission protocols, and can intelligently select the optimal transmission strategy according to the actual situation, so this method combines the advantages of multiple protocols, such as the implementation of simple object access protocol from memory The advantage of no need for external storage; the advantages of simple implementation of the file transfer protocol, mature technology, and wide adaptability; the advantages of high reliability, high security, and support for third-party transmission of the grid file transfer protocol. At the same time, it avoids the disadvantages of inflexibility and low performance in some cases caused by adopting a single transmission protocol. Since the transmission protocols supported by the present invention are all standard transmission protocols, which have nothing to do with the operating platform, the transmission method is suitable for the data transmission process between services on heterogeneous platforms, which conforms to the idea of SOA.

The present invention can not only utilize the characteristics of relevant data transmission protocols to provide a reliable and efficient data transmission method between services, but also dynamically select an appropriate data transmission protocol and mode for data transmission according to different network conditions and different data volumes. When the amount of data is small, the present invention can select the simple object access protocol to transfer from the memory to the memory, avoiding the overhead of external memory access; when the amount of data is large, the present invention can select the file transfer protocol or grid file transfer protocol for data transmission, avoiding the generation and parsing of extensible markup language-based packages when using simple object access protocol The security authentication certificate mechanism supported by it meets the requirements; when the data transfer speed is required to be very high, the present invention can choose to use the block parallel transfer mechanism supported by the file transfer protocol or the grid file transfer protocol. In a word, the present invention can dynamically select a suitable protocol and method according to service requirements, and transfer data to other services as required.

Description of drawings

Fig. 1 is the hardware structural diagram of the method implemented by the present invention.

Fig. 2 is a structural diagram of the software composition on the server providing the service in the method implemented by the present invention.

Figure 3a is a schematic diagram of the data transmission adapter using the simple object access protocol to directly transmit data during the implementation of the present invention,

Fig. 3b is a schematic diagram of data transmission using the file transfer protocol or grid file transfer protocol by the data optimization transfer adapter during the implementation of the present invention.

Fig. 4 is the flow chart of the implementation method of the present invention.

Figure 5. Data transmission optimization adapter execution flow chart.

Detailed ways

The present invention realizes by following technical method:

The data transmission method described in the present invention includes two aspects of hardware and software. The hardware includes: servers providing services, data servers and networks, and the software includes: operating systems running on servers providing services, SOA system underlying software infrastructure, services running on infrastructure, data transmission optimization adapters; running on data Server-side support software on the server.

The servers that provide the service are connected by a network. In the process of data transmission, if a file transfer protocol or grid file transfer protocol is used, which requires data transfer through external storage, a data server needs to be introduced. The role of the data server is to transfer data between services. The server providing the service and the data server are also connected through the network. Data servers sometimes need to transfer data to each other, so the data servers are also interconnected through the network. The network here covers most of the forms of the network, which can be various forms such as local area network, wide area network, wireless network, Internet, etc., or any combination of them.

The software running on the server that provides the service includes the operating system, the underlying software infrastructure of the SOA system, the services running on the infrastructure, and the data transmission optimization adapter. Note that multiple services can run concurrently on the SOA infrastructure enabling software on a server providing the service. When a certain service in the system needs to transmit data to another service due to user instructions or calculation completion, it only needs to pass the data and the address of the destination service to the data optimization transmission adapter, and the optimization transmission adapter determines the specific transmission agreements and methods. The specific transmission protocol and method are determined by optimizing the transmission adapter according to relevant conditions (such as known data transmission size, performance requirements, security requirements, etc.) between the network bandwidth, network delay and other information, etc.) to determine. Of course, it should be pointed out that the data transmission optimization adapter does not need to perform estimation and trial operations before each data transmission between services, because doing so requires more resources and is not efficient, so it needs to be started once Performing this expensive operation before the data transmission process between unfamiliar services, or performing this operation regularly, will greatly improve performance. After deciding on the specific transmission protocol and method, if you choose a protocol that does not require external storage transfer such as Simple Object Access Protocol for transmission, the local optimized transmission adapter will send the data to the optimized server on the service provider where the destination service is located. On the transmission adapter, the optimized transmission adapter on the service-providing server where the target service is located forwards the data to the target service; The optimized adapter first transmits the data to a certain data server in the system through the selected protocol, and then notifies the optimized transmission adapter on the server providing the service where the target service is located where the data is located (that is, the specific address of the data on the data server), After that, the optimized transmission adapter on the service-providing server where the destination service is located downloads the data from the data server and forwards it to the destination service.

The specific description of the operation mechanism of the data-optimized transmission adapter is as follows: the data-optimized transmission adapter receives a data transmission request from a service on the machine, and then the data-optimized transmission adapter analyzes the address of the destination service, and checks in the cache whether it has recently sent The service provider server where the target service is located has transmitted data, if so, obtains the cached optimized data transmission table, and selects an appropriate data transmission method to transmit data between services. If not, use different data transmission methods to send test data with different data volumes (10KB, 100KB, 1MB, 10MB by default) to the destination data optimization transmission adapter, and wait for the destination data optimization transmission adapter after sending The receipt confirmation of each is calculated, and their respective time intervals are calculated to obtain an optimized data transmission table. Then the local data optimization transmission adapter selects a transmission method most suitable for this data transmission by judging the amount of data to be sent by the local service and testing the data transmission time, and performs data transmission.

Figure 1 shows a dynamic optimization method for data transmission between services. Among them, the service provider server (1) is the basic component of the hardware of the SOA system, on which the operating system, SOA infrastructure software and the basic component element of the SOA system—services—run. Service providing servers (1) are interconnected by a network (3). The data server (2) is also a hardware component of the SOA system, and they are also interconnected by the network (3). The function of the data server (2) is to provide data storage and data transfer of the SOA system. In the process of data transmission between services, according to the actual situation, the server providing the service dynamically selects a protocol and method for data transmission to the server where the other service is located. The possible methods are: the server providing the service (1) through the network ( 3), using the simple object access protocol to directly send data to the server (1) that provides the service at the location of other services; the server (1) that provides the service transmits the data through the network (3) using the file transfer protocol or grid file transfer protocol Go to one or several data servers (2), and then notify other service-providing servers (1) through Simple Object Access Protocol how to obtain data (including data address, user name, password and other information required for login, and even access The required certificate), after the server (1) providing the service obtains the relevant information, it obtains data from the corresponding data server (2) through the network (3) using the file transfer protocol or grid file transfer protocol.

Fig. 2 shows the composition and structure diagram of the software running on the server (1) providing services in Fig. 1, the software running on the server (1) providing services includes operating system, SOA underlying support software, multiple services (4) and a data-optimized transfer adapter (5). Service (4) is the basic component of SOA system software, and its release, packaging, invocation, communication and other processes all comply with a series of standard protocol specifications. The data optimization transmission adapter (5) is a relatively important part of the present invention, and its introduction makes the dynamic and efficient data transmission between services transparent. In other words, for services that require data transmission services and data reception, there is no need to care about the protocol and usage method used for specific data transmission. In order to realize the above functions, the data optimized transmission adapter (5) needs to support standard protocols used in data transmission, such as simple object access protocol, file transfer protocol, grid file transfer protocol and data transfer protocols that may be produced in the future. The protocol supported by the data optimization transmission adapter (5) is extensible. Because of this, for the new general data transmission protocol that may appear in the future, as long as the support for this protocol is added to the optimization transmission adapter, it can be used in the SOA environment. It is very convenient to use the new protocol in the process of data transmission between services under the hood. Specifically, the working principle of the data optimization transmission adapter (5) and the rules for selecting the data transmission protocol to be followed are variable, that is to say, the user can set the rules by himself, and the following is just an example, that is, the following rules are set as The execution rules of the data transmission adapter, in fact, according to the needs of the actual situation, we can formulate different rules and strategies for the data transmission adapter: the data optimization transmission adapter (5) is connected to a service (4) of the machine Data transmission request, the data optimization transmission adapter (5) analyzes the address of the destination service (4), checks in the cache whether the data has been transmitted to the service provider server (1) where the destination service (4) is located recently, and if so, obtains the Optimize the cached data transfer table, select the appropriate data transfer method for data transfer between services. If not, then use different data transmission methods to send test data (10KB, 100KB, 1MB, 10MB) with different data volumes to the data optimization transmission adapter (5) of the destination, and wait for the destination data after sending Optimize the reception confirmation of the transmission adapter (5), calculate their respective time intervals, and obtain an optimized data transmission table. Then the local data optimization transmission adapter (5) selects a transmission method most suitable for this data transmission by judging the size of the data volume to be sent by the local service (4) and the test data transmission time, and performs data transmission.

Fig. 3a is a schematic diagram of direct data transmission by the data optimization transmission adapter (5) in the server (1) providing the service using Simple Object Access Protocol. In this case, the data optimization transmission adapter (5) receives the data sent by the service (1), and the data to be sent has the characteristics of small data volume and no need for backup, so it is suitable to use a simple object access protocol from memory to memory Make a direct transfer. Then the data optimized transmission adapter (5) is connected to the server (1) providing the service through the network (3), and the data is transmitted by simple object access protocol on another data optimized transmission adapter (5) thereon, and the The data optimization transmission adapter (5) parses and delivers to the target service (1).

Fig. 3b is a schematic diagram of data transmission by the data optimization transmission adapter (5) in the server (1) providing the service using the file transfer protocol or the grid file transfer protocol. The difference between this situation and that shown in Figure 3a is: the data optimized transmission adapter (5) receives the data sent by the service (1), and the data is characterized by a large amount of data, or there are other difficulties in direct sending. At this time, the data optimization transmission adapter (5) connects to one or more data servers (2) through the network (3) using the file transfer protocol or grid file transfer protocol and uploads the data files. After the transmission is completed, send a message to the server (1) that provides the service, and then the data optimization transmission adapter (5) in the server (1) that provides the service will generally connect to the corresponding one or more data servers (2) and download the data files. Once complete, pass the data to the target service (1).

The above is the basic description of the technical concept of the present invention. Any equivalent transformation performed according to the technical method of the present invention, especially the change of strategy in the part of the data transmission adapter, shall fall within the scope of protection of the present invention.

The following is a practical example: the service a1 on the server a that provides the service needs to transmit data to the service b1 on the server b that provides the service, and the data volume is 20M bytes. However, there was no communication between a and b before, so the data transmission optimization adapter on a received a request from a1 and found no special requirements, so two data transmission protocols, FTP and SOAP, were used to transmit the test between a and b Data, the data size is 10KB, 100KB, 1MB, 10MB respectively, and the result is: (unit: millisecond) 10KB 100KB 1MB 10MB FTP 46 85 324 3257 SOAP 13 68 1620 54810

Note that using the SOAP protocol, the transmission and parsing time does not increase linearly with the increase in the amount of data. This is due to the SOAP message encoding mechanism based on XML, and we will not describe the specific problems in detail here.

Then the data transmission optimization adapter gets the information: to perform data transmission between a and b, when the data volume is less than 100KB, SOAP should be selected for data transmission; when the data volume is greater than 100KB, FTP should be selected for data transmission, and the information is stored into the buffer table. The amount of data to be transferred between a1 and b1 is 20MB, so the data transfer optimization adapter selects the FTP method for data transfer. It uploads the data to a data server in the system, and then notifies the data transmission optimization adapter on the target server b. After receiving the message, the data transmission optimization adapter on the target server b retrieves the data from the data server and passes it to b1, and the operation ends .

Claims (5)

1, the dynamic optimization method of data transmission between services under the service-oriented architecture environment, it is characterized in that, this method carries out following steps successively based on following service-oriented architecture SOA: Step 1. Initialize The operating system, service-oriented architecture SOA infrastructure software, at least one service running on the infrastructure, and data transmission dynamic optimization adapter are running on the server providing the service; the servers providing the service are interconnected by the network ; The data server is provided with: server support software, the function of the data server is that, in addition to transferring data between the servers providing services, in the data transmission process, it is used to include file transfer protocol FTP, grid file transfer When protocols such as the protocol GridFTP need to carry out data transfer through external storage, data transfer between services is performed, and between data servers, between the data server and the server providing the service, are connected through the network; The data transmission optimization adapter determines the specific transmission protocol and method according to the data transmission request including the data and the address of the destination service sent by a local service, and determines the specific transmission protocol and method according to the address of the given destination service. transfer data; Step 2. A service on the local service-providing server initiates a data transmission, and sends the data and the address of the destination service to the local data transmission optimization adapter; Step 3. The data transmission optimization adapter described in step 2 analyzes the address of the destination service, and searches in the cache whether the data has been transmitted to the server providing the service where the destination service is located recently; if so, obtains the optimized data transmission table that has been cached, Select the transport protocol on the cache, and proceed to step 5; if not, proceed to step 4; Step 4. According to the judgment of step 3, if not, the data transmission optimization adapter described in step 2 estimates a test packet according to relevant conditions including the size of the data to be transmitted, performance requirements, security requirements, and sends to the destination The data transmission optimization adapter on the service-providing server where the service is located is sent for testing, and then the data obtained through the response judgment is related to the service-providing server where the target service is located or information including network bandwidth and delay between the data server and the data server. Choose a protocol to proceed to the next step; Step 5. The local data transmission optimization adapter performs the following different operations according to different protocols: If the SOAP protocol is selected, the data is directly sent to the data transmission optimization adapter in the server providing the service where the destination service is located through the network after SOAP encoding, and the data transmission optimization adapter in the server providing the service where the destination service is located is then transferred to the destination server; If the FTP protocol is selected, the data is uploaded to the FTP data server through the network, and at the same time the data transmission optimization adapter in the server providing the service where the destination service is notified is passed through the network by the data transmission optimization adapter through the network. Obtain data from the server and send it to the destination service; If the GridFTP protocol is selected, the data is uploaded to the GridFTP data server through the network, and the data transmission optimization adapter in the server providing the service where the destination service is notified is passed through the network at the same time. The data is obtained from the server and sent to the destination service. 2. The method for dynamically optimizing data transmission between services under a service-oriented architecture environment according to claim 1, wherein said network includes various network forms including local area network, wide area network and Internet. 3. The method for dynamically optimizing data transmission between services in a service-oriented architecture environment according to claim 1, wherein the data transmission optimization adapter performs network bandwidth optimization before starting an unfamiliar data transmission process between services. , delay, etc., or perform this operation at regular intervals. 4. According to the method for dynamically optimizing data transmission between services under the service-oriented architecture environment of claim 1, all supported protocols can perform data transmission for heterogeneous platforms, including those currently used in the field of web services and Grid Simple object access protocol SOAP, file transfer protocol FTP, and grid file transfer protocol GridFTP for inter-service communication; in addition, the ability to support future inter-service communication protocols is reserved. 5. The method for dynamically optimizing data transmission between services under a service-oriented architecture environment according to claim 1, characterized in that, when the local service proposes a security requirement when transmitting data, the data transmission optimization adapter, Select the GridFTP protocol for data transfer.

Images