Distributed Architecture Design
Foreword
⚫ This lesson describes the Huawei Cloud distributed architecture
solution, introduces Huawei Cloud distributed cloud services, and
reviews five design principles for a distributed architecture.
2
Objectives
⚫ Upon completion of this course, you will have learned:
Huawei Cloud services for designing a distributed architecture.
Five design principles for a distributed architecture.
3
Contents
1. Distributed Architecture Design
2. Huawei Cloud Distributed Architecture Design and Implementation
3. Five Principles of Solution Design
4. Solution Design and Practice
4
Customer Requirements
"As traffic increases, a single server may be not strong enough to handle it. To
address this issue, we need a distributed architecture that helps us make full use
of Huawei Cloud resources and enable them to work together. We also want to
reduce the dependencies between application components, so changes in one
component do not affect the performance of another component."
– Huawei Cloud customer feedback
5
Benefits of Distributed Deployment
A massive resource pool Robust load balancing for Better fault recovery
for massive performance stable performance when there are errors
6
Common Distributed Models
Synchronous
communication and Request distribution Task polling
coordination
7
Why Is a Loosely Coupled Model Required?
This is a distributed model not a
loosely coupled model.
All servers are coupled with each other.
• Each frontend device is connected to all
backend devices.
• Each backend device is connected to all
frontend devices.
Network Application
servers servers
8
Loosely Coupled Models
• The EIP hides the private IP address of the
backend server.
• User requests are not distributed to a specific
server.
Server Server • When a server is faulty or being upgraded, bind an
EIP to another server to handle requests.
IMS
9
Loosely Coupled Models
Topic Email
publisher
• Message pub/sub model
• Publishers post topics as needed.
SMN SMS message • Subscribers only need to subscribe to topics. If
Message topic someone publishes a topic, the subscriber will be
notified.
• Publishers and subscribers do not know each
Invoking a
function other.
Topic subscription
10
Contents
1. Distributed Architecture Design
2. Huawei Cloud Distributed Architecture Design and Implementation
3. Five Principles of Solution Design
4. Solution Design and Practice
11
ELB Overview
Elastic Load Balance (ELB) automatically distributes incoming traffic across multiple backend
servers based on the listening rules you configure. ELB expands the service capabilities of your
applications and improves their availability by eliminating single points of failure (SPOFs).
Database
User ELB
ECS
12
Security Considerations
• You can add a whitelist • You can create a redirect for an HTTP
or blacklist to control listener so that HTTP requests are
access to a listener. redirected to an HTTPS listener.
• You can deploy both the • You can bind an EIP to a load
server certificate and the balancer to protect the
client certificate for mutual private IP address of a
authentication. You can also backend server from being
enable Server Name exposed to the Internet.
Indication (SNI) when you
add an HTTPS listener.
13
Working Principles of ELB
User
• Load balancers can work across AZs.
• Backend servers can also be
deployed across AZs.
ELB
?
• Sticky session can be enabled. But it
still does not solve the problem.
a b c d Server
Backend server group
Stateless servers are required.
14
Health Check
You can configure a health check to prevent requests
from being routed to faulty backend servers.
How does a health check work?
➢ ELB connects to a specific TCP port used by the
backend server.
• A connection is established with the port.
a b c d
➢ ELB accesses a specific HTTP page, generally the
root directory.
• A 2xx code is returned.
You should select a dedicated page for health checks.
15
Health Check Principles
ELB Server port Health check
process
Access: Check path
Continuous
200 returned: update
healthy
Timeout
duration
Other codes
returned: unhealthy
Timeout: unhealthy
Internal health
Interval
check cache with
a timestamp
If the maximum
number of consecutive
failed health checks is
reached, the backend
server is declared
unhealthy.
16
Load Balancing Algorithm
Client Client client Client
172.168.3.1 172.168.4.1
New requests
50% 50%
① ②
Existing Existing
connections: 50 connections: 100
Weight: 100 Weight: 100
Weighted Weighted least Source IP hash Connection ID
round robin connections
17
Application Scenarios
• For an application with heavy traffic, such as a large web portal or mobile app
store, ELB evenly distributes incoming traffic across multiple backend servers,
Heavy-traffic balancing the load while ensuring steady performance.
applications
• For services that require high reliability, ELB routinely performs health checks
on backend servers to monitor their healthy state. If a backend server becomes
faulty, ELB automatically distributes incoming requests to healthy backend
Zero SPOFs servers, ensuring service continuity.
• For services that require high service availability, ELB can distribute traffic
Cross-AZ across AZs. If an AZ becomes faulty, ELB distributes the traffic to backend
load servers in other AZs that are running properly.
balancing
18
ELB distributes traffic across
backend servers within a given
region.
What should we do if we want
load balancing across regions?
Domain Name Service (DNS)
Domain name registration and public DNS resolution
DNS resolution within VPCs
Tens of millions of concurrent connections
Multiple DNS resolution policies
DNS
20
DNS Resolution Policies
www.example.com W www.example.com L www.example.com R T
1.2.3.4 2 4.5.6.7 Carrier A 4.5.6.7 China North China
2.3.4.5 4 5.6.7.8 Carrier B 5.6.7.8 America Brazil
3.4.5.6 5 6.7.8.9 Carrier C 6.7.8.9 Europe UK
Weighted routing ISP-based routing Geographic routing
Support for hybrid deployment, Improved user experience, Nearby access,
more accurate DNS request routing lower latency lower latency
21
Geographic DNS Routing
End users can access the nearest DR design is possible, and
network from a nearby access sites can be switched
point, which reduces latency and through DNS resolution.
improves user experience.
22
Distributed Message Service (DMS)
High throughput and availability
Rapid deployment and open-source compatibility with
Kafka, RocketMQ, and RabbitMQ
DMS 10x million TPS
24/7 O&M assurance, monitoring, automatic fault detection
23
DMS Branches
DMS for RabbitMQ provides
message broadcast, transactional
messages, message routing, dead
letter queues, and priority queues
for flash sales, flow control, and
system decoupling scenarios.
DMS for Kafka is a fully- DMS for RocketMQ supports
managed service that delivers ordered message delivery,
high reliability, concurrency, intentional delivery delay,
and scalability in real-time data message retry, and
transmission, stream processing, transactional, dead letter, and
system decoupling, and traffic session messages.
balancing use cases.
24
Performance (QPS)
• Single-node RabbitMQ: 105 QPS
• Single-node Kafka: 106 QPS
• Single-broker RocketMQ: 105 QPS
(higher than RabbitMQ but lower
than Kafka)
25
Data Reliability
RocketMQ: Three replicas.
More consistent than
RabbitMQ: Multiple replicas Kafka: Multiple replicas RabbitMQ and Kafka (Raft
consistency protocol).
26
Contents
1. Distributed Architecture Design
2. Huawei Cloud Distributed Architecture Design and Implementation
3. Five Principles of Solution Design
4. Solution Design and Practice
27
Five Principles of Solution Design
Architecture
Security Reliability Performance Cost- Maintainability
effectiveness
28
Security
Internet
Access control
Region
DMS can hold hundreds of
Network resilience millions of messages to help
VPC ELB you easily handle peak-
• VPCs are used to control hours traffic that is 10+
network security. times higher than usual.
• Only the ports required by Node management
the engine are opened. DMS
DMS Too many nodes add
Subnet 1 Subnet 2
Incident response complexity to
... ... management. Use as
There are node exception ECS few modes as possible.
ECS ECS ECS
drills.
... ...
Data security
EVS EVS EVS EVS
In addition to data security, data OBS
consistency also needs to be AZ 1 AZ 2
considered when you design a Auditing and logging
distributed architecture. Access logs are stored
in OBS.
29
Reliability
⚫ There is no health check for DNS. You can use the website monitoring
provided by Cloud Eye for automatic switchover. This will be discussed
in the DR section.
⚫ The distributed architecture improves reliability.
However, you still need to check whether there are any single points of failure.
Stateful servers affect reliability.
30
Performance
⚫ Monitoring is key to tracking service performance.
⚫ Load balancing across regions is hard to design and may compromise user experience.
⚫ It is recommended in the preceding course that the performance of a single service should
be improved.
⚫ With a distributed architecture, there are no capacity limits.
⚫ Distributed coordination may reduce some capabilities.
Complex coordination mechanisms are not recommended for a loosely coupled architecture.
⚫ Use as few locks, at least pessimistic locks, as possible.
Do not change a parallel system to a serial system.
⚫ Sharding is recommended for distributed queues.
31
Cost-effectiveness
⚫ A distributed system seems to have more nodes, but that does not mean it is
more expensive.
The key is whether a node is working.
⚫
Cost management on nodes will be discussed later.
Delete load balancers that are no longer used.
$
⚫ Note queue costs.
32
Maintainability
⚫ As nodes are added, automatic maintenance is required to reduce:
Pressure.
Errors.
⚫ O&M will be discussed further in the next course.
⚫ Monitor the health status.
⚫ Practice the recovery plan.
33
Quiz
(Single-choice question) ELB distributes incoming traffic across backend servers
based on the listening rules you specify. Which of the following load balancing
algorithms is used to distribute requests to different backend servers based on
their weights?
A. Weighted round robin
B. Weighted least connections
C. Source IP hash
D. Connection ID
34
Summary
⚫ This lesson described the Huawei Cloud distributed architecture
solution, introduced Huawei Cloud distributed cloud services, and
reviewed the five design principles for a distributed architecture.
35
Acronyms and Abbreviations
⚫ ECS: Elastic Cloud Server
⚫ EVS: Elastic Volume Service
⚫ ELB: Elastic Load Balance
⚫ EIP: Elastic IP
⚫ DMS: Distributed Message Service
⚫ DNS: Domain Name Service
⚫ VPC: Virtual Private Cloud
36
Thank You.
Copyright © 2024 Huawei Technologies Co., Ltd. All Rights Reserved.
The information in this document may contain predictive statements including,
without limitation, statements regarding the future financial and operating results,
future product portfolio, new technology, etc. There are a number of factors that
could cause actual results and developments to differ materially from those
expressed or implied in the predictive statements. Therefore, such information is
provided for reference purpose only and constitutes neither an offer nor an
acceptance. Huawei may change the information at any time without notice.
