8000 Improvements to etcd set up (#9088) · ArduinoBot/website@a2c651f · GitHub
[go: up one dir, main page]
Skip to content

Commit a2c651f

Browse files
chuckhak8s-ci-robot
chuckha
authored and
k8s-ci-robot
committed
Improvements to etcd set up (kubernetes#9088)
Signed-off-by: Chuck Ha <ha.chuck@gmail.com>
1 parent 5bfb804 commit a2c651f

File tree

1 file changed

+200
-165
lines changed

1 file changed

+200
-165
lines changed

‎content/en/docs/tasks/administer-cluster/setup-ha-etcd-with-kubeadm.md

Lines changed: 200 additions & 165 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,14 @@
11
---
22
reviewers:
3-
- chuckha
4-
title: Set up a High-Availability Etcd Cluster With Kubeadm
3+
- sig-cluster-lifecycle
4+
title: Set up a Highly Availabile etcd Cluster With kubeadm
55
content_template: templates/task
66
---
77

88
{{% capture overview %}}
99

1010
Kubeadm defaults to running a single member etcd cluster in a static pod managed
11-
by the kubelet on the control plane node. This is not a highly-available setup
11+
by the kubelet on the control plane node. This is not a highly available setup
1212
as the the etcd cluster contains only one member and cannot sustain any members
1313
becoming unavailable. This task walks through the process of creating a highly
1414
available etcd cluster of three members that can be used as an external etcd
@@ -30,174 +30,209 @@ when using kubeadm to set up a kubernetes cluster.
3030

3131
{{% capture steps %}}
3232

33+
## Setting up the cluster
34+
3335
The general approach is to generate all certs on one node and only distribute
34-
the *necessary* files to the other nodes. Note that kubeadm contains all the necessary
35-
crytographic machinery to generate the certificates described below; no other cryptographic tooling
36-
is required for this exercise.
37-
38-
## Create configuration files for kubeadm
39-
40-
Using the template provided below, create one kubeadm configuration file for
41-
each host that will have an etcd member running on it. Update the value of
42-
`CURRENT_HOST` and `NAME` before running the `cat` command.
43-
44-
```
45-
export HOST0=10.0.0.1 # Update HOST0, HOST1, and HOST2 with the IPs or resolvable names of your hosts
46-
export HOST1=10.0.0.2
47-
export HOST2=10.0.0.3
48-
49-
# Create temp directories to store files that will end up on other hosts.
50-
mkdir -p /tmp/${HOST0}/certs /tmp/${HOST1}/certs /tmp/${HOST2}/certs
51-
52-
export CURRENT_HOST="${HOST0}" # Update on each ranging through HOST0, HOST1 and HOST2
53-
export NAME=infra0 # Update to use infra0 for HOST0, infra1 for HOST1 and infra2 for HOST2
54-
55-
cat << EOF > /tmp/${CURRENT_HOST}/kubeadmcfg.yaml
56-
apiVersion: "kubeadm.k8s.io/v1alpha1"
57-
kind: MasterConfiguration
58-
etcd:
59-
serverCertSANs:
60-
- "${CURRENT_HOST}"
61-
peerCertSANs:
62-
- "${CURRENT_HOST}"
63-
extraArgs:
64-
initial-cluster: infra0=https://${HOST0}:2380,infra1=https://${HOST1}:2380,infra2=https://${HOST2}:2380
65-
initial-cluster-state: new
66-
name: ${NAME}
67-
listen-peer-urls: https://${CURRENT_HOST}:2380
68-
listen-client-urls: https://${CURRENT_HOST}:2379
69-
advertise-client-urls: https://${CURRENT_HOST}:2379
70-
initial-advertise-peer-urls: https://${CURRENT_HOST}:2380
71-
EOF
72-
```
73-
74-
## Generate certificates needed for the etcd cluster
75-
76-
### Certificate Authority
77-
78-
If you already have a CA then the only action that is copying the CA's `crt` and
79-
`key` file to `/etc/kubernetes/pki/etcd/ca.crt` and
80-
`/etc/kubernetes/pki/etcd/ca.key`. After those files have been copied, please
81-
skip to the Certificate Swizzling section below.
82-
83-
If you do not already have a CA then run this command on `$HOST0` (where you
84-
generated the configuration files for kubeadm).
85-
86-
```
87-
kubeadm alpha phase certs etcd-ca
88-
```
89-
90-
This creates two files
91-
92-
1. `/etc/kubernetes/pki/etcd/ca.crt`
93-
2. `/etc/kubernetes/pki/etcd/ca.key`
94-
95-
### Create certificates for each member
96-
97-
In this step we create all the certs for each host in our cluster.
98-
99-
```
100-
kubeadm alpha phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml
101-
kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml
102-
kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
103-
# Move the generated certs out of the generated directory
104-
find /etc/kubernetes/pki/etcd -not -name ca.crt -not -name ca.key -type f -exec mv {} /tmp/${HOST2}/certs \;
105-
cp /etc/kubernetes/pki/etcd/ca.crt /tmp/${HOST2}/certs
106-
107-
kubeadm alpha phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml
108-
kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml
109-
kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
110-
# Move the generated certs out of the generated directory
111-
find /etc/kubernetes/pki/etcd -not -name ca.crt -not -name ca.key -type f -exec mv {} /tmp/${HOST1}/certs \;
112-
cp /etc/kubernetes/pki/etcd/ca.crt /tmp/${HOST1}/certs
113-
114-
kubeadm alpha phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml
115-
kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml
116-
kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
117-
# No need to move the certs because they are for HOST0
118-
```
119-
120-
### Copy certs and configs to other hosts
121-
122-
Copy the certs and configs in each tmp directory to the respective hosts and put
123-
the certs owned by root:root in `/etc/kubernetes/pki/etcd/`.
124-
125-
The steps to get these files on `$HOST1` might look like this if you can ssh
126-
between hosts:
127-
128-
```
129-
root@HOST0 $ scp -i /home/ubuntu/.ssh/id_rsa -r /tmp/${HOST1}/* ubuntu@${HOST1}:/home/ubuntu
130-
root@HOST0 $ ssh -i /home/ubuntu/.ssh/id_rsa ubuntu@${HOST1}
131-
ubuntu@HOST1 $ sudo -s
132-
root@HOST1 $ chown -R root:root certs
133-
root@HOST1 $ mv certs/* /etc/kubernetes/pki/etcd/
134-
# Repeat for HOST2
135-
```
136-
137-
### List of all generated certs
138-
139-
This is a list of all the files you have generated and where on which host they
140-
should live.
141-
142-
#### Host 0
143-
144-
1. `/etc/kubernetes/pki/etcd/ca.crt`
145-
1. `/etc/kubernetes/pki/etcd/ca.key`
146-
1. `/etc/kubernetes/pki/etcd/server.crt`
147-
1. `/etc/kubernetes/pki/etcd/server.key`
148-
1. `/etc/kubernetes/pki/etcd/peer.crt`
149-
1. `/etc/kubernetes/pki/etcd/peer.key`
150-
1. `/etc/kubernetes/pki/etcd/healthcheck-client.crt`
151-
1. `/etc/kubernetes/pki/etcd/healthcheck-client.key`
152-
1. `/tmp/${HOST0}/kubeadmcfg.yaml`
153-
154-
#### Host 1
155-
156-
1. `/etc/kubernetes/pki/etcd/ca.crt`
157-
1. `/etc/kubernetes/pki/etcd/server.crt`
158-
1. `/etc/kubernetes/pki/etcd/server.key`
159-
1. `/etc/kubernetes/pki/etcd/peer.crt`
160-
1. `/etc/kubernetes/pki/etcd/peer.key`
161-
1. `/etc/kubernetes/pki/etcd/healthcheck-client.crt`
162-
1. `/etc/kubernetes/pki/etcd/healthcheck-client.key`
163-
1. `/home/ubuntu/kubeadmcfg.yaml`
164-
165-
#### Host 2
166-
167-
1. `/etc/kubernetes/pki/etcd/ca.crt`
168-
1. `/etc/kubernetes/pki/etcd/server.crt`
169-
1. `/etc/kubernetes/pki/etcd/server.key`
170-
1. `/etc/kubernetes/pki/etcd/peer.crt`
171-
1. `/etc/kubernetes/pki/etcd/peer.key`
172-
1. `/etc/kubernetes/pki/etcd/healthcheck-client.crt`
173-
1. `/etc/kubernetes/pki/etcd/healthcheck-client.key`
174-
1. `/home/ubuntu/kubeadmcfg.yaml`
175-
176-
## Manifests
177-
178-
Now that the certs and configs are in place we can create the manifest. On each
179-
host run the `kubeadm` command to generate a static manifest for etcd.
180-
181-
```
182-
root@HOST0 $ kubeadm alpha phase etcd local --config=/tmp/${HOST0}/kubeadmcfg.yaml
183-
root@HOST1 $ kubeadm alpha phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml
184-
root@HOST2 $ kubeadm alpha phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml
185-
```
186-
187-
## Optional: Check the cluster health
188-
189-
```
190-
docker run --rm -it --net host -v /etc/kubernetes:/etc/kubernetes quay.io/coreos/etcd:v3.2.14 etcdctl --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --ca-file /etc/kubernetes/pki/etcd/ca.crt --endpoints https://${HOST0}:2379 cluster-health
191-
...
192-
cluster is healthy
193-
```
36+
the *necessary* files to the other nodes. Note that kubeadm contains all the
37+
necessary crytographic machinery to generate the certificates described below;
38+
no other cryptographic tooling is required for this example.
39+
40+
1. Create configuration files for kubeadm.
41+
42+
Generate one kubeadm configuration file for each host that will have an etcd
43+
member running on it using the following script.
44+
45+
```sh
46+
# Update HOST0, HOST1, and HOST2 with the IPs or resolvable names of your hosts
47+
export HOST0=10.0.0.6
48+
export HOST1=10.0.0.7
49+
export HOST2=10.0.0.8
50+
51+
# Create temp directories to store files that will end up on other hosts.
52+
mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/ /tmp/${HOST2}/
53+
54+
ETCDHOSTS=(${HOST0} ${HOST1} ${HOST2})
55+
NAMES=("infra0" "infra1" "infra2")
56+
57+
for i in "${!ETCDHOSTS[@]}"; do
58+
HOST=${ETCDHOSTS[$i]}
59+
NAME=${NAMES[$i]}
60+
cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
61+
apiVersion: "kubeadm.k8s.io/v1alpha2"
62+
kind: MasterConfiguration
63+
etcd:
64+
localEtcd:
65+
serverCertSANs:
66+
- "${HOST}"
67+
peerCertSANs:
68+
- "${HOST}"
69+
extraArgs:
70+
initial-cluster: infra0=https://${ETCDHOST0}:2380,infra1=https://${ETCDHOST1}:2380,infra2=https://${ETCDHOST2:2380
71+
initial-cluster-state: new
72+
name: ${NAME}
73+
listen-peer-urls: https://${HOST}:2380
74+
listen-client-urls: https://${HOST}:2379
75+
advertise-client-urls: https://${HOST}:2379
76+
initial-advertise-peer-urls: https://${HOST}:2380
77+
EOF
78+
done
79+
```
80+
81+
1. Generate the certificate authority
82+
83+
If you already have a CA then the only action that is copying the CA's `crt` and
84+
`key` file to `/etc/kubernetes/pki/etcd/ca.crt` and
85+
`/etc/kubernetes/pki/etcd/ca.key`. After those files have been copied, please
86+
skip this step.
87+
88+
If you do not already have a CA then run this command on `$HOST0` (where you
89+
generated the configuration files for kubeadm).
90+
91+
```
92+
kubeadm alpha phase certs etcd-ca
93+
```
94+
95+
This creates two files
96+
97+
- `/etc/kubernetes/pki/etcd/ca.crt`
98+
- `/etc/kubernetes/pki/etcd/ca.key`
99+
100+
1. Create certificates for each member
101+
102+
```sh
103+
kubeadm alpha phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml
104+
kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml
105+
kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
106+
kubeadm alpha phase certs apiserver-etcd-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
107+
cp -R /etc/kubernetes/pki /tmp/${HOST2}/
108+
# cleanup non-reusable certificates
109+
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
110+
111+
kubeadm alpha phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml
112+
kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml
113+
kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
114+
kubeadm alpha phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
115+
cp -R /etc/kubernetes/pki /tmp/${HOST2}/
116+
find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete
117+
118+
kubeadm alpha phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml
119+
kubeadm alpha phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml
120+
kubeadm alpha phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
121+
kubeadm alpha phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
122+
# No need to move the certs because they are for HOST0
123+
124+
# clean up certs that should not be copied off this host
125+
find /tmp/${HOST2} -name ca.key -type f -delete
126+
find /tmp/${HOST1} -name ca.key -type f -delete
127+
```
128+
129+
1. Copy certificates and kubeadm configs
130+
131+
The certificates have been generated and now they must be moved to their
132+
respective hosts.
133+
134+
```sh
135+
USER=ubuntu
136+
HOST=${HOST1}
137+
scp -r /tmp/${HOST}/* ${USER}@${HOST}:
138+
ssh ${USER}@${HOST}
139+
USER@HOST $ sudo -Es
140+
root@HOST $ chown -R root:root pki
141+
root@HOST $ mv pki /etc/kubernetes/
142+
```
143+
144+
1. Ensure all expected files exist
145+
146+
The complete list of required files on `$HOST0` is:
147+
148+
```
149+
/tmp/${HOST0}
150+
└── kubeadmcfg.yaml
151+
---
152+
/etc/kubernetes/pki
153+
├── apiserver-etcd-client.crt
154+
├── apiserver-etcd-client.key
155+
└── etcd
156+
├── ca.crt
157+
├── ca.key
158+
├── healthcheck-client.crt
159+
├── healthcheck-client.key
160+
├── peer.crt
161+
├── peer.key
162+
├── server.crt
163+
└── server.key
164+
```
165+
166+
On `$HOST1`:
167+
168+
```
169+
$HOME
170+
└── kubeadmcfg.yaml
171+
---
172+
/etc/kubernetes/pki
173+
├── apiserver-etcd-client.crt
174+
├── apiserver-etcd-client.key
175+
└── etcd
176+
├── ca.crt
177+
├── healthcheck-client.crt
178+
├── healthcheck-client.key
179+
├── peer.crt
180+
├── peer.key
181+
├── server.crt
182+
└── server.key
183+
```
184+
185+
On `$HOST2`
186+
187+
```
188+
$HOME
189+
└── kubeadmcfg.yaml
190+
---
191+
/etc/kubernetes/pki
192+
├── apiserver-etcd-client.crt
193+
├── apiserver-etcd-client.key
194+
└── etcd
195+
├── ca.crt
196+
├── healthcheck-client.crt
197+
├── healthcheck-client.key
198+
├── peer.crt
199+
├── peer.key
200+
├── server.crt
201+
└── server.key
202+
```
203+
204+
1. Create the static pod manifests
205+
206+
Now that the certificates and configs are in place it's time to create the
207+
manifests. On each host run the `kubeadm` command to generate a static manifest
208+
for etcd.
209+
210+
```sh
211+
root@HOST0 $ kubeadm alpha phase etcd local --config=/tmp/${HOST0}/kubeadmcfg.yaml
212+
root@HOST1 $ kubeadm alpha phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml
213+
root@HOST2 $ kubeadm alpha phase etcd local --config=/home/ubuntu/kubeadmcfg.yaml
214+
```
215+
216+
1. Optional: Check the cluster health
217+
218+
```sh
219+
docker run --rm -it \
220+
--net host \
221+
-v /etc/kubernetes:/etc/kubernetes quay.io/coreos/etcd:v3.2.18 etcdctl \
222+
--cert-file /etc/kubernetes/pki/etcd/peer.crt \
223+
--key-file /etc/kubernetes/pki/etcd/peer.key \
224+
--ca-file /etc/kubernetes/pki/etcd/ca.crt \
225+
--endpoints https://${HOST0}:2379 cluster-health
226+
...
227+
cluster is healthy
228+
```
194229
195230
{{% /capture %}}
196231
197232
{{% capture whatsnext %}}
198233
199-
Once your have a working 3 member etcd cluster, you can continue [setting up an
200-
HA control plane using
234+
Once your have a working 3 member etcd cluster, you can continue setting up a
235+
highly available control plane using the [external etcd method with
201236
kubeadm](/docs/setup/independent/high-availability/).
202237
203238
{{% /capture %}}

0 commit comments

Comments
 (0)
0