Kubernetes RBAC 101
Oleg Chunikhin | CTO, Kublr
Introductions
Oleg Chunikhin
CTO, Kublr
• 20 years in software architecture &
development
• Working w/ Kubernetes since its release in 2015
• Software architect behind Kublr—an enterprise
ready container management platform
• Twitter @olgch
Enterprise Kubernetes Needs
Developers SRE/Ops/DevOps/SecOps
• Self-service • Governance
• Compatible • Org multi-tenancy
• Conformant • Single pane of glass
• Configurable • Operations
• Open & Flexible • Monitoring
• Log collection
• Security
• Image management
• Reliability
• Identity management
• Performance
• Portability
@olgch; @kublr
OPERATIONS SECURITY &
GOVERNANCE
Automation Infrastructure RBAC IAM
Ingress Storage Networking Container CI / CD App Mgmt
Registry
Logging Monitoring
Air Gap TLS
Custom Container Runtime Kubernetes
Observability
Clusters Certificate
Audit
Rotation
Usage
Infrastructure
API
Reporting
@olgch; @kublr
Kubernetes Access Control
• Who can do what with which resource
• Authentication
• Authorization
• RBAC
• Use-cases and gotchas
@olgch; @kublr
Who Can Do What in a Cluster
Three groups
Subjects Operations Resources
list
User
get get Pods
create post Nodes
update put ConfigMaps
Group
delete delete Secrets
watch Deployments
patch …
Service account
Connected through access control
@olgch; @kublr
Kubernetes API Request Attributes
• Authentication Verbs
• User – the user string
• Common API resource request :
• Group – the list of group names get, list, watch,
• Extra – a map of arbitrary keys create, update, patch,
• API – non-resource or API resource flag delete, deletecollection
• API resource request
• Special API resource request:
• API request verb – lowercased resource verb use (PodSecurityPolicy),
• Namespace – the namespace bind, escalate (Role, ClusterRole),
• API group - The API Group being accessed impersonate (User, Group, SA),
• Resource – the resource ID userextras
• Subresource – the sub-resource
• HTTP request verbs:
• Non-resource request get, head, post, put,
• HTTP request verb – lowercased HTTP method patch, delete
• Request path – non-resource request path.
@olgch; @kublr
Kubernetes API Client, Tools
• Any HTTP client
• curl – good for experiments
• kubectl – Kubernetes CLI
HTTP Request
Client Kubernetes API
• jq – honorable mention – JSON visualization and processing
@olgch; @kublr
Kubernetes API Client, Example
• curl curl -k -v -XGET -H 'Authorization: Bearer ***' \
'https://52.44.121.181:443/api/v1/nodes?limit=50' | jq -C . | less -R
apiVersion: v1
• kubectl kind: Config
clusters:
- name: demo-rbac
kubectl --kubeconfig=kc.yaml get nodes cluster:
certificate-authority-data: ***
export KUBECONFIG="$(pwd)/config.yaml" server: https://52.44.121.181:443
kubectl get nodes users:
- name: demo-rbac-admin-token
kubectl get nodes –-v=9 user:
token: ***
contexts:
- name: demo-rbac
context:
cluster: demo-rbac
user: demo-rbac-admin-token
current-context: demo-rbac
@olgch; @kublr
Authentication: Mechanisms
Mechanism Secret Source Usage
X509 Client Certs CSR generated externally and Enterprise CA / PKI
signed with the cluster CA key
Via Kubernetes API Kubernetes cluster admin
CertificateSigningRequest
Bearer token Bootstrap token Internal use
Node authentication token Internal use
Static token file Insecure
ServiceAccount token Pods, containers, applications, users
OIDC token Users
HTTP Basic auth Static password file Insecure
Auth proxy N/A (trust proxy) Integration
Impersonate N/A (trust account) Integration and administration
Authentication: X509 Client Cert, PKI
Admin k8s-api-server-client-ca.key
Admin (issuer) must has access to
the server CA private key
1. client.csr 2. client.crt
3. HTTPS/TLS handshake with client cert
4. Request
Client Kubernetes API
client.key k8s-api-server-client-ca.key
k8s-api-server-client-ca.crt
--client-ca-file=k8s-api-server-client-ca.key
@olgch; @kublr
Authentication: X509 Client Cert, PKI Example
• User: generate user private key (if not exist)
openssl genrsa -out user1.key 2048
• User: generate user CSR
openssl req -new -key user1.key -out user1.csr -subj "/CN=user1/O=group1/O=group2"
• Admin: sign user client cert
openssl x509 -req -in user1.csr -CA cluster-ca.crt -CAkey cluster-ca.key \
-set_serial 101 -extensions client -days 365 -outform PEM -out user1.crt
• User: use with kubectl via options or kubeconfig
kubectl --client-key=user1.key --client-certificate=user1.crt get nodes
kubectl config set-credentials user1 --client-key user1.key --client-certificate user1.crt --embed-certs
kubectl config set-context user1 --cluster demo-rbac --user user1
kubectl --context=user1 get nodes
kubectl config use-context user1
kubectl config get-contexts
kubectl get nodes
@olgch; @kublr
Authentication: X509 Client Cert, K8S CSR
Admin
3. kubectl create csr ...
4. kubectl certificate approve csr ...
5. kubectl get csr ...
1. client.csr 6. client.crt
7. HTTPS/TLS handshake with client cert
8. Request
Client Kubernetes API
client.key k8s-api-server-client-ca.key
k8s-api-server-client-ca.crt
--client-ca-file=k8s-api-server-client-ca.key
@olgch; @kublr
Authentication: X509 Client Cert, K8S CSR
• User: generate user CSR
openssl req -new -key user2.key -out user2.csr -subj "/CN=user2/O=group1/O=group2"
• Admin: use Kubernetes API server to sign the CSR
kubectl apply -f - <<EOF kubectl certificate approve user2
apiVersion: certificates.k8s.io/v1beta1 kubectl certificate deny user2
kind: CertificateSigningRequest
metadata: kubectl get csr user2 –o jsonpath='{.status.certificate}' | \
name: user2 base64 --decode > user2.crt
spec:
request: $(cat user2.csr | base64 | tr -d '\n')
usages: ['digital signature', 'key encipherment',
'client auth']
EOF
• User: use with kubectl via options or kubeconfig
kubectl --client-key=user2.key --client-certificate=user2.crt get nodes
kubectl config set-credentials user2 --client-key user2.key --client-certificate user2.crt --embed-certs
kubectl config set-context user2 --cluster demo-rbac --user user2
@olgch; @kublr
Authentication: Service Account
1. create service account
2. get service account token
3. Request (token in Authorization header)
Client Kubernetes API
@olgch; @kublr
Authentication: Service Account, Example
• Create service account
kubectl create serviceaccount sa1
• Get service account token
kubectl get –o yaml sa sa1
SA_SECRET="$(kubectl get sa sa1 -o jsonpath='{.secrets[0].name}')"
kubectl get -o yaml secret "${SA_SECRET}"
SA_TOKEN="$(kubectl get secret "${SA_SECRET}" -o jsonpath='{.data.token}' | base64 -d)"
• Send request
kubectl "--token=${SA_TOKEN}" get nodes
kubectl config set-credentials sa1 "--token=${SA_TOKEN}"
kubectl config set-context sa1 --cluster demo-rbac --user sa1
@olgch; @kublr
Authentication: OIDC
OIDC IdP
ken 4.
o Ve
s ht rify
efre to
ke
/r en
s n
u th k
a To
1. 2.
3. Request with the token
Client Kubernetes API
--oidc-client-id=kubernetes
--oidc-groups-claim=user_groups
--oidc-issuer-url=https://idp.example.com
--oidc-username-claim=preferred_username
@olgch; @kublr
Authentication: OIDC, Demo
Kublr uses Keycloak by default
• Multiple “realms”
• OIDC, SAML, LDAP, AD, Kerberos support
• User federation and Identity Broker support
Demo
• “demo-app” realm
• “kubernetes” OIDC client
• “demo-rbac” Kublr cluster with OIDC auth configured for the client
spec:
master:
kublrAgentConfig:
kublr:
kube_api_server_flag:
oidc_client_id: '--oidc-client-id=kubernetes'
oidc_groups_claim: '--oidc-groups-claim=user_groups'
oidc_issuer_url: '--oidc-issuer-url=https://***'
oidc_username_claim: '--oidc-username-claim=preferred_username'
@olgch; @kublr
Authentication: OIDC, Example
Login (visualization)
curl \
-d "grant_type=password" \
-d "scope=openid" \
-d "client_id=kubernetes" \
-d "client_secret=${CLIENT_SECRET}" \
-d "username=da-admin" \
-d "password=${USER_PASSWORD}" \
https://kcp.kublr-demo.com/auth/realms/demo-app/protocol/openid-connect/token | jq .
Login (CLI)
eval "$(curl -d "grant_type=password" -d "scope=openid" -d "client_id=kubernetes" \
-d "client_secret=${CLIENT_SECRET}" -d "username=da-admin" -d "password=${USER_PASSWORD}" \
https://kcp.kublr-demo.com/auth/realms/demo-app/protocol/openid-connect/token | \
jq -r '"REFRESH_TOKEN="+.refresh_token,"TOKEN="+.access_token,"ID_TOKEN="+.id_token')" ; \
echo ; echo "TOKEN=${TOKEN}" ; echo ; echo "ID_TOKEN=${ID_TOKEN}" ; echo ; \
echo "REFRESH_TOKEN=${REFRESH_TOKEN}" ; echo
@olgch; @kublr
Authentication: OIDC, Example
Refresh (visualization)
curl \
-d "grant_type=refresh_token" \
-d "client_id=kubernetes" \
-d "client_secret=${CLIENT_SECRET}" \
-d "refresh_token=${REFRESH_TOKEN}" \
https://kcp.kublr-demo.com/auth/realms/demo-app/protocol/openid-connect/token | jq -r .
Refresh (CLI)
eval "$(curl -d "grant_type=refresh_token" -d "client_id=kubernetes" \
-d "client_secret=${CLIENT_SECRET}" -d "refresh_token=${REFRESH_TOKEN}" \
https://kcp.kublr-demo.com/auth/realms/demo-app/protocol/openid-connect/token | \
jq -r '"REFRESH_TOKEN="+.refresh_token,"TOKEN="+.access_token,"ID_TOKEN="+.id_token')" ; \
echo ; echo "TOKEN=${TOKEN}" ; echo ; echo "ID_TOKEN=${ID_TOKEN}" ; echo ; \
echo "REFRESH_TOKEN=${REFRESH_TOKEN}"
@olgch; @kublr
Authentication: OIDC, Example
Token introspection
curl \
--user "kubernetes:${CLIENT_SECRET}" \
-d "token=${TOKEN}" \
https://kcp.kublr-demo.com/auth/realms/demo-app/protocol/openid-connect/token/introspect | jq .
kubectl configuration
kubectl config set-credentials da-admin \
"--auth-provider=oidc" \
"--auth-provider-arg=idp-issuer-url=https://kcp.kublr-demo.com/auth/realms/demo-app" \
"--auth-provider-arg=client-id=kubernetes" \
"--auth-provider-arg=client-secret=${CLIENT_SECRET}" \
"--auth-provider-arg=refresh-token=${REFRESH_TOKEN}" \
"--auth-provider-arg=id-token=${ID_TOKEN}"
kubectl config set-context da-admin --cluster=demo-rbac --user=da-admin
kubectl --context=da-admin get nodes
@olgch; @kublr
Authentication: Authenticating Proxy
Auth Proxy
2. Request “X-Remote-*” headers
X-Remote-User: user1
X-Remote-Groups: group1
1. Request X-Remote-Extra-Key1: value1
Client Kubernetes API
--requestheader-username-headers=X-Remote-User
--requestheader-group-headers=X-Remote-Group
--requestheader-extra-headers-prefix=X-Remote-Extra-
@olgch; @kublr
Authentication: Impersonation
1. Request with “Impersonate-*” headers
Client Kubernetes API
Impersonate-User: jane.doe@example.com
Impersonate-Group: developers
Impersonate-Group: admins
Impersonate-Extra-dn: cn=jane,ou=engineers
Impersonate-Extra-scopes: view
kubectl get nodes \
--as "system:serviceaccount:default:sa1" \
--as-group g1 \
--as-group g2
@olgch; @kublr
Authorization: Mechanisms
Mechanism Decision source Usage
Node API Server built-in Internal use (kubelets)
ABAC Static file Insecure, deprecated
RBAC API Objects Users and administrators
WebHook External services Integration
AlwaysDeny API Server built-in Testing
AlwaysAllow
@olgch; @kublr
Authorization, tools
• kubectl auth can-i ...
• kubectl whoami
• kubectl --v=8 ...
@olgch; @kublr
Authorization: WebHook
Authorization
Service
2. Verify
1. Request
Client Kubernetes API
--authorization-webhook-config-file=auth-cfg.yaml
Config file in kubeconfig format
@olgch; @kublr
Authorization: RBAC
Three groups
Subjects Operations Resources
list
User
get get Pods
create post Nodes
update put ConfigMaps
Group
delete delete Secrets
watch Deployments
patch …
Service Account
Connected through RBAC
@olgch; @kublr
Roles and ClusterRoles
apiVersion: rbac.authorization.k8s.io/v1 • Roles and ClusterRoles define a set of
kind: Role
allowed actions on resources
metadata:
namespace: default • Role is namespaced
name: role1
• Cannot include non-namespaces
rules: resources or non-resource URLs
- apiGroups: ['*']
resources: ['nodes', 'pods', 'pods/log']
verbs: ['get', 'list']
- apiGroups: ['*']
resources: ['configmaps']
resourceNames: ['my-configmap']
verbs: ['get', 'list']
@olgch; @kublr
ClusterRoles
apiVersion: rbac.authorization.k8s.io/v1 • ClusterRole is not namespaced
kind: ClusterRole
• non-namespaced resources access
metadata:
namespace: default
name: clusterRole1
• non-resource URLs access
rules:
- apiGroups: ['*']
resources: ['nodes', ‘pods']
verbs: ['get', 'list']
- nonResourceURLs: ['/api', '/healthz*']
verbs: ['get', 'head']
@olgch; @kublr
Aggregated ClusterRoles
Aggregated ClusterRole combines rules from other cluster roles
aggregationRule:
clusterRoleSelectors:
- matchLabels:
apiVersion: rbac.authorization.k8s.io/v1 label1: value1
kind: ClusterRole
Aggregated
metadata: ClusterRole
name: aggregatedClusterRole1
aggregationRule:
clusterRoleSelectors:
- matchLabels:
label1: value1
# The control plane automatically fills in the rules clusterRole1 clusterRole1 clusterRole1
rules: [] metadata: metadata: metadata:
labels: labels: labels:
label1: value1 label1: value1 label2: value2
label2: value2
@olgch; @kublr
RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding user1 user2
metadata: ClusterRole
name: roleBinding1
namespace: default
roleRef: namespace
apiGroup: rbac.authorization.k8s.io resources only
kind: Role
group
name: role1
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: user1
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: group1 Role RoleBinding ServiceAccount
- kind: ServiceAccount
name: sa1
namespace: default
namespace
@olgch; @kublr
ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding user1 user2
metadata: ClusterRole ClusterRoleBinding
name: roleBinding1
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io group
kind: ClusterRole
name: clusterRole1
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: user1
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: group1 Role ServiceAccount
- kind: ServiceAccount
name: sa1
namespace: default
namespace
@olgch; @kublr
Built-in ClusterRoles
Role Access Can do
cluster-admin Full access to all cluster resources Anything in the cluster
admin Full access to all namespace resources Anything in a namespace
edit Full access to all namespace resources Anything in a namespace except granting and
except Role , RoleBinding , checking access
LocalSubjectAccessReviews
view RO access to all namespace resources View and list non-sensitive objects in a
except sensitive ones namespace
@olgch; @kublr
Use-cases
• Start with build-in roles
• Cluster admin
• Namespace admin
• Namespace developer
• Namespace read-only user
• Define new roles as needed
• Beware of “gotchas”
@olgch; @kublr
“Gotchas”
• Privilege escalation via pod creation
• Non-namespaced objects
• CRD, PriorityClass, PodSecurityPolicy
• Often needed for development, especially in advanced DevOps / SRE culture
• As a result, developers need self-service dev cluster management capabilities
• Role and role binding conventions
• Name
• Role bindings – one per subject, one per role, or mixed
@olgch; @kublr
Next steps
• PodSecurityPolicy
• NetworkPolicy
• Limits and Quotas
• Admission Controllers
• Dynamic admission control
• Dynamic policies, OPA
@olgch; @kublr
Kubernetes API Groups and Objects
API Group API Obects
rbac.authorization.k8s.io/v1 ClusterRole
Role
ClusterRoleBinding
RoleBinding
authentication.k8s.io/v1 TokenReview
admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
ValidatingWebhookConfiguration
authorization.k8s.io/v1 LocalSubjectAccessReview
SelfSubjectAccessReview
SelfSubjectRulesReview
SubjectAccessReview
certificates.k8s.io/v1beta1 CertificateSigningRequest
policy/v1beta1 PodSecurityPolicy
@olgch; @kublr
References
https://github.com/rajatjindal/kubectl-whoami
https://kubernetes.io/docs/reference/access-authn-authz/rbac/
https://kubernetes.io/docs/concepts/cluster-administration/certificates/
https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/
@olgch; @kublr
Q&A
@olgch; @kublr
Weekly Virtual Event
Virtual Meetups Webinars
➔ Small intimate meetups ➔ Traditional webinar
➔ With attendee intro round (ideally on
camera)
Tomorrow: Designing reliable,
➔ Open discussion at the end
self-healing cloud native apps
Next week Thu: Cloud Abstraction, the
Tomorrow 4pm ET: A No-BS Checklist
Often Overlooked Power of
for Enterprise-grade Kubernetes
Kubernetes
Deployments
Meetup.com group: All Things Kubernetes Register on our website (under resources)
Meetup & Happy Hour
Signup for our newsletter Oleg Chunikhin
CTO
at kublr.com oleg@kublr.com
@olgch
Kublr | kublr.com
@kublr
@olgch; @kublr
