DefectDojo on Kubernetes

DefetDojo Kubernetes utilizes Helm, a package manager for Kubernetes. Helm Charts help you define, install, and upgrade even the most complex Kubernetes application.

For development purposes, minikube and Helm can be installed locally by following this guide.

Kubernetes Local Quickstart

Requirements:

Helm installed locally
Minikube installed locally
Latest cloned copy of DefectDojo

git clone https://github.com/DefectDojo/django-DefectDojo
cd django-DefectDojo

minikube start
minikube addons enable ingress

Helm <= v2

helm init
helm repo update

Helm >= v3

helm repo add stable https://charts.helm.sh/stable
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

Then pull the dependent charts:

helm dependency update ./helm/defectdojo

Now, install the helm chart into minikube.

If you have setup an ingress controller:

DJANGO_INGRESS_ENABLED=true

else:

DJANGO_INGRESS_ENABLED=false

If you have configured TLS:

DJANGO_INGRESS_ACTIVATE_TLS=true

else:

DJANGO_INGRESS_ACTIVATE_TLS=false

Warning: Use the createSecret*=true flags only upon first install. For re-installs, see §Re-install the chart

Helm <= v2:

helm install \
  ./helm/defectdojo \
  --name=defectdojo \
  --set django.ingress.enabled=${DJANGO_INGRESS_ENABLED} \
  --set django.ingress.activateTLS=${DJANGO_INGRESS_ACTIVATE_TLS} \
  --set createSecret=true \
  --set createRabbitMqSecret=true \
  --set createRedisSecret=true \
  --set createMysqlSecret=true \
  --set createPostgresqlSecret=true

Helm >= v3:

helm install \
  defectdojo \
  ./helm/defectdojo \
  --set django.ingress.enabled=${DJANGO_INGRESS_ENABLED} \
  --set django.ingress.activateTLS=${DJANGO_INGRESS_ACTIVATE_TLS} \
  --set createSecret=true \
  --set createRabbitMqSecret=true \
  --set createRedisSecret=true \
  --set createMysqlSecret=true \
  --set createPostgresqlSecret=true

Note that you need only one of:

postgresql or mysql
rabbitmq or redis

It usually takes up to a minute for the services to startup and the status of the containers can be viewed by starting up minikube dashboard. Note: If the containers are not cached locally the services will start once the containers have been pulled locally.

To be able to access DefectDojo, set up an ingress or access the service directly by running the following command:

kubectl port-forward --namespace=default \
service/defectdojo-django 8080:80

As you set your host value to defectdojo.default.minikube.local, make sure that it resolves to the localhost IP address, e.g. by adding the following two lines to /etc/hosts:

::1       defectdojo.default.minikube.local
127.0.0.1 defectdojo.default.minikube.local

To find out the password, run the following command:

echo "DefectDojo admin password: $(kubectl \
  get secret defectdojo \
  --namespace=default \
  --output jsonpath='{.data.DD_ADMIN_PASSWORD}' \
  | base64 --decode)"

To access DefectDojo, go to http://defectdojo.default.minikube.local:8080. Log in with username admin and the password from the previous command.

Minikube with locally built containers

If testing containers locally, then set the imagePullPolicy to Never, which ensures containers are not pulled from Docker hub.

Use the same commands as before but add:

  --set imagePullPolicy=Never

Installing from a private registry

If you have stored your images in a private registry, you can install defectdojo chart with (helm 3).

First create a secret named "defectdojoregistrykey" based on the credentials that can pull from the registry: see https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
Then install the chart with the same commands as before but adding:

  --set repositoryPrefix=<myregistry.com/path> \
  --set imagePullSecrets=defectdojoregistrykey

Build Images Locally

# Build images
docker build -t defectdojo/defectdojo-django -f Dockerfile.django .
docker build -t defectdojo/defectdojo-nginx -f Dockerfile.nginx .

# Build images behind proxy
docker build --build-arg http_proxy=http://myproxy.com:8080 --build-arg https_proxy=http://myproxy.com:8080 -t defectdojo/defectdojo-django -f Dockerfile.django .
docker build --build-arg http_proxy=http://myproxy.com:8080 --build-arg https_proxy=http://myproxy.com:8080 -t defectdojo/defectdojo-nginx -f Dockerfile.nginx .

Debug uWSGI with ptvsd

You can set breakpoints in code that is handled by uWSGI. The feature is meant to be used when you run locally on minikube, and mimics what can be done with docker-compose.

The port is currently hard-coded to 3000.

In values.yaml, ensure the value for enable_ptvsd is set to true (the default is false). Make sure the change is taken into account in your deployment.
Have DD_DEBUG set to True.
Port forward port 3000 to the pod, such as kubectl port-forward defectdojo-django-7886f49466-7cwm7 3000.

Upgrade the chart

If you want to change kubernetes configuration of use an updated docker image (evolution of defectDojo code), upgrade the application:

kubectl delete job defectdojo-initializer
helm upgrade  defectdojo ./helm/defectdojo/ \
   --set django.ingress.enabled=${DJANGO_INGRESS_ENABLED} \
   --set django.ingress.activateTLS=${DJANGO_INGRESS_ACTIVATE_TLS}

Re-install the chart

In case of issue or in any other situation where you need to re-install the chart, you can do it and re-use the same secrets.

Note that when using mysql, this will create a new database, while with postgresql you'll keep the same database (more information below)

# helm 3
helm uninstall defectdojo
helm install \
  defectdojo \
  ./helm/defectdojo \
  --set django.ingress.enabled=${DJANGO_INGRESS_ENABLED} \
  --set django.ingress.activateTLS=${DJANGO_INGRESS_ACTIVATE_TLS}

Kubernetes Production

When running defectdojo in production be aware that you understood the full setup and always have a backup.

Encryption to Kubernetes

Optionally, for TLS locally, you need to install a TLS certificate into your Kubernetes cluster. For development purposes, you can create your own certificate authority as described here.

# https://kubernetes.io/docs/concepts/services-networking/ingress/#tls
# Create a TLS secret called minikube-tls as mentioned above, e.g.
K8S_NAMESPACE="default"
TLS_CERT_DOMAIN="${K8S_NAMESPACE}.minikube.local"
kubectl --namespace "${K8S_NAMESPACE}" create secret tls defectdojo-tls \
  --key <(openssl rsa \
    -in "${CA_DIR}/private/${TLS_CERT_DOMAIN}.key.pem" \
    -passin "pass:${TLS_CERT_PASSWORD}") \
  --cert <(cat \
    "${CA_DIR}/certs/${TLS_CERT_DOMAIN}.cert.pem" \
    "${CA_DIR}/chain.pem")

Encryption in Kubernetes and End-to-End Encryption

With the TLS certificate from your Kubernetes cluster all traffic to you cluster is encrypted, but the traffic in your cluster is still unencrypted.

If you want to encrypt the traffic to the nginx server you can use the option --set nginx.tls.enabled=true and --set nginx.tls.generateCertificate=true to generate a self signed certificate and use the https config. The option to add you own pregenerated certificate is generelly possible but not implemented in the helm chart yet.

Be aware that the traffic to the database and celery broker are unencrypted at the moment.

Installation

# Install Helm chart. Choose a host name that matches the certificate above
helm install \
  ./helm/defectdojo \
  --name=defectdojo \
  --namespace="${K8S_NAMESPACE}" \
  --set host="defectdojo.${TLS_CERT_DOMAIN}" \
  --set django.ingress.secretName="minikube-tls" \
  --set createSecret=true \
  --set createRabbitMqSecret=true \
  --set createRedisSecret=true \
  --set createMysqlSecret=true \
  --set createPostgresqlSecret=true

# For high availability deploy multiple instances of Django, Celery and RabbitMQ
helm install \
  ./helm/defectdojo \
  --name=defectdojo \
  --namespace="${K8S_NAMESPACE}" \
  --set host="defectdojo.${TLS_CERT_DOMAIN}" \
  --set django.ingress.secretName="minikube-tls" \
  --set django.replicas=3 \
  --set celery.replicas=3 \
  --set rabbitmq.replicas=3 \
  --set createSecret=true \
  --set createRabbitMqSecret=true \
  --set createRedisSecret=true \
  --set createMysqlSecret=true \
  --set createPostgresqlSecret=true

# Run highly available PostgreSQL cluster instead of MySQL - recommended setup
# for production environment.
helm install \
  ./helm/defectdojo \
  --name=defectdojo \
  --namespace="${K8S_NAMESPACE}" \
  --set host="defectdojo.${TLS_CERT_DOMAIN}" \
  --set django.replicas=3 \
  --set celery.replicas=3 \
  --set rabbitmq.replicas=3 \
  --set django.ingress.secretName="minikube-tls" \
  --set mysql.enabled=false \
  --set database=postgresql \
  --set postgresql.enabled=true \
  --set postgresql.replication.enabled=true \
  --set postgresql.replication.slaveReplicas=3 \
  --set createSecret=true \
  --set createRabbitMqSecret=true \
  --set createRedisSecret=true \
  --set createMysqlSecret=true \
  --set createPostgresqlSecret=true

# Note: If you run `helm install defectdojo before, you will get an error
# message like `Error: release defectdojo failed: secrets "defectdojo" already
# exists`. This is because the secret is kept across installations.
# To prevent recreating the secret, add --set createSecret=false` to your
# command.

# Run test. If there are any errors, re-run the command without `--cleanup` and
# inspect the test container.
# helm 2
helm test defectdojo --cleanup
# helm 3
helm test defectdojo

# Navigate to <https://defectdojo.default.minikube.local>.

TODO: The MySQL volumes aren't persistent across helm uninstall operations. To make them persistent, you need to add an annotation to the persistent volume claim:

kubectl --namespace "${K8S_NAMESPACE}" patch pvc defectdojo-mysql -p \
  '{"metadata": {"annotations": {"\"helm.sh/resource-policy\"": "keep"}}}'

See also https://github.com/helm/charts/blob/master/stable/mysql/templates/pvc.yaml.

However, that doesn't work and I haven't found out why. In a production environment, a redundant PostgreSQL cluster is the better option. As it uses statefulsets that are kept by default, the problem doesn't exist there.

Prometheus metrics

It's possible to enable Nginx prometheus exporter by setting --set monitoring.enabled=true and --set monitoring.prometheus.enabled=true. This adds the Nginx exporter sidecar and the standard Prometheus pod annotations to django deployment.

Useful stuff

# View logs of a specific pod
kubectl logs $(kubectl get pod --selector=defectdojo.org/component=${POD} \
  -o jsonpath="{.items[0].metadata.name}") -f

# Open a shell in a specific pod
kubectl exec -it $(kubectl get pod --selector=defectdojo.org/component=${POD} \
  -o jsonpath="{.items[0].metadata.name}") -- /bin/bash
# Or:
kubectl exec defectdojo-django-<xxx-xxx> -c uwsgi -it /bin/sh

# Open a Python shell in a specific pod
kubectl exec -it $(kubectl get pod --selector=defectdojo.org/component=${POD} \
  -o jsonpath="{.items[0].metadata.name}") -- python manage.py shell

Clean up Kubernetes

Helm <= v2

# Uninstall Helm chart
helm delete defectdojo --purge

Helm >= v3

helm uninstall defectdojo

To remove persistent objects not removed by uninstall (this will remove any database):

kubectl delete secrets defectdojo defectdojo-redis-specific defectdojo-rabbitmq-specific defectdojo-postgresql-specific defectdojo-mysql-specific
kubectl delete pvc data-defectdojo-rabbitmq-0 data-defectdojo-postgresql-0