apache_airflow_install.
md 2024-01-15
Remove existing airflow helm chart
helm delete <release-name> -n <release_namespace>
Install new release
helm repo add apache-airflow https://airflow.apache.org
helm upgrade --install airflow apache-airflow/airflow --namespace airflow --
create-namespace
Extract configuration from the running Airflow
helm show values apache-airflow/airflow > values.yaml
Update configuration
Open the values.yaml file and make the necessary changes to the configuration.
Here are our configuration updates:
Airflow Executor: By default, the executor is set to "CeleryExecutor". In our
case we prefer to use the Kubernetes executor, so update the executor value to
"KubernetesExecutor".
# Airflow executor
# One of: LocalExecutor, LocalKubernetesExecutor, CeleryExecutor,
KubernetesExecutor, CeleryKubernetesExecutor
executor: "KubernetesExecutor"
images:
airflow:
repository: our_own_apache_airflow
tag: "1.0"
# Specifying digest takes precedence over tag.
digest: ~
pullPolicy: IfNotPresent
Create Airflow Dockerfile
#Dockerfile
FROM apache/airflow:2.6.2
1/2
apache_airflow_install.md 2024-01-15
COPY requirements.txt /
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r
/requirements.txt
Create requirement.txt file:
apache-airflow-providers-cncf-kubernetes
apache-airflow-providers-apache-spark
Build the image:
docker build -t our_own_apache_airflow:1.0 .
Load the created image to Kubernetes internal repository
It should be that the Docker image exists in the cluster because the image was
created on the local machine.
Update airflow installation
helm upgrade --install airflow apache-airflow/airflow --namespace airflow --
values values.yaml
Let’s check access to the web UI
kubectl port-forward svc/airflow-webserver 8080:8080 --namespace airflow
Next, get to http://localhost:8080 with these creds:
User: admin
Password: admin
Now you should see Airflow webserver UI.
2/2
