8000 feat: add minikube jenkinsfile and test by qdzlug · Pull Request #127 · nginxinc/kic-reference-architectures · GitHub
[go: up one dir, main page]
Skip to content

feat: add minikube jenkinsfile and test #127

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Apr 19, 2022
Merged

feat: add minikube jenkinsfile and test #127

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a5dd498
feat: add minikube deployment test
Apr 18, 2022
3719e5f
fix: update AWS jenkinsfile for new deployment methodology
Apr 18, 2022
9b6c420
feat: updated docs to indicate minikube success
Apr 18, 2022
23a7fb8
fix: documentation changes suggested in review
Apr 19, 2022
b43d4d9
fix: added in some links for concepts as raised in review
Apr 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 53 additions & 42 deletions docs/status-and-issues.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,53 +18,64 @@ includes the following:
- FQDN/IP: How does the project handle the IP addressing and FQDN for the certificates?
- Notes: Any additional information on the provider / project interaction.

All of these configurations use Pulumi code within Python as the Infrastructure as Code (IaC) manager.

| K8 Provider | Tested | Infrastructure Support | IC Options | FQDN/IP | Notes |
All of these configurations use Pulumi code within Python as the Infrastructand configuredure as Code (
| K8 Provider | Tested | Infrastructure Support | IC Options | FQDN/IP | Notes |
|-----------------|--------|-----------------------------|---------------------------------|-----------------|--------------------------------------------------|
| AWS EKS | Yes | Full Infrastructure Standup | Build, Pull (uses ECR) | Provided | |
| Digtal Ocean | Yes | Full Infrastructure Standup |NGINX / NGINX Plus (w/ JWT) (1) | Manual FQDN (2)||
| Azure AKS | Yes | Kubeconfig Only (3) | NGINX / NGINX Plus (w/ JWT) (1) | Manual FQDN (2) | |
| Google GKE | Yes | Kubeconfig Only (3) | NGINX / NGINX Plus (w/ JWT) (1) | Manual FQDN (2) | |
| MicroK8s | Yes | Kubeconfig Only (3) | NGINX / NGINX Plus (w/ JWT) (1) | Manual FQDN (2) | Storage, DNS, and Metallb need to be Enabled (4) |
| Harvester/RKE2 | Yes | Kubeconfig Only (3) | NGINX / NGINX Plus (w/ JWT) (1) | Manual FQDN (2) | Needs Storage, K8 LoadBalancer |
| K3S | Yes | Kubeconfig Only (3) | NGINX / NGINX Plus (w/ JWT) (1) | Manual FQDN (2) | Needs Storage, K8 LoadBalancer |
| Rancher Desktop | No | Kubeconfig Only (3) | NGINX / NGINX Plus (w/ JWT) (1) | Manual FQDN (2) | Needs Storage, K8 LoadBalancer |
| Minikube | No | Kubeconfig Only (3) | NGINX / NGINX Plus (w/ JWT) (1) | Manual FQDN (2) | Needs Storage, K8 LoadBalancer |
| AWS EKS | Yes | Full Infrastructure Standup | Build, Pull (uses ECR) | Provided | | | Digtal Ocean | Yes |
Full Infrastructure Standup |NGINX / NGINX Plus (w/ JWT) (1) | Manual FQDN (2)|| | Azure AKS | Yes | Kubeconfig
Only (3) | NGINX / NGINX Plus (w/ JWT) (1) | Manual FQDN (2) | | | Google GKE | Yes | Kubeconfig Only (3)
| NGINX / NGINX Plus (w/ JWT) (1) | Manual FQDN (2) | | | MicroK8s | Yes | Kubeconfig Only (3) | NGINX / NGINX
Plus (w/ JWT) (1) | Manual FQDN (2) | Storage, DNS, and Metallb need to be Enabled (4) | | Harvester/RKE2 | Yes |
Kubeconfig Only (3) | NGINX / NGINX Plus (w/ JWT) (1) | Manual FQDN (2) | Needs Storage, K8 LoadBalancer | | K3S
| Yes | Kubeconfig Only (3) | NGINX / NGINX Plus (w/ JWT) (1) | Manual FQDN (2) | | | Rancher Desktop | No |
Kubeconfig Only (3) | NGINX / NGINX Plus (w/ JWT) (1) | Manual FQDN (2) | Needs Storage, K8 LoadBalancer | |
Minikube | Yes | Kubeconfig Only (3) | NGINX / NGINX Plus (w/ JWT) (1) | Manual FQDN (2) | Needs Metallb Enabled
and configured |

### Notes:
1. The NGINX IC build/deploy process is currently under active development and support for IC will be standardized across
all providers. Follow [#81](https://github.com/nginxinc/kic-reference-architectures/issues/81) and
[#86](https://github.com/nginxinc/kic-reference-architectures/issues/86) for details. Currently, for all non-AWS environments
you have the option to specify either NGINX or NGINX Plus as your IC. The latter does require an active subscription and a
JWT to be included at build time. Please see the documentation for more details.
2. The process via which the IP and FQDN are created and used is currently under active development, and will be streamlined
and standardized for all providers. Follow [#82](https://github.com/nginxinc/kic-reference-architectures/issues/82) for
details.
3. The initial deployment was entirely built to work with AWS. As part of our reorganization the ability to use a
kubeconfig file was added, along with the necessary configuration to support additional standup options. This is currently
in active development and will result in this process being streamlined for these additional environments. Please follow
[#80](https://github.com/nginxinc/kic-reference-architectures/issues/80) for details.
4. We are currently using filebeat as our logging agent. This deployment requires that the correct paths to the container
log directory are present in the deployment data. We have discovered that this differs based on the K8 provider. Please
see [#76](https://github.com/nginxinc/kic-reference-architectures/issues/76) for more detail.

1. The NGINX IC build/deploy process is currently under active development and support for IC will be standardized
across all providers. Follow [#81](https://github.com/nginxinc/kic-reference-architectures/issues/81) and
[#86](https://github.com/nginxinc/kic-reference-architectures/issues/86) for details. Currently, for all non-AWS
environments you have the option to specify either NGINX or NGINX Plus as your IC. The latter does require an active
subscription and a JWT to be included at build time. Please see the documentation for more details.
2. The process via which the IP and FQDN are created and used is currently under active development, and will be
streamlined and standardized for all providers.
Follow [#82](https://github.com/nginxinc/kic-reference-architectures/issues/82) for details.
3. The initial deployment was entirely built to work with AWS. As part of our reorganization the ability to use a
kubeconfig file was added, along with the necessary configuration to support additional standup options. This is
currently in active development and will result in this process being streamlined for these additional environments.
Please follow
[#80](https://github.com/nginxinc/kic-reference-architectures/issues/80) for details.
4. We are currently using filebeat as our logging agent. This deployment requires that the correct paths to the
container log directory are present in the deployment data. We have discovered that this differs based on the K8
provider. Please see [#76](https://github.com/nginxinc/kic-reference-architectures/issues/76) for more detail.

## Known Issues / Caveats

1. Currently, the use of the Elastic tooling has shown to be problematic under heavy load, with containers falling over
and causing disruptions. Please see the tuning variables in the configuration file to adjust the number of replicas
deployed for the Elastic logstore to tune to your environment.
2. The default Helm timeout is 5 minutes, which is acceptable for most managed clouds but tends to be too short for
single-vm or workstation deployments. Please see the configuration file variables to adjust the helm timeout as required
for your environment.
3. When load testing the Bank of Sirius using Locust, you will likely see a high failure rate as you increase the max
users and spawn rate. This is "normal" and is an area we want to expose and explore for troubleshooting, determining
which metrics/traces are helpful, etc.
and causing disruptions. Please see the [example configuration file](../config/pulumi/Pulumi.stackname.yaml.example)
variables to adjust the number of replicas deployed for the Elastic logstore to tune to your environment. These will
need to be added/updated in the configuration for your stack, which is located in `./config/pulumi` and
is named `Pulumi.$STACK.yaml`.
2. The default Helm timeout is 5 minutes, which is acceptable for most managed clouds but tends to be too short for
single-vm or workstation deployments. Please see
the [example configuration file](../config/pulumi/Pulumi.stackname.yaml.example)
variables to adjust the helm timeout as required for your environment. These will need to be added/updated in the
configuration for your stack, which is located in `./config/pulumi` and is named `Pulumi.$STACK.yaml`.
3. When load testing the Bank of Sirius using [Locust](https://locust.io/), you will likely see a high failure rate as
you increase the max users and spawn rate. This is "normal" and is an area we want to expose and explore for
troubleshooting, determining which metrics/traces are helpful, etc.
4. The most common failure modes for non-cloud environments tend towards the following failures:
1. Unable to provision persistent storage; correct by ensuring you have a persistent volume provider and can provision a volume.
2. Unable to provision an External IP; correct by adding an IP provider such as kubevip or metallb.
3. Resource starvation (not enough CPU, Memory); expand the size of the VM or detune the environment.
4. Timeouts in helm; increase the helm timeout in the configuration file.
5. If you are using a cloud provider with timed credentials, such as AWS, one failure mode that can arise is when the
credentials expire. This will result in a number of strange and seemingly confusing errors. Double check to make sure that
the credentials are valid.
1. Unable to provision persistent storage; correct by ensuring you have a
[persistent volume provider](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) and can provision a
volume.
2. Unable to provision an External IP; correct by adding an IP provider such
as [kubevip](https://kube-vip.chipzoller.dev/)
or [metallb](https://metallb.org/).
3. Resource starvation (not enough CPU, Memory); expand the size of the VM or detune the environment.
4. Timeouts in helm; increase the helm timeout in the configuration file.
5. If you are using a cloud provider with timed credentials, such as AWS, one failure mode that can arise is when the
credentials expire. This will result in a number of strange and seemingly confusing errors. Double check to make sure
that the credentials are valid.
6. Currently, the build/test process is highly manual. This will be addressed in the future.
2 changes: 1 addition & 1 deletion extras/jenkins/AWS/Jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ pipeline {
$WORKSPACE/pulumi/python/venv/bin/pulumi config set kic-helm:fqdn "marajenks${BUILD_NUMBER}.zathras.io" -C pulumi/python/config -s marajenkaws${BUILD_NUMBER}
$WORKSPACE/pulumi/python/venv/bin/pulumi config set kic-helm:helm_timeout "600" -C pulumi/python/config -s marajenkaws${BUILD_NUMBER}
$WORKSPACE/pulumi/python/venv/bin/pulumi config set kubernetes:infra_type "AWS" -C pulumi/python/config -s marajenkaws${BUILD_NUMBER}
$WORKSPACE/pulumi/python/venv/bin/pulumi config set kubernetes:kubeconfig "/home/jerkins/.kube/config" -C pulumi/python/config -s marajenkaws${BUILD_NUMBER}
$WORKSPACE/pulumi/python/venv/bin/pulumi config set kubernetes:kubeconfig "$HOME/.kube/config" -C pulumi/python/config -s marajenkaws${BUILD_NUMBER}
$WORKSPACE/pulumi/python/venv/bin/pulumi config set logagent:helm_timeout "600" -C pulumi/python/config -s marajenkaws${BUILD_NUMBER}
$WORKSPACE/pulumi/python/venv/bin/pulumi config set logstore:helm_timeout "600" -C pulumi/python/config -s marajenkaws${BUILD_NUMBER}
$WORKSPACE/pulumi/python/venv/bin/pulumi config set prometheus:adminpass "password" -C pulumi/python/config -s marajenkaws${BUILD_NUMBER}
Expand Down
Loading
0