diff --git a/.vscode/settings.json b/.vscode/settings.json index 93b329f8a21a5..f2cf72b7d8ae0 100644 --- a/.vscode/settings.json +++ b/.vscode/settings.json @@ -57,5 +57,8 @@ "[css][html][markdown][yaml]": { "editor.defaultFormatter": "esbenp.prettier-vscode" }, - "typos.config": ".github/workflows/typos.toml" + "typos.config": ".github/workflows/typos.toml", + "[markdown]": { + "editor.defaultFormatter": "DavidAnson.vscode-markdownlint" + } } diff --git a/coderd/provisionerdserver/acquirer_test.go b/coderd/provisionerdserver/acquirer_test.go index 22794c72657cc..91e5964f1e8ed 100644 --- a/coderd/provisionerdserver/acquirer_test.go +++ b/coderd/provisionerdserver/acquirer_test.go @@ -518,7 +518,7 @@ func TestAcquirer_MatchTags(t *testing.T) { t.Run("GenTable", func(t *testing.T) { t.Parallel() - // Generate a table that can be copy-pasted into docs/admin/provisioners.md + // Generate a table that can be copy-pasted into docs/admin/provisioners/index.md lines := []string{ "\n", "| Provisioner Tags | Job Tags | Same Org | Can Run Job? |", @@ -547,7 +547,7 @@ func TestAcquirer_MatchTags(t *testing.T) { s := fmt.Sprintf("| %s | %s | %s | %s |", kvs(tt.acquireJobTags), kvs(tt.provisionerJobTags), sameOrg, acquire) lines = append(lines, s) } - t.Log("You can paste this into docs/admin/provisioners.md") + t.Log("You can paste this into docs/admin/provisioners/index.md") t.Log(strings.Join(lines, "\n")) }) } diff --git a/docs/admin/infrastructure/architecture.md b/docs/admin/infrastructure/architecture.md index 9b2c2365a4966..dbac881bddeb8 100644 --- a/docs/admin/infrastructure/architecture.md +++ b/docs/admin/infrastructure/architecture.md @@ -42,7 +42,7 @@ _provisionerd_ is the execution context for infrastructure modifying providers. At the moment, the only provider is Terraform (running `terraform`). By default, the Coder server runs multiple provisioner daemons. -[External provisioners](../provisioners.md) can be added for security or +[External provisioners](../provisioners/index.md) can be added for security or scalability purposes. ### Workspaces diff --git a/docs/admin/monitoring/health-check.md b/docs/admin/monitoring/health-check.md index cd14810883f52..456d52e0bce8b 100644 --- a/docs/admin/monitoring/health-check.md +++ b/docs/admin/monitoring/health-check.md @@ -294,7 +294,7 @@ be built until there is at least one provisioner daemon running. **Solution:** If you are using -[External Provisioner Daemons](../provisioners.md#external-provisioners), ensure +[External Provisioner Daemons](../provisioners/index.md#external-provisioners), ensure that they are able to successfully connect to Coder. Otherwise, ensure [`--provisioner-daemons`](../../reference/cli/server.md#--provisioner-daemons) is set to a value greater than 0. diff --git a/docs/admin/monitoring/logs.md b/docs/admin/monitoring/logs.md index 49861090800ac..f1a5b499075f3 100644 --- a/docs/admin/monitoring/logs.md +++ b/docs/admin/monitoring/logs.md @@ -24,7 +24,7 @@ Connect logs are all captured in the `coderd` logs. ## `provisionerd` Logs -Logs for [external provisioners](../provisioners.md) are structured +Logs for [external provisioners](../provisioners/index.md) are structured [and configured](../../reference/cli/provisioner_start.md#--log-human) similarly to `coderd` logs. Use these logs to troubleshoot and monitor the Terraform operations behind workspaces and templates. diff --git a/docs/admin/provisioners.md b/docs/admin/provisioners/index.md similarity index 91% rename from docs/admin/provisioners.md rename to docs/admin/provisioners/index.md index 35be50162c395..ac8cbfb48b39b 100644 --- a/docs/admin/provisioners.md +++ b/docs/admin/provisioners/index.md @@ -1,7 +1,7 @@ # External provisioners By default, the Coder server runs -[built-in provisioner daemons](../reference/cli/server.md#--provisioner-daemons), +[built-in provisioner daemons](../../reference/cli/server.md#--provisioner-daemons), which execute `terraform` during workspace and template builds. However, there are often benefits to running external provisioner daemons: @@ -11,7 +11,7 @@ are often benefits to running external provisioner daemons: - **Isolate APIs:** Deploy provisioners in isolated environments (on-prem, AWS, Azure) instead of exposing APIs (Docker, Kubernetes, VMware) to the Coder server. See - [Provider Authentication](../admin/templates/extending-templates/provider-authentication.md) + [Provider Authentication](../../admin/templates/extending-templates/provider-authentication.md) for more details. - **Isolate secrets**: Keep Coder unaware of cloud secrets, manage/rotate @@ -19,19 +19,21 @@ are often benefits to running external provisioner daemons: - **Reduce server load**: External provisioners reduce load and build queue times from the Coder server. See - [Scaling Coder](../admin/infrastructure/index.md#scale-tests) for more + [Scaling Coder](../../admin/infrastructure/index.md#scale-tests) for more details. Each provisioner runs a single -[concurrent workspace build](../admin/infrastructure/scale-testing.md#control-plane-provisionerd). +[concurrent workspace build](../../admin/infrastructure/scale-testing.md#control-plane-provisionerd). For example, running 30 provisioner containers will allow 30 users to start workspaces at the same time. Provisioners are started with the -[`coder provisioner start`](../reference/cli/provisioner_start.md) command in +[`coder provisioner start`](../../reference/cli/provisioner_start.md) command in the [full Coder binary](https://github.com/coder/coder/releases). Keep reading to learn how to start provisioners via Docker, Kubernetes, Systemd, etc. +You can use the dashboard, CLI, or API to [manage provisioners](./manage-provisioner-jobs.md). + ## Authentication The provisioner daemon must authenticate with your Coder deployment. @@ -83,7 +85,7 @@ Kubernetes/Docker/etc. A user account with the role `Template Admin` or `Owner` can start provisioners using their user account. This may be beneficial if you are running provisioners -via [automation](../reference/index.md). +via [automation](../../reference/index.md). ```sh coder login https:// @@ -110,7 +112,7 @@ Global pre-shared keys (PSK) make it difficult to rotate keys or isolate provisi A deployment-wide PSK can be used to authenticate any provisioner. To use a global PSK, set a -[provisioner daemon pre-shared key (PSK)](../reference/cli/server.md#--provisioner-daemon-psk) +[provisioner daemon pre-shared key (PSK)](../../reference/cli/server.md#--provisioner-daemon-psk) on the Coder server. Next, start the provisioner: @@ -157,12 +159,12 @@ coder templates push on-prem-chicago \ This can also be done in the UI when building a template: -![template tags](../images/admin/provisioner-tags.png) +![template tags](../../images/admin/provisioner-tags.png) Alternatively, a template can target a provisioner via [workspace tags](https://github.com/coder/coder/tree/main/examples/workspace-tags) inside the Terraform. See the -[workspace tags documentation](../admin/templates/extending-templates/workspace-tags.md) +[workspace tags documentation](../../admin/templates/extending-templates/workspace-tags.md) for more information. > [!NOTE] @@ -237,17 +239,17 @@ This is illustrated in the below table: Provisioners can broadly be categorized by scope: `organization` or `user`. The scope of a provisioner can be specified with -[`-tag=scope=`](../reference/cli/provisioner_start.md#-t---tag) when +[`-tag=scope=`](../../reference/cli/provisioner_start.md#-t---tag) when starting the provisioner daemon. Only users with at least the -[Template Admin](./users/index.md#roles) role or higher may create +[Template Admin](../users/index.md#roles) role or higher may create organization-scoped provisioner daemons. There are two exceptions: -- [Built-in provisioners](../reference/cli/server.md#--provisioner-daemons) are +- [Built-in provisioners](../../reference/cli/server.md#--provisioner-daemons) are always organization-scoped. - External provisioners started using a - [pre-shared key (PSK)](../reference/cli/provisioner_start.md#--psk) are always + [pre-shared key (PSK)](../../reference/cli/provisioner_start.md#--psk) are always organization-scoped. ### Organization-Scoped Provisioners @@ -371,7 +373,7 @@ docker run --rm -it \ As mentioned above, the Coder server will run built-in provisioners by default. This can be disabled with a server-wide -[flag or environment variable](../reference/cli/server.md#--provisioner-daemons). +[flag or environment variable](../../reference/cli/server.md#--provisioner-daemons). ```sh coder server --provisioner-daemons=0 @@ -390,3 +392,7 @@ address. If you have provisioners daemons deployed as pods, it is advised to monitor them separately. + +## Next + +- [Manage Provisioners](./manage-provisioner-jobs.md) diff --git a/docs/admin/provisioners/manage-provisioner-jobs.md b/docs/admin/provisioners/manage-provisioner-jobs.md new file mode 100644 index 0000000000000..05d5d9dddff9f --- /dev/null +++ b/docs/admin/provisioners/manage-provisioner-jobs.md @@ -0,0 +1,80 @@ +# Manage provisioner jobs + +[Provisioners](./index.md) start and run provisioner jobs to create or delete workspaces. +Each time a workspace is built, rebuilt, or destroyed, it generates a new job and assigns +the job to an available provisioner daemon for execution. + +While most jobs complete smoothly, issues with templates, cloud resources, or misconfigured +provisioners can cause jobs to fail or hang indefinitely (these are in a `Pending` state). + +![Provisioner jobs in the dashboard](../../images/admin/provisioners/provisioner-jobs.png) + +## How to find provisioner jobs + +Coder admins can view and manage provisioner jobs. + +Use the dashboard, CLI, or API: + +- **Dashboard**: + + Select **Admin settings** > **Organizations** > **Provisioner Jobs** + + Provisioners are organization-specific. If you have more than one organization, select it first. + +- **CLI**: `coder provisioner jobs list` +- **API**: `/api/v2/provisioner/jobs` + +## Manage provisioner jobs from the dashboard + +View more information about and manage your provisioner jobs from the Coder dashboard. + +1. Under **Admin settings** select **Organizations**, then select **Provisioner jobs**. + +1. Select the **>** to expand each entry for more information. + +1. To delete a job, select the 🚫 at the end of the entry's row. + + If your user doesn't have the correct permissions, this option is greyed out. + +## Provisioner job status + +Each provisioner job has a lifecycle state: + +| Status | Description | +|---------------|----------------------------------------------------------------| +| **Pending** | Job is queued but has not yet been picked up by a provisioner. | +| **Running** | A provisioner is actively working on the job. | +| **Completed** | Job succeeded. | +| **Failed** | Provisioner encountered an error while executing the job. | +| **Canceled** | Job was manually terminated by an admin. | + +## When to cancel provisioner jobs + +A job might need to be cancelled when: + +- It has been stuck in **Pending** for too long. This can be due to misconfigured tags or unavailable provisioners. +- It is **Running** indefinitely, often caused by external system failures or buggy templates. +- An admin wants to abort a failed attempt, fix the root cause, and retry provisioning. +- A workspace was deleted in the UI but the underlying cloud resource wasn’t cleaned up, causing a hanging delete job. + +Cancelling a job does not automatically retry the operation. +It clears the stuck state and allows the admin or user to trigger the action again if needed. + +## Troubleshoot provisioner jobs + +Provisioner jobs can fail or slow workspace creation for a number of reasons. +Follow these steps to identify problematic jobs or daemons: + +1. Filter jobs by `pending` status in the dashboard, or use the CLI: + + ```bash + coder provisioner jobs list -s pending + ``` + +1. Look for daemons with multiple failed jobs and for template [tag mismatches](./index.md#provisioner-tags). + +1. Cancel the job through the dashboard, or use the CLI: + + ```shell + coder provisioner jobs cancel + ``` diff --git a/docs/admin/security/secrets.md b/docs/admin/security/secrets.md index 7985c73ba8390..25ff1a6467f02 100644 --- a/docs/admin/security/secrets.md +++ b/docs/admin/security/secrets.md @@ -7,7 +7,7 @@ guide to This article explains how to use secrets in a workspace. To authenticate the workspace provisioner, see the -provisioners documentation. +provisioners documentation. ## Before you begin diff --git a/docs/admin/setup/index.md b/docs/admin/setup/index.md index cf01d14fbc30b..96000292266e2 100644 --- a/docs/admin/setup/index.md +++ b/docs/admin/setup/index.md @@ -154,4 +154,4 @@ more information. ## Up Next - [Setup and manage templates](../templates/index.md) -- [Setup external provisioners](../provisioners.md) +- [Setup external provisioners](../provisioners/index.md) diff --git a/docs/admin/templates/creating-templates.md b/docs/admin/templates/creating-templates.md index 50b35b07d52b6..a0a6b54366948 100644 --- a/docs/admin/templates/creating-templates.md +++ b/docs/admin/templates/creating-templates.md @@ -68,7 +68,7 @@ coder templates push > [!NOTE] > If `template push` fails, Coder is likely not authorized to deploy > infrastructure in the given location. Learn how to configure -> [provisioner authentication](../provisioners.md). +> [provisioner authentication](../provisioners/index.md). You can edit the metadata of the template such as the display name with the [`templates edit`](../../reference/cli/templates_edit.md) command: diff --git a/docs/admin/templates/extending-templates/modules.md b/docs/admin/templates/extending-templates/modules.md index 488d43eb616f0..1f454bb26540c 100644 --- a/docs/admin/templates/extending-templates/modules.md +++ b/docs/admin/templates/extending-templates/modules.md @@ -120,7 +120,7 @@ template as the underlying module. ### Private git repository If you are importing a module from a private git repository, the Coder server or -[provisioner](../../provisioners.md) needs git credentials. Since this token +[provisioner](../../provisioners/index.md) needs git credentials. Since this token will only be used for cloning your repositories with modules, it is best to create a token with access limited to the repository and no extra permissions. In GitHub, you can generate a diff --git a/docs/admin/templates/extending-templates/provider-authentication.md b/docs/admin/templates/extending-templates/provider-authentication.md index fe2572814358d..4ddf23fa38fb2 100644 --- a/docs/admin/templates/extending-templates/provider-authentication.md +++ b/docs/admin/templates/extending-templates/provider-authentication.md @@ -46,7 +46,7 @@ There are two ways to use a remote Docker host for authentication: - Configure the Docker provider to use a [remote host over SSH or TCP](https://registry.terraform.io/providers/kreuzwerker/docker/latest/docs#remote-hosts). -- Run an [external provisioner](../../provisioners.md) on the remote docker +- Run an [external provisioner](../../provisioners/index.md) on the remote docker host. Other providers might also support authenticated environments. Check the diff --git a/docs/admin/users/groups-roles.md b/docs/admin/users/groups-roles.md index ffcf610235c72..a748eacbc9886 100644 --- a/docs/admin/users/groups-roles.md +++ b/docs/admin/users/groups-roles.md @@ -24,7 +24,7 @@ Roles determine which actions users can take within the platform. | Manage **ALL** Templates | | | ✅ | ✅ | | View **ALL** Workspaces | | | ✅ | ✅ | | Update and delete **ALL** Workspaces | | | | ✅ | -| Run [external provisioners](../provisioners.md) | | | ✅ | ✅ | +| Run [external provisioners](../provisioners/index.md) | | | ✅ | ✅ | | Execute and use **ALL** Workspaces | | | | ✅ | | View all user operation [Audit Logs](../security/audit-logs.md) | ✅ | | | ✅ | @@ -80,7 +80,7 @@ Note that these permissions only apply to the scope of an A malicious Template Admin could write a template that executes commands on the host (or `coder server` container), which potentially escalates their privileges or shuts down the Coder server. To avoid this, run -[external provisioners](../provisioners.md). +[external provisioners](../provisioners/index.md). In low-trust environments, we do not recommend giving users direct access to edit templates. Instead, use diff --git a/docs/admin/users/organizations.md b/docs/admin/users/organizations.md index 47691d6dd6ea9..b38c46cd48549 100644 --- a/docs/admin/users/organizations.md +++ b/docs/admin/users/organizations.md @@ -37,7 +37,7 @@ From there, you can manage the name, icon, description, users, and groups: Any additional organizations have unique admins, users, templates, provisioners, groups, and workspaces. Each organization must have at least one dedicated -[provisioner](../provisioners.md) since the built-in provisioners only apply to +[provisioner](../provisioners/index.md) since the built-in provisioners only apply to the default organization. You can configure [organization/role/group sync](./idp-sync.md) from your @@ -71,7 +71,7 @@ Next deploy a provisioner and template for this organization. ### 2. Deploy a provisioner -[Provisioners](../provisioners.md) are organization-scoped and are responsible +[Provisioners](../provisioners/index.md) are organization-scoped and are responsible for executing Terraform/OpenTofu to provision the infrastructure for workspaces and testing templates. Before creating templates, we must deploy at least one provisioner as the built-in provisioners are scoped to the default organization. @@ -90,7 +90,7 @@ provisioner as the built-in provisioners are scoped to the default organization. In this example, start the provisioner using the Coder CLI on a host with Docker. For instructions on using other platforms like Kubernetes, see our - [provisioner documentation](../provisioners.md). + [provisioner documentation](../provisioners/index.md). ```sh export CODER_URL=https:// diff --git a/docs/images/admin/provisioners/provisioner-jobs.png b/docs/images/admin/provisioners/provisioner-jobs.png new file mode 100644 index 0000000000000..817f5cb5e341d Binary files /dev/null and b/docs/images/admin/provisioners/provisioner-jobs.png differ diff --git a/docs/manifest.json b/docs/manifest.json index f37f9a9db67f7..a343da589ee31 100644 --- a/docs/manifest.json +++ b/docs/manifest.json @@ -481,9 +481,17 @@ { "title": "External Provisioners", "description": "Learn how to run external provisioners with Coder", - "path": "./admin/provisioners.md", + "path": "./admin/provisioners/index.md", "icon_path": "./images/icons/key.svg", - "state": ["enterprise", "premium"] + "state": ["enterprise", "premium"], + "children": [ + { + "title": "Manage Provisioner Jobs", + "description": "Learn how to run external provisioners with Coder", + "path": "./admin/provisioners/manage-provisioner-jobs.md", + "state": ["enterprise", "premium"] + } + ] }, { "title": "External Auth", diff --git a/docs/tutorials/best-practices/scale-coder.md b/docs/tutorials/best-practices/scale-coder.md index 9a640a051be58..9b248a6339692 100644 --- a/docs/tutorials/best-practices/scale-coder.md +++ b/docs/tutorials/best-practices/scale-coder.md @@ -141,7 +141,7 @@ maintenance window to minimize disruption. ### Locality We recommend that you run one or more -[provisioner daemon deployments external to Coder Server](../../admin/provisioners.md) +[provisioner daemon deployments external to Coder Server](../../admin/provisioners/index.md) and disable provisioner daemons within your Coder Server. This allows you to scale them independently of the Coder Server: diff --git a/docs/tutorials/best-practices/security-best-practices.md b/docs/tutorials/best-practices/security-best-practices.md index 7fc360616d302..c6f6cbe13a5c8 100644 --- a/docs/tutorials/best-practices/security-best-practices.md +++ b/docs/tutorials/best-practices/security-best-practices.md @@ -161,7 +161,7 @@ provision: ### Authentication -1. Use a [scoped key](../../admin/provisioners.md#scoped-key-recommended) to +1. Use a [scoped key](../../admin/provisioners/index.md#scoped-key-recommended) to authenticate the provisioner daemons with Coder. These keys can only be used to authenticate provisioner daemons (not other APIs on the Coder Server). diff --git a/docs/tutorials/best-practices/speed-up-templates.md b/docs/tutorials/best-practices/speed-up-templates.md index 046e00c8c65cb..91e885d27dc39 100644 --- a/docs/tutorials/best-practices/speed-up-templates.md +++ b/docs/tutorials/best-practices/speed-up-templates.md @@ -83,7 +83,7 @@ config option. You risk overloading Coder if you use too many built-in provisioners, so we recommend a maximum of five built-in provisioners per `coderd` replica. For more than five provisioners, we recommend that you move to -[External Provisioners](../../admin/provisioners.md) and also consider +[External Provisioners](../../admin/provisioners/index.md) and also consider [High Availability](../../admin/networking/high-availability.md) to run multiple `coderd` replicas. @@ -165,4 +165,4 @@ directory. Ensure that this directory is set to a location on disk which will persist across restarts of Coder or -[external provisioners](../../admin/provisioners.md), if you're using them. +[external provisioners](../../admin/provisioners/index.md), if you're using them.