8000 feat: implement expiration policy logic for prebuilds by ssncferreira · Pull Request #17996 · coder/coder · GitHub
[go: up one dir, main page]

Skip to content

feat: implement expiration policy logic for prebuilds #17996

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
May 26, 2025
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: update code according to latest terraform schema expiration_poli…
…cy.ttl
  • Loading branch information
ssncferreira committed May 23, 2025
commit c7e442ca8b3e054c00a5e7cb184444caa0a8d8fa
6 changes: 3 additions & 3 deletions coderd/database/queries.sql.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion coderd/database/queries/prebuilds.sql
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ SELECT
tvp.id,
tvp.name,
tvp.desired_instances AS desired_instances,
tvp.invalidate_after_secs AS invalidate_after_secs,
tvp.invalidate_after_secs AS ttl,
tvp.prebuild_status,
t.deleted,
t.deprecated != '' AS deprecated
Expand Down
8 changes: 4 additions & 4 deletions coderd/prebuilds/global_snapshot.go
Original file line number Diff line number Diff line change
Expand Up @@ -81,14 +81,14 @@ func (s GlobalSnapshot) FilterByPreset(presetID uuid.UUID) (*PresetSnapshot, err
}

// filterExpiredWorkspaces splits running workspaces into expired and non-expired
// based on the preset's InvalidateAfterSecs TTL. If TTL is missing or zero,
// all workspaces are considered non-expired.
// based on the preset's TTL.
// If TTL is missing or zero, all workspaces are considered non-expired.
func filterExpiredWorkspaces(preset database.GetTemplatePresetsWithPrebuildsRow, runningWorkspaces []database.GetRunningPrebuiltWorkspacesRow) (nonExpired []database.GetRunningPrebuiltWorkspacesRow, expired []database.GetRunningPrebuiltWorkspacesRow) {
Copy link
Contributor Author
@ssncferreira ssncferreira May 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There’s an alternative to perform this filtering directly in the database, but for now, opted for an in-memory approach to keep the implementation simpler and the PR smaller. The main trade-off here is performance at scale, while in-memory filtering is sufficient for the current scale, we may want to revisit a DB-based solution if we encounter issues at higher scale. Let me know if you think a database solution would be a better fit now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand this correctly, the un-filtered data still comes directly from the database but is filtered in-memory.

From what I can tell, the changes required to move this logic inside the database would involve passing in the preset TTL and doing the TTL calculation inside the database. Is that correct?

If so, I agree that it can be kept in-memory for the moment, but we should add a follow-up issue to track this.

Copy link
Contributor Author
@ssncferreira ssncferreira May 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand this correctly, the un-filtered data still comes directly from the database but is filtered in-memory.

Yes, that's correct, the filtering currently happens in-memory after fetching all the running prebuilds (expired and non-expired) from the database.

From what I can tell, the changes required to move this logic inside the database would involve passing in the preset TTL and doing the TTL calculation inside the database. Is that correct?

Right, moving it into the database would require doing the TTL check within the query. We don’t need to pass the preset TTL manually since it's already stored in the database, so the query can join against the template_version_presets table and compute the expiration inline.
We’d also need to update the current GetRunningPrebuiltWorkspacesRow function to return only the non-expired running prebuilds.

If so, I agree that it can be kept in-memory for the moment, but we should add a follow-up issue to track this.

I’ll create a follow-up issue to track moving it into the database for better efficiency/scalability 👍

if !preset.InvalidateAfterSecs.Valid {
if !preset.Ttl.Valid {
return runningWorkspaces, expired
}

ttl := time.Duration(preset.InvalidateAfterSecs.Int32) * time.Second
ttl := time.Duration(preset.Ttl.Int32) * time.Second
if ttl <= 0 {
return runningWorkspaces, expired
}
Expand Down
6 changes: 3 additions & 3 deletions coderd/prebuilds/preset_snapshot_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -670,14 +670,14 @@ func TestExpiredPrebuilds(t *testing.T) {
// GIVEN: running prebuilt workspaces for the preset.
running := make([]database.GetRunningPrebuiltWorkspacesRow, 0, tc.running)
expiredCount := 0
cacheInvalidationDuration := time.Duration(defaultPreset.InvalidateAfterSecs.Int32)
ttlDuration := time.Duration(defaultPreset.Ttl.Int32)
for range tc.running {
name, err := prebuilds.GenerateName()
require.NoError(t, err)
prebuildCreateAt := time.Now()
if int(tc.expired) > expiredCount {
// Update the prebuild workspace createdAt to exceed its TTL
prebuildCreateAt = prebuildCreateAt.Add(-cacheInvalidationDuration - 2*time.Second)
prebuildCreateAt = prebuildCreateAt.Add(-ttlDuration - 2*time.Second)
expiredCount++
}
running = append(running, database.GetRunningPrebuiltWorkspacesRow{
Expand Down Expand Up @@ -937,7 +937,7 @@ func preset(active bool, instances int32, opts options, muts ...func(row databas
},
Deleted: false,
Deprecated: false,
InvalidateAfterSecs: sql.NullInt32{
Ttl: sql.NullInt32{
Valid: true,
Int32: 2,
},
Expand Down
10 changes: 5 additions & 5 deletions coderd/provisionerdserver/provisionerdserver.go
Original file line number Diff line number Diff line change
Expand Up @@ -2059,15 +2059,15 @@ func InsertWorkspacePresetsAndParameters(ctx context.Context, logger slog.Logger

func InsertWorkspacePresetAndParameters(ctx context.Context, db database.Store, templateVersionID uuid.UUID, protoPreset *sdkproto.Preset, t time.Time) error {
err := db.InTx(func(tx database.Store) error {
var desiredInstances, invalidateAfterSecs sql.NullInt32
var desiredInstances, ttl sql.NullInt32
if protoPreset != nil && protoPreset.Prebuild != nil {
desiredInstances = sql.NullInt32{
Int32: protoPreset.Prebuild.Instances,
Valid: true,
}
if protoPreset.Prebuild.CacheInvalidation != nil {
invalidateAfterSecs = sql.NullInt32{
Int32: protoPreset.Prebuild.CacheInvalidation.InvalidateAfterSecs,
if protoPreset.Prebuild.ExpirationPolicy != nil {
ttl = sql.NullInt32{
Int32: protoPreset.Prebuild.ExpirationPolicy.Ttl,
Valid: true,
}
}
Expand All @@ -2078,7 +2078,7 @@ func InsertWorkspacePresetAndParameters(ctx context.Context, db database.Store,
Name: protoPreset.Name,
CreatedAt: t,
DesiredInstances: desiredInstances,
InvalidateAfterSecs: invalidateAfterSecs,
InvalidateAfterSecs: ttl,
})
if err != nil {
return xerrors.Errorf("insert preset: %w", err)
Expand Down
18 changes: 9 additions & 9 deletions docs/admin/templates/extending-templates/prebuilt-workspaces.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ Prebuilt workspaces are tightly integrated with [workspace presets](./parameters
## Enable prebuilt workspaces for template presets

In your template, add a `prebuilds` block within a `coder_workspace_preset` definition to identify the number of prebuilt
instances your Coder deployment should maintain, and optionally configure a `cache_invalidation` block to set a TTL
instances your Coder deployment should maintain, and optionally configure a `expiration_policy` block to set a TTL
(Time To Live) for unclaimed prebuilt workspaces to ensure stale resources are automatically cleaned up.

```hcl
Expand All @@ -43,9 +43,9 @@ instances your Coder deployment should maintain, and optionally configure a `cac
memory = 16
}
prebuilds {
instances = 3 # Number of prebuilt workspaces to maintain
cache_invalidation {
invalidate_after_secs = 86400 # Time (in seconds) after which unclaimed prebuilds are expired (1 day)
instances = 3 # Number of prebuilt workspaces to maintain
expiration_policy {
ttl = 86400 # Time (in seconds) after which unclaimed prebuilds are expired (1 day)
}
}
}
Expand All @@ -54,8 +54,8 @@ instances your Coder deployment should maintain, and optionally configure a `cac
After you publish a new template version, Coder will automatically provision and maintain prebuilt workspaces through an
internal reconciliation loop (similar to Kubernetes) to ensure the defined `instances` count are running.

The `cache_invalidation` block ensures that any prebuilt workspaces left unclaimed for more than `invalidate_after_secs`
seconds is considered expired and automatically cleaned up.
The `expiration_policy` block ensures that any prebuilt workspaces left unclaimed for more than `ttl` seconds is considered
expired and automatically cleaned up.

## Prebuilt workspace lifecycle

Expand Down Expand Up @@ -102,10 +102,10 @@ Unclaimed prebuilt workspaces can be interacted with in the same way as any othe
However, if a Prebuilt workspace is stopped, the reconciliation loop will not destroy it.
This gives template admins the ability to park problematic prebuilt workspaces in a stopped state for further investigation.

### Cache Invalidation
### Expiration Policy

Prebuilt workspaces support cache invalidation through the `invalidate_after_secs` setting inside the `cache_invalidation`
block. This value defines the Time To Live (TTL) of a prebuilt workspace, i.e., the duration in seconds that an unclaimed
Prebuilt workspaces support expiration policies through the `ttl` setting inside the `expiration_policy` block.
This value defines the Time To Live (TTL) of a prebuilt workspace, i.e., the duration in seconds that an unclaimed
prebuilt workspace can remain before it is considered expired and eligible for cleanup.

Expired prebuilt workspaces are removed during the reconciliation loop to avoid stale environments and resource waste.
Expand Down
12 changes: 6 additions & 6 deletions provisioner/terraform/resources.go
Original file line number Diff line number Diff line change
Expand Up @@ -897,21 +897,21 @@
)
}
var prebuildInstances int32
var cacheInvalidation *proto.CacheInvalidation
var expirationPolicy *proto.ExpirationPolicy
if len(preset.Prebuilds) > 0 {
prebuildInstances = int32(math.Min(math.MaxInt32, float64(preset.Prebuilds[0].Instances)))
if len(preset.Prebuilds[0].CacheInvalidation) > 0 {
cacheInvalidation = &proto.CacheInvalidation{
InvalidateAfterSecs: int32(math.Min(math.MaxInt32, float64(preset.Prebuilds[0].CacheInvalidation[0].InvalidateAfterSecs))),
if len(preset.Prebuilds[0].ExpirationPolicy) > 0 {

Check failure on line 903 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / gen

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 903 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / offlinedocs

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 903 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / test-e2e

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 903 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / lint

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 903 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / lint

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 903 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / lint

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 903 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / test-go-pg-16

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 903 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / test-go-pg (ubuntu-latest)

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 903 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / test-go-pg (ubuntu-latest)

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 903 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / test-go (ubuntu-latest)

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 903 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / test-go-pg (macos-latest)

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 903 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / test-go (windows-2022)

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 903 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / test-go-race

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 903 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / test-go (macos-latest)

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)
expirationPolicy = &proto.ExpirationPolicy{
Ttl: int32(math.Min(math.MaxInt32, float64(preset.Prebuilds[0].ExpirationPolicy[0].TTL))),

Check failure on line 905 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / gen

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 905 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / offlinedocs

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 905 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / test-e2e

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 905 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / lint

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)) (typecheck)

Check failure on line 905 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / lint

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)) (typecheck)

Check failure on line 905 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / lint

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)) (typecheck)

Check failure on line 905 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / test-go-pg-16

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 905 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / test-go-pg (ubuntu-latest)

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 905 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / test-go-pg (ubuntu-latest)

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 905 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / test-go (ubuntu-latest)

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 905 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / test-go-pg (macos-latest)

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 905 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / test-go (windows-2022)

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 905 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / test-go-race

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)

Check failure on line 905 in provisioner/terraform/resources.go

View workflow job for this annotation

GitHub Actions / test-go (macos-latest)

preset.Prebuilds[0].ExpirationPolicy undefined (type provider.WorkspacePrebuild has no field or method ExpirationPolicy)
}
}
}
protoPreset := &proto.Preset{
Name: preset.Name,
Parameters: presetParameters,
Prebuild: &proto.Prebuild{
Instances: prebuildInstances,
CacheInvalidation: cacheInvalidation,
Instances: prebuildInstances,
ExpirationPolicy: expirationPolicy,
},
}

Expand Down
4 changes: 2 additions & 2 deletions provisioner/terraform/resources_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -830,8 +830,8 @@ func TestConvertResources(t *testing.T) {
}},
Prebuild: &proto.Prebuild{
Instances: 4,
CacheInvalidation: &proto.CacheInvalidation{
InvalidateAfterSecs: 86400,
ExpirationPolicy: &proto.ExpirationPolicy{
Ttl: 86400,
},
},
}},
Expand Down
4 changes: 2 additions & 2 deletions provisioner/terraform/testdata/resources/presets/presets.tf
76A1
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ data "coder_workspace_preset" "MyFirstProject" {
}
prebuilds {
instances = 4
cache_invalidation {
invalidate_after_secs = 86400
expiration_policy {
ttl = 86400
}
}
}
Expand Down
Loading
Loading
0