Pipeline Forking and Dry Runs #4403
-
|
First off, fantastic work on this project. I'm currently exploring possibilities around how we might go about efficiently handling some of these use cases. Keen to hear if anyone has worked out some patterns for these kind of scenarios.
Apologies if some of these things are basic to implement, I'm still working through the documentation. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
Thanks for the kinds words @patrickgalbraith! The Enterprise edition has features like checkpointing, sync to S3 and mechanisms to start new pipelines off of other pipelines' checkpoints (backfill avoidance, experimental but actively worked on). Those are features you'd use to implement dry runs, blue green upgrades, and fork pipelines. Feel free to DM me on our community Slack if you'd like to do a deep dive there. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the reply. Glad to hear there are some options for this. I'll reach out when I'm a bit further along in. |
Beta Was this translation helpful? Give feedback.
Thanks for the kinds words @patrickgalbraith!
The Enterprise edition has features like checkpointing, sync to S3 and mechanisms to start new pipelines off of other pipelines' checkpoints (backfill avoidance, experimental but actively worked on). Those are features you'd use to implement dry runs, blue green upgrades, and fork pipelines.
Feel free to DM me on our community Slack if you'd like to do a deep dive there.