Flat is a VSCode extension that steps you through the process of creating a Flat action, which makes it easy to fetch data and commit it to your repository as flatfiles.
Just as materialized views make it easier and faster to work the results of a query, Flat makes it easy to materialize data from anywhere into your workspace.
Flat streamlines a pattern popularized by Simon Willison and Alex Gaynor). The pattern of pulling data into git is an interesting one that deserves a dead-simple developer experience. Flat is an experiment from GitHub's Office of the CTO to make it easier for anyone to employ this pattern.
To use Flat, install the extension and click the Flat icon in the sidebar Tree View Container.
You will prompted to authenticate with GitHub (nb: Flat uses the user:email and repo scopes).
Once authenticated, Flat will either guide you through the steps for creating a Flat action, or list all workflow runs of said action.
To create an HTTP action, you'll be asked for the following inputs:
- A URL
- A CRON schedule
- A name
These inputs will be used to generate and commit a .github/workflows/flat.yaml file to your repository.
To create a SQL action, you'll be asked for the following inputs:
- A database connection string *
- A CRON schedule
- A format for saving your data (CSV or JSON)
- A name
* Note that we will encrypt this value and create a GitHub secret in your repository for this connection string. No sensitive data will be committed to your repository.
These inputs will be used to generate and commit a .github/workflows/flat.yaml file to your repository.
If Flat determines that you already have an action with runs, it will instead list these runs.
The action currently has two fetching modes:
http: GETs a supplied URLsql: Queries a SQL datastore
These two modes are exclusive; you cannot mix settings for these two in one Flat step for a workflow job.
This value is used as the basis for the file created to store data fetched by Flat. An extension is appended to this value at runtime to determine the actual filename. The behavior depends on Flat's fetching mode:
http- If the response headers contain a
content-dispositionheader, determine an extension from that. - Otherwise, attempt to infer an extension based on a
content-typeheader, if available. - Give up and use
outfile_basenameas the entire filename, without an extension.
- If the response headers contain a
sql: the extension is determined on the basis of thesql_formatinput.
A URL from which to fetch data. Specifying this input puts Flat into http mode.
A URI-style database connection string. Flat will use this connection string to connect to the database and issue the query.
⚠️ Don't write secrets into your workflow.yml!Most connection strings contain an authentication secret like a username and password. GitHub provides an encrypted vault for secrets like these which can be used by the action when it runs. Create a secret on the repository where the Flat action will run, and use that secret in your workflow.yaml like so:
sql_connstring: ${{NAME_OF_THE_CREATED_SECRET}}If you're using the flat-vscode extension, this is handled for you.
One of csv or json. SQL query results will be serialized to disk in the specified format. Defaults to json.
⚠️ While the JSON is not pretty-printed, CSV is often a more efficient serialization for tabular data.
The name of the file containing the SQL query that will be issued to the database. Defaults to query.sql, which must be placed in .github/workflows alongside the workflow.yml.
A signed number describing the number of bytes that changed in this run. If the new data is smaller than the existing, committed data, this will be a negative number.