10000 Update flat input parameters and general improvements · githubocto/flat@079aa8b · GitHub
[go: up one dir, main page]

Skip to content

Commit 079aa8b

Browse files
author
Irene Alvarado
committed
Update flat input parameters and general improvements
1 parent 15535ec commit 079aa8b

File tree

1 file changed

+34
-23
lines changed

1 file changed

+34
-23
lines changed

README.md

Lines changed: 34 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -91,21 +91,27 @@ The action currently has two fetching modes:
9191

9292
These two modes are exclusive; you cannot mix settings for these two in one Flat step for a workflow job.
9393

94-
### `outfile_basename`
94+
### HTTP Mode
9595

96-
This value is used as the basis for the file created to store data fetched by Flat. An extension is appended to this value at runtime to determine the actual filename. The behavior depends on Flat's fetching mode:
96+
#### `http_url`
9797

98-
- `http`
99-
1. If the response headers contain a `content-disposition` header, determine an extension from that.
100-
2. Otherwise, attempt to infer an extension based on a `content-type` header, if available.
101-
3. Give up and use `outfile_basename` as the entire filename, without an extension.
102-
- `sql`: the extension is determined on the basis of the `sql_format` input.
98+
A URL from which to fetch data. Specifying this input puts Flat into `http` mode.
10399

104-
### `http_url`
100+
This can be any endpoint: a json, csv, png, zip, xlsx, etc.
105101

106-
A URL from which to fetch data. Specifying this input puts Flat into `http` mode.
102+
#### `downloaded_filename`
107103

108-
### `sql_connstring`
104+
The name of the file to store data fetched by Flat.
105+
106+
In `http` mode this can be anything. This can be any endpoint: a json, csv, txt, png, zip, xlsx, etc. file
107+
108+
#### `postprocess` (optional)
109+
110+
A path to a local Deno javascript or typescript file for postprocessing the `downloaded_filename` file. Read more in the "Postprocessing section".
111+
112+
### SQL Mode
113+
114+
#### `sql_connstring`
109115

110116
A URI-style database connection string. Flat will use this connection string to connect to the database and issue the query.
111117

@@ -119,15 +125,22 @@ A URI-style database connection string. Flat will use this connection string to
119125
>
120126
> If you're using the [flat-vscode extension](https://github.com/githubocto/flat-vscode), this is handled for you.
121127

122-
### `sql_format`
123128

124-
One of `csv` or `json`. SQL query results will be serialized to disk in the specified format. Defaults to `json`.
129+
#### `sql_queryfile`
130+
131+
The pathname of the file containing the SQL query that will be issued to the database. Defaults to `.github/workflows/query.sql`. This path is relative to the root of your repo.
132+
133+
#### `downloaded_filename`
134+
135+
The name of the file to store data fetched by Flat.
136+
137+
In `sql` mode this should be one of `csv` or `json`. SQL query results will be serialized to disk in the specified format.
125138

126139
> ⚠️ While the JSON is not pretty-printed, CSV is often a more efficient serialization for tabular data.
127140

128-
### `sql_queryfile`
141+
#### `postprocess` (optional)
129142

130-
The pathname of the file containing the SQL query that will be issued to the database. Defaults to `.github/workflows/query.sql`. This path is relative to the root of your repo.
143+
A path to a local Deno javascript or typescript file for postprocessing the `downloaded_filename` file. Read more in the "Postprocessing section".
131144

132145
## Outputs
133146

@@ -137,40 +150,38 @@ A signed number describing the number of bytes that changed in this run. If the
137150

138151
## Postprocessing
139152

140-
### `postprocess`
141-
142-
This is the path to a [deno](https://deno.land) script that will be invoked to postprocess your data after it is fetched. This path is relative to the root of your repo.
153+
You can add a `postprocess` input in the Action which is path to a [deno](https://deno.land) Javascript or Typescript script that will be invoked to postprocess your data after it is fetched. This path is relative to the root of your repo.
143154

144-
The script will be invoked with the path to the file that was fetched, and it **must** output, as its last line of output, the name of the file to use after postprocessing. Her 106D1 e's a simple postprocessing example:
155+
The script can use either `Deno.args[0]` or the name of the `downloaded_filename` to access the file fetched by Flat Data.
145156

146157
```ts
147158
import { readJSON, writeJSON } from 'https://deno.land/x/flat/mod.ts'
148159
149160
// The filename is the first invocation argument
150-
const filename = Deno.args[0]
161+
const filename = Deno.args[0] // Same name as downloaded_filename
151162
const data = await readJSON(filename)
152163
153164
// Pluck a specific key off
154165
// and write it out to a different file
155166
// Careful! any uncaught errors and the workflow will fail, committing nothing.
156167
const newfile = `subset_of_${filename}`
157168
await writeJSON(newfile, data.path.to.something)
158-
console.log(newfile)
159169
```
160170

161-
You can write to `stdout` or use `console.log()` as much as you like within your postprocessing script; the results should show up in your actions log. The only hard requirement is that the last line must contain the path to the processed file, and nothing else.
171+
You can use `console.log()` as much as you like within your postprocessing script; the results should show up in your actions log.
162172

163173
### Why deno?
164174

165175
Deno's import-by-url model makes it easy to author lightweight scripts that can include dependencies without forcing you to set up a bundler.
166176

167177
### How is my script invoked?
168178

169-
The postprocessing script is invoked with `deno run -q -A {your script} {your fetched data file}`. Note that the `-A` grants your script full permissions to access network, disk — everything! Make sure you trust any dependencies you pull in, as they aren't restricted. We will likely revisit this in the future with another setting that specifies which permissions to grant deno.
179+
The postprocessing script is invoked with `deno run -q -A --unstable {your script} {your fetched data file}`. Note that the `-A` grants your script full permissions to access network, disk — everything! Make sure you trust any dependencies you pull in, as they aren't restricted. We will likely revisit this in the future with another setting that specifies which permissions to grant deno.
170180

171181
### How do I do ...?
172182

173-
We're putting together a collection of helpers at [deno.land/x/flat](https://deno.land/x/flat)
183+
The learn more about the possibilities for postprocessing check out our [helper and examples postprocessing repo](https://github.com/githubocto/flat-postprocessing).
184+
174185

175186
# Contributing
176187

0 commit comments

Comments
 (0)
0