You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+34-23Lines changed: 34 additions & 23 deletions
Original file line number
Diff line number
Diff line change
@@ -91,21 +91,27 @@ The action currently has two fetching modes:
91
91
92
92
These two modes are exclusive; you cannot mix settings for these two in one Flat step for a workflow job.
93
93
94
-
### `outfile_basename`
94
+
### HTTP Mode
95
95
96
-
This value is used as the basis for the file created to store data fetched by Flat. An extension is appended to this value at runtime to determine the actual filename. The behavior depends on Flat's fetching mode:
96
+
#### `http_url`
97
97
98
-
- `http`
99
-
1. If the response headers contain a `content-disposition` header, determine an extension from that.
100
-
2. Otherwise, attempt to infer an extension based on a `content-type` header, if available.
101
-
3. Give up and use `outfile_basename` as the entire filename, without an extension.
102
-
- `sql`: the extension is determined on the basis of the `sql_format` input.
98
+
A URL from which to fetch data. Specifying this input puts Flat into `http` mode.
103
99
104
-
### `http_url`
100
+
This can be any endpoint: a json, csv, png, zip, xlsx, etc.
105
101
106
-
A URL from which to fetch data. Specifying this input puts Flat into `http` mode.
102
+
#### `downloaded_filename`
107
103
108
-
### `sql_connstring`
104
+
The name of the file to store data fetched by Flat.
105
+
106
+
In `http` mode this can be anything. This can be any endpoint: a json, csv, txt, png, zip, xlsx, etc. file
107
+
108
+
#### `postprocess` (optional)
109
+
110
+
A path to a local Deno javascript or typescript file for postprocessing the `downloaded_filename` file. Read more in the "Postprocessing section".
111
+
112
+
### SQL Mode
113
+
114
+
#### `sql_connstring`
109
115
110
116
A URI-style database connection string. Flat will use this connection string to connect to the database and issue the query.
111
117
@@ -119,15 +125,22 @@ A URI-style database connection string. Flat will use this connection string to
119
125
>
120
126
> If you're using the [flat-vscode extension](https://github.com/githubocto/flat-vscode), this is handled for you.
121
127
122
-
### `sql_format`
123
128
124
-
One of `csv` or `json`. SQL query results will be serialized to disk in the specified format. Defaults to `json`.
129
+
#### `sql_queryfile`
130
+
131
+
The pathname of the file containing the SQL query that will be issued to the database. Defaults to `.github/workflows/query.sql`. This path is relative to the root of your repo.
132
+
133
+
#### `downloaded_filename`
134
+
135
+
The name of the file to store data fetched by Flat.
136
+
137
+
In `sql` mode this should be one of `csv` or `json`. SQL query results will be serialized to disk in the specified format.
125
138
126
139
> ⚠️ While the JSON is not pretty-printed, CSV is often a more efficient serialization for tabular data.
127
140
128
-
### `sql_queryfile`
141
+
#### `postprocess` (optional)
129
142
130
-
The pathname of the file containing the SQL query that will be issued to the database. Defaults to `.github/workflows/query.sql`. This path is relative to the root of your repo.
143
+
A path to a local Deno javascript or typescript file for postprocessing the `downloaded_filename` file. Read more in the "Postprocessing section".
131
144
132
145
## Outputs
133
146
@@ -137,40 +150,38 @@ A signed number describing the number of bytes that changed in this run. If the
137
150
138
151
## Postprocessing
139
152
140
-
### `postprocess`
141
-
142
-
This is the path to a [deno](https://deno.land) script that will be invoked to postprocess your data after it is fetched. This path is relative to the root of your repo.
153
+
You can add a `postprocess` input in the Action which is path to a [deno](https://deno.land) Javascript or Typescript script that will be invoked to postprocess your data after it is fetched. This path is relative to the root of your repo.
143
154
144
-
The script will be invoked with the path to the file that was fetched, and it **must** output, as its last line of output, the name of the file to use after postprocessing. Her
106D1
e's a simple postprocessing example:
155
+
The script can use either `Deno.args[0]` or the name of the `downloaded_filename` to access the file fetched by Flat Data.
145
156
146
157
```ts
147
158
import { readJSON, writeJSON } from 'https://deno.land/x/flat/mod.ts'
148
159
149
160
// The filename is the first invocation argument
150
-
const filename = Deno.args[0]
161
+
const filename = Deno.args[0] // Same name as downloaded_filename
151
162
const data = await readJSON(filename)
152
163
153
164
// Pluck a specific key off
154
165
// and write it out to a different file
155
166
// Careful! any uncaught errors and the workflow will fail, committing nothing.
156
167
const newfile = `subset_of_${filename}`
157
168
await writeJSON(newfile, data.path.to.something)
158
-
console.log(newfile)
159
169
```
160
170
161
-
You can write to `stdout` or use `console.log()` as much as you like within your postprocessing script; the results should show up in your actions log. The only hard requirement is that the last line must contain the path to the processed file, and nothing else.
171
+
You can use `console.log()` as much as you like within your postprocessing script; the results should show up in your actions log.
162
172
163
173
### Why deno?
164
174
165
175
Deno's import-by-url model makes it easy to author lightweight scripts that can include dependencies without forcing you to set up a bundler.
166
176
167
177
### How is my script invoked?
168
178
169
-
The postprocessing script is invoked with `deno run -q -A {your script} {your fetched data file}`. Note that the `-A` grants your script full permissions to access network, disk — everything! Make sure you trust any dependencies you pull in, as they aren't restricted. We will likely revisit this in the future with another setting that specifies which permissions to grant deno.
179
+
The postprocessing script is invoked with `deno run -q -A --unstable {your script} {your fetched data file}`. Note that the `-A` grants your script full permissions to access network, disk — everything! Make sure you trust any dependencies you pull in, as they aren't restricted. We will likely revisit this in the future with another setting that specifies which permissions to grant deno.
170
180
171
181
### How do I do ...?
172
182
173
-
We're putting together a collection of helpers at [deno.land/x/flat](https://deno.land/x/flat)
183
+
The learn more about the possibilities for postprocessing check out our [helper and examples postprocessing repo](https://github.com/githubocto/flat-postprocessing).
0 commit comments