10000 feat(core): receive query result using async iterable by sranka · Pull Request #592 · influxdata/influxdb-client-js · GitHub
[go: up one dir, main page]
Skip to content

feat(core): receive query result using async iterable #592

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 36 commits into from
Oct 10, 2022
Merged

feat(core): receive query result using async iterable #592

merged 36 commits into from
Oct 10, 2022

Conversation

sranka
Copy link
Contributor
@sranka sranka commented Sep 30, 2022 •
edited
Loading

Fixes #581

The main goal of this PR is to allow developers to consume query results using async iterable:

for await (const {values, tableMeta} of queryApi.iterateRows(fluxQuery)){
   // consume a particular row at will 
  console.log(JSON.stringify(tableMeta.toObject(values)))
}

All the details are demonstrated in updated examples. The present approach (with an observable) is still valid in many use cases, it is however more verbose and a bit harder to use in an asynchronous context.

Proposed Changes

  • allow receiving query results using async iterable (for-await loop)
    • enhance transport interface with an 'iterate' member function that returns async iterable of UInt8Array buffers,
      implement it in both node.js and browser transport
      • the latest node v14 is required because of the maturity of the response iterable implementation
    • expose the possibility to use AbortSignal in both node and browser transport (for the sake of completeness)
    • enhance AnnotatedCSVResponse to return iterable of lines or rows
    • enhance QueryApi to return CSV results lines or rows using async iterable (new iterateLines and iterateRows method)
  • change examples to demonstrate usage of async iterable
    • examples recommend using node v16
    • query.ts example was reorganized a bit so that top-level await is avoided (not supported by esbuild yet)
    • queryWithParameters.mjs now uses async iterables in a top-level for-await loop, which is possible when executing directly with node.js
    • invokableScripts.mjs uses async iterable, it is thus a bit simpler
    • a few other cosmetic changes in other examples
      • follow-redirects.mjs now demosntrate a rowMapper function
      • index.html is indented and produces better console logs
  • typescript transpilation changed to produce EcmaScript2018 output, it is required to support async iterable and for-await loops, it was es2015 before this PR ...
    • es2018 is properly implemented these days in modern browsers, deno and node.js environments
    • the situation is explained in the release notes
    • CI also shifted to node v16 (lts version until 2022-10-18)

Checklist

  • CHANGELOG.md updated
  • Rebased/mergeable
  • A test has been added if appropriate
  • yarn test completes successfully
  • Commit messages are conventional

@codecov-commenter
Copy link
codecov-commenter commented Sep 30, 2022 •
edited
Loading

Codecov Report

Base: 100.00% // Head: 100.00% // No change to project coverage 👍

Coverage data is based on head (cbe6076) compared to base (bc9a19c).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files 
@@            Coverage Diff             @@
##            master      #592    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files           33        35     +2     
  Lines         1333      1446   +113     
  Branches       323       344    +21     
==========================================
+ Hits          1333      1446   +113
Impacted Files Coverage Δ
packages/core/src/results/chunksToLines.ts 100.00% <ø> (ø)
packages/core/src/results/linesToTables.ts 100.00% <ø> (ø)
packages/core/src/InfluxDB.ts 100.00% <100.00%> (ø)
packages/core/src/impl/QueryApiImpl.ts 100.00% <100.00%> (ø)
packages/core/src/impl/browser/FetchTransport.ts 100.00% <100.00%> (ø)
packages/core/src/impl/node/NodeHttpTransport.ts 100.00% <100.00%> (ø)
...kages/core/src/results/AnnotatedCSVResponseImpl.ts 100.00% <100.00%> (ø)
packages/core/src/results/chunksToLinesIterable.ts 100.00% <100.00%> (ø)
packages/core/src/results/index.ts 100.00% <100.00%> (ø)
packages/core/src/results/linesToRowsIterable.ts 100.00% <100.00%> (ø)
... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@sranka sranka added this to the 1.32.0 milestone Oct 4, 2022
@sranka sranka force-pushed the feat/iterate_results branch from 01cd744 to 1a5553e Compare October 4, 2022 13:17
@sranka sranka requested review from vlastahajek and bednar October 4, 2022 13:18
@sranka sranka marked this pull request as ready for review October 4, 2022 13:18
@sranka sranka changed the title feat(core): receive query result using async iterables feat(core): receive query result using async iterable Oct 4, 2022
vlastahajek
vlastahajek reviewed Oct 4, 2022
View reviewed changes
@sranka sranka force-pushed the feat/iterate_results branch from 9e5d9c1 to 0cf8cc1 Compare October 4, 2022 16:47
@sranka sranka requested a review from vlastahajek October 4, 2022 16:50
vlastahajek
vlastahajek reviewed Oct 4, 2022
View reviewed changes
vlastahajek
vlastahajek reviewed Oct 4, 2022
View reviewed changes
vlastahajek
vlastahajek approved these changes Oct 4, 2022 10000
View reviewed changes
Copy link
Contributor
@vlastahajek vlastahajek left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome improvements! 👏

@sranka sranka self-assigned this Oct 5, 2022
@sranka sranka mentioned this pull request Oct 5, 2022
Closed
bednar
bednar requested changes Oct 5, 2022
View reviewed changes
Copy link
Contributor
@bednar bednar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your PR 👍 There is only minor requirement:

@sranka sranka requested a review from bednar October 5, 2022 07:54
bednar
bednar approved these changes Oct 5, 2022
View reviewed changes
Copy link
Contributor
@bednar bednar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🍻

sranka added 24 commits October 5, 2022 11:10
@sranka
feat(core): reuse returned row to optimize code
04d2899
@sranka
feat(core): allow to process annotated CSV response as iterable of li…
cfdfc40 
…nes or rows
@sranka
feat(core): add iterateLines and iterateRows to QueryApi
af7cf3b
@sranka
chore(examples): exclude mjs files in tsconfig
faddb98
@sranka
feat(examples): change deno example to use for-await loop
bbb4763
@sranka
feat(examples): use for-await to get query results in influxdb-1.8 ex…
fde1c8f 
…ample
@sranka
chore(examples): indent browser example
8e657df
@sranka
feat(examples): use for-await to print results in queryWithParams exa…
ef17017 
…mple
@sranka
chore(examples): improve influxdb-1.8 example
e388206
@sranka
feat(examples): receive results using async iterables in query.ts exa…
b3d17a1 
…mple
@sranka
feat(examples): demonstrate collectRows row mapper fn
200ddc8
@sranka
chore: update platform requirements
d9a8f69
@sranka
feat(examples): use async iterable in invokableScripts example
8291db4
@sranka
chore(ci): switch CI jobs to run with node 16
d36abbc
@sranka
chore(examples): improve docs in query.ts example
b3f55a9
@sranka
chore(example): target es2018 in examples
9ff67da
@sranka
chore(core): avoid code branching
79526e5
@sranka
chore(core): improve tsdoc
9624b8d
@sranka
chore: update changelog
d99d3aa
@sranka
chore(examples): repair comment
2075d83
@sranka
chore: update changelog
4877199
@sranka
chore: repair typos in tsdoc
218189a
@sranka
chore(core): improve tsdoc
43b7b92
@sranka
chore: improve description of a breaking change
cbe6076
@sranka sranka force-pushed the feat/iterate_results branch from dfcf4d1 to cbe6076 Compare October 5, 2022 09:14
@AzuladoToujours
Copy link
ABD9

Nice! Just what i was searching for. Thanks.

@sranka sranka merged commit 225f43f into master Oct 10, 2022
@sranka sranka deleted the feat/iterate_results branch October 10, 2022 05:47
@ivankudibal ivankudibal mentioned this pull request Oct 10, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bednar bednar bednar approved these changes

@vlastahajek vlastahajek vlastahajek approved these changes

Assignees

@sranka sranka

Labels
None yet
Projects
None yet
Milestone
1.32.0
Development

Successfully merging this pull request may close these issues.

Ability to query rows using async iterables
5 participants
@sranka @codecov-commenter @AzuladoToujours @bednar @vlastahajek
0