We'll eventually have an implementation of Buffer, which will allow us to read binary files and process them in chunks from JavaScript.

Until that's available, I think there are two alternatives available at least for processing CSV and JSON files.
They should work incrementally and process the input file line-wise (not quite true for CSV but think of them as working line-wise). They allow supplying a callback function that is executed whenever a record was read. You can then use the callback to process the data and put it into the database.

Example invocation for CSV files:

var internal = require("internal");
// var options = { separator: ",", quote: "\"" };
internal.processCsvFile("test.csv", function (data) { internal.print(data); }, options );

And for p```
var internal = require("internal");
internal.processCsvFile("test.csv", function (data) { internal.print(data); } );


Processing JSON files is similar:

var internal = require("internal");
internal.processJsonFile("test.json", function (data) { internal.print(data); } );


Note that the above function aren't general purpose file-processing functions, but targeted for handling UTF-8 encoded CSV and JSON data. 
For arbitrary file formats, we'll need an implementation of Buffer in Javascript.

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions