gh-132631: Fix "I/O operation on closed file" when parsing JSON Lines file #132632
Conversation
| @@ -55,7 +55,7 @@ def main(): | |||
| infile = open(options.infile, encoding='utf-8') | |||
| try: | |||
| if options.json_lines: | |||
| objs = (json.loads(line) for line in infile) | |||
| objs = tuple(json.loads(line) for line in infile) | |||
There was a problem hiding this comment.
This has significant memory implications for large .jsonl files. It might be better to instead move the finally clause either to the outer try or just after the with outfile block, but that does have implications for the infile == outfile case.
There was a problem hiding this comment.
Hmm, yeah, moving it gives a blank file when infile == outfile...
There was a problem hiding this comment.
Or we could load all the lines into a list. This uses more memory than before, but less than the tuple as we're only deserialising each line when consumed from the generator:
| objs = tuple(json.loads(line) for line in infile) | |
| lines = infile.readlines() | |
| objs = (json.loads(line) for line in lines) |
I don't think it's worth adding much extra complexity for this CLI, as if someone needs to process extremely large .jsonl files they can write their own code to do exactly what they need.
|
All commit authors signed the Contributor License Agreement. |
Co-authored-by: Brian Schubert <brianm.schubert@gmail.com>
Before
After