8000 feat: support stripping outer array in iterate_many by mariam851 · Pull Request #2587 · simdjson/simdjson · GitHub
[go: up one dir, main page]
Skip to content

feat: support stripping outer array in iterate_many #2587

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
mariam851 wants to merge 1 commit into
base: master
Choose a base branch
from
Open

feat: support stripping outer array in iterate_many #2587

mariam851 wants to merge 1 commit into from

Conversation

@mariam851
Copy link
@mariam851 mariam851 commented Jan 14, 2026

Hi Daniel (@lemire),

I hope you're doing well.

This PR addresses issue #2252 by enabling iterate_many to handle JSON documents wrapped in a top-level array.

As discussed in the issue, many streaming sources wrap their output in a large array. This change allows simdjson to process these documents efficiently without requiring the user to manually strip the brackets or wait for the entire array to be loaded.

Key Changes:

  • Automatic Array Stripping: Updated iterate_many to skip the leading whitespace and the initial [ character if present.
  • Improved Flexibility: The parser now detects the start of the array and automatically sets allow_comma_separated to true to handle the subsequent objects within the array.
  • Backward Compatibility: Existing iterate_many functionality for NDJSON/multiple concatenated objects remains unaffected.

Motivation

The goal is to provide a seamless "Out-of-the-box" experience for users dealing with array-wrapped streaming JSON, which is a common pattern in REST APIs.

I would appreciate your feedback on this implementation. I am a big admirer of your work on high-performance parsers and would be honored to contribute more to simdjson.

Best regards,
Mariam

@lemire
Copy link
Member
lemire commented Jan 14, 2026

I think that this does not work and it might derive from a misunderstanding as to what is sought. A reasonable newline separated JSON input might be the following:

[1,2,3]
[1,2,3]
[1,2,3]
[1,2,3]
...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mariam851 @lemire
0