E52C Add Semantic Similarity chunker by KrystofS · Pull Request #6994 · dotnet/extensions · GitHub
[go: up one dir, main page]
Skip to content

Add Semantic Similarity chunker#6994

Merged
adamsitnik merged 21 commits intodotnet:mainfrom
KrystofS:feature/chunkers
Nov 5, 2025
Merged

Add Semantic Similarity chunker#6994
adamsitnik merged 21 commits intodotnet:mainfrom
KrystofS:feature/chunkers

Conversation

@KrystofS
Copy link
Contributor
@KrystofS KrystofS commented Oct 31, 2025 •
edited by dotnet-policy-service bot
Loading
Microsoft Reviewers: Open in CodeFlow

Copilot AI review requested due to automatic review settings October 31, 2025 19:25
@KrystofS
Copy link
Contributor Author
KrystofS commented Oct 31, 2025

@KrystofS please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@dotnet-policy-service agree [company="{your company}"]

Options:

  • (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@dotnet-policy-service agree
  • (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@dotnet-policy-service agree company="Microsoft"

Contributor License Agreement

@dotnet-policy-service agree

Copilot AI reviewed Oct 31, 2025
View reviewed changes
Copy link
Contributor
Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces new chunking implementations for the DataIngestion library, adding support for splitting documents into chunks based on different strategies: semantic similarity, sections, markdown structure, and token-based chunking. The key changes include:

  • New chunker implementations: SemanticSimilarityChunker, SectionChunker, MarkdownChunker, and DocumentTokenChunker
  • Comprehensive test coverage for all new chunkers
  • Addition of System.Numerics.Tensors package dependency for cosine similarity calculations
  • Addition of Microsoft.ML.Tokenizers.Data.O200kBase package for tokenization support
  • Test infrastructure with shared TestEmbeddingGenerator for testing AI-based chunking

Reviewed Changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
src/Libraries/Microsoft.Extensions.DataIngestion/Microsoft.Extensions.DataIngestion.csproj Adds System.Numerics.Tensors package dependency and removes trailing whitespace
src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/SemanticSimilarityChunker.cs Implements semantic similarity-based chunking using embeddings and cosine similarity
src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/SectionChunker.cs Implements section-based chunking that treats document sections as separate entities
src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/MarkdownChunker.cs Implements markdown header-based chunking with configurable header levels
src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/DocumentTokenChunker.cs Implements token-based chunking with configurable overlap
test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Microsoft.Extensions.DataIngestion.Tests.csproj Adds test dependencies and links to shared test helpers
test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/SemanticSimilarityChunkerTests.cs Comprehensive tests for semantic similarity chunker
test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/SectionChunkerTests.cs Tests for section-based chunking including nested sections
test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/MarkdownChunkerTests.cs Tests for markdown chunker with various header configurations
test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/DocumentTokenChunkerTests.cs Base tests for token-based chunking
test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/OverlapDocumentTokenChunkerTests.cs Tests for token chunking with overlap
test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/NoOverlapDocumentTokenChunkerTests.cs Tests for token chunking without overlap
test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/DocumentChunkerTests.cs Base test class with common test scenarios
test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/ChunkAssertions.cs Helper class for chunk assertions

adamsitnik
adamsitnik reviewed Nov 3, 2025
View reviewed changes
Copy link
Member
@adamsitnik adamsitnik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Big thanks for all your hard work and contribution to our product @KrystofS !

I've left some comments, PTAL.

adamsitnik
adamsitnik reviewed Nov 5, 2025
View reviewed changes
Copy link
Member
@adamsitnik adamsitnik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good to me. PTAL at my comments (mostly nits). Thank you again @KrystofS !

adamsitnik
adamsitnik approved these changes Nov 5, 2025
View reviewed changes
Copy link
Member
@adamsitnik adamsitnik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you for your contribution @KrystofS !

@adamsitnik adamsitnik changed the title Add new document chunkers Add Semantic Similarity chunker Nov 5, 2025
stephentoub
stephentoub reviewed Nov 5, 2025
View reviewed changes
KrystofS and others added 17 commits November 5, 2025 18:04
@KrystofS
Chunkers
b1c27a2
@KrystofS
add tests
761843f
@KrystofS
fix chunker tests
df3e02c
@KrystofS
Apply suggestions from code review
d9e05f2 
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@KrystofS
resolve suggestions from copilot
74177de
@KrystofS @adamsitnik
Update src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/Sema…
3793004 
…nticSimilarityChunker.cs

Co-authored-by: Adam Sitnik <adam.sitnik@gmail.com>
@KrystofS @adamsitnik
Apply suggestions from code review
7d7ee81 
Co-authored-by: Adam Sitnik <adam.sitnik@gmail.com>
@KrystofS
refactors based on review
f60bf3b
@KrystofS
fix tests
d214a95
@KrystofS
fix missed warning
476866e
@KrystofS
update semanctic similarity chunker documentation
d767031
@KrystofS
fix typo
91e26a6
@KrystofS
Update SemanticSimilarityChunker.cs
7107417
@KrystofS
remove chunkers from PR
ada19ef
@KrystofS
remove left over tests
d1d4028
@KrystofS
Delete ChunkAssertions.cs
d2d736a
@KrystofS @adamsitnik
Apply suggestions from code review
78afabb 
Co-authored-by: Adam Sitnik <adam.sitnik@gmail.com>
@adamsitnik adamsitnik enabled auto-merge (squash) November 5, 2025 20:19
@adamsitnik adamsitnik merged commit 192782e into dotnet:main Nov 5, 2025
6 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Dec 6, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@stephentoub stephentoub stephentoub left review comments

@adamsitnik adamsitnik adamsitnik approved these changes

Copilot code review Copilot Copilot left review comments

Assignees

@KrystofS KrystofS

Labels

area-data-ingestion

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@KrystofS @stephentoub @adamsitnik
0