8000 Introduce Markdown readers by adamsitnik · Pull Request #6969 · dotnet/extensions · GitHub
[go: up one dir, main page]
Skip to content

Introduce Markdown readers#6969

Merged
adamsitnik merged 17 commits intodotnet:mainfrom
adamsitnik:markdownReaders
Oct 31, 2025
Merged

Introduce Markdown readers#6969
adamsitnik merged 17 commits intodotnet:mainfrom
adamsitnik:markdownReaders

Conversation

@adamsitnik
Copy link
Member
@adamsitnik adamsitnik commented Oct 27, 2025 •
edited by dotnet-policy-service bot
Loading
Microsoft Reviewers: Open in CodeFlow

Copilot AI review requested due to automatic review settings October 27, 2025 16:43
@adamsitnik adamsitnik requested a review from a team as a code owner October 27, 2025 16:43
Copilot AI reviewed Oct 27, 2025
View reviewed changes
Copy link
Contributor
Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces Markdown reading capabilities to the data ingestion library through two new readers: MarkdownReader for native Markdown files and MarkItDownReader that leverages the external MarkItDown tool to convert various document formats to Markdown before parsing.

Key changes:

  • Adds MarkdownReader for parsing .md files using the Markdig library
  • Adds MarkItDownReader that wraps the MarkItDown CLI tool to convert documents (PDF, DOCX, etc.) to Markdown
  • Introduces shared MarkdownParser to parse Markdig AST into IngestionDocument model
  • Implements comprehensive test suite with conformance tests and format-specific test cases

Reviewed Changes

Copilot reviewed 12 out of 19 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
MarkdownReader.cs Implements reader for native Markdown files using Markdig parser
MarkdownParser.cs Core parsing logic converting Markdig AST to IngestionDocument model
MarkItDownReader.cs Wraps MarkItDown CLI tool to convert various document formats to Markdown
Microsoft.Extensions.DataIngestion.Markdown.csproj Project file for MarkdownReader with Markdig dependency
Microsoft.Extensions.DataIngestion.MarkItDown.csproj Project file for MarkItDownReader, shares MarkdownParser code
DocumentReaderConformanceTests.cs Base test class defining conformance tests for document readers
MarkdownReaderTests.cs Tests specific to MarkdownReader functionality
MarkItDownReaderTests.cs Tests specific to MarkItDownReader with CLI availability checks
ArrayUtils.cs Test utility for mapping 2D arrays used in table assertions
Microsoft.Extensions.DataIngestion.Tests.csproj Updated project file adding references and test file configuration
General.props Adds Markdig package reference
Versions.props Specifies Markdig version 0.42.0

@adamsitnik adamsitnik requested review from cincuranet and roji October 29, 2025 12:54
@ericstj ericstj requested a review from Copilot October 30, 2025 01:15
Copilot AI reviewed Oct 30, 2025
View reviewed changes
Copy link
Contributor
Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 13 out of 20 changed files in this pull request and generated 3 comments.

cincuranet
cincuranet reviewed Oct 30, 2025
View reviewed changes
@adamsitnik
Merge remote-tracking branch 'upstream/main' into markdownReaders
c01c970 
# Conflicts:
#	test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Microsoft.Extensions.DataIngestion.Tests.csproj
@adamsitnik
address code review feedback:
19ece18 
- delete temporary file when .CopyToAsync fails
- handle all image types
@adamsitnik adamsitnik requested a review from cincuranet October 30, 2025 15:44
@adamsitnik adamsitnik requested a review from ericstj October 30, 2025 16:03
This was referenced Nov 26, 2025
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Merged
Closed
Closed
Closed
Merged
Closed
Closed
Closed
Closed
Closed
Closed
Merged
Closed
Closed
Merged
Closed
Closed
@github-actions github-actions bot locked and limited conversation to collaborators Dec 1, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@stephentoub stephentoub stephentoub left review comments

Copilot code review Copilot Copilot left review comments

@cincuranet cincuranet cincuranet approved these changes

@roji roji Awaiting requested review from roji

@ericstj ericstj Awaiting requested review from ericstj

Assignees

@adamsitnik adamsitnik

Labels

area-telemetry

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@adamsitnik @roji @stephentoub @cincuranet @ericstj
0