8000 Search metadata by ManyTheFish · Pull Request #5926 · meilisearch/meilisearch · GitHub
[go: up one dir, main page]
Skip to content

Search metadata #5926

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Search metadata #5926

wants to merge 8 commits into from
+657 −43

Conversation

ManyTheFish
Copy link
Member
@ManyTheFish ManyTheFish commented Oct 2, 2025 •
edited
Loading

Overview

Added opt-in metadata to search responses for query traceability. Metadata is only included when the Meili-Include-Metadata header is present, ensuring backward compatibility.

API Changes

New Header

  • Header: Meili-Include-Metadata
  • Value: true or 1 to activate it
  • Default: Metadata not included (backward compatible)

New Response Fields

Single Search (/indexes/{indexUid}/search)

{
  "hits": [...],
  "metadata": {
    "queryUid": "0199a41a-8186-70b3-b6e1-90e8cb582f35",
    "indexUid": "movies",
    "primaryKey": "id"
  }
}

Multi-Search (/multi-search)

{
  "results": [
    {
      "indexUid": "movies",
      "hits": [...],
      "metadata": {
        "queryUid": "0199a41a-8186-70b3-b6e1-90e8cb582f35",
        "indexUid": "movies", 
        "primaryKey": "id"
      }
    }
  ]
}

Federated Search (/multi-search with federation)

{
  "hits": [...],
  "metadata": [
    {
      "queryUid": "0199a41a-8186-70b3-b6e1-90e8cb582f35",
      "indexUid": "movies",
      "primaryKey": "id"
    },
    {
      "queryUid": "0199a41a-8186-70b3-b6e1-90e8cb582f36", 
      "indexUid": "remote_movies",
      "primaryKey": "id",
      "remote": "remote-1"
    }
  ]
}

Metadata Fields

Field Type Description When Present
queryUid UUID v7 Unique identifier for the query Always
indexUid String Index identifier Always
primaryKey String Primary key field name None if the index has no primary key
remote String Remote instance name Remote queries only

Usage Examples

Basic Search with Metadata

Request: 
curl -X POST 'http://localhost:7700/indexes/movies/search' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer masterKey' \
  -H 'Meili-Include-Metadata: true' \
  -d '{"q": "batman"}'
Response: 
{
  "hits": [
    {
      "id": 1,
      "title": "Batman Begins",
      "overview": "After training with his mentor, Batman begins his fight to free crime-ridden Gotham City from corruption."
    }
  ],
  "query": "batman",
  "processingTimeMs": 2,
  "limit": 20,
  "offset": 0,
  "estimatedTotalHits": 1,
  "requestUid": "0199a41a-60e2-7b91-9843-9ff0f0e36a3d",
  "metadata": {
    "queryUid": "0199a41a-8186-70b3-b6e1-90e8cb582f35",
    "indexUid": "movies",
    "primaryKey": "id"
  }
}

Multi-Search with Metadata

Request: 
curl -X POST 'http://localhost:7700/multi-search' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer masterKey' \
  -H 'Meili-Include-Metadata: true' \
  -d '{
    "queries": [
      {"indexUid": "movies", "q": "batman"},
      {"indexUid": "books", "q": "harry"}
    ]
  }'
Response: 
{
  "results": [
    {
      "indexUid": "movies",
      "hits": [
        {
          "id": 1,
          "title": "Batman Begins",
          "overview": "After training with his mentor, Batman begins his fight to free crime-ridden Gotham City from corruption."
        }
      ],
      "query": "batman",
      "processingTimeMs": 0,
      "limit": 20,
      "offset": 0,
      "estimatedTotalHits": 1,
      "requestUid": "0199a41a-cbf3-78e1-a04a-a93c3cca0f06",
      "metadata": {
        "queryUid": "0199a41a-cbf4-73d1-a80a-0aa43823cae9",
        "indexUid": "movies",
        "primaryKey": "id"
      }
    },
    {
      "indexUid": "books",
      "hits": [],
      "query": "harry",
      "processingTimeMs": 0,
      "limit": 20,
      "offset": 0,
      "estimatedTotalHits": 0,
      "requestUid": "0199a41a-cbf3-78e1-a04a-a93c3cca0f06",
      "metadata": {
        "queryUid": "0199a41a-cbf4-73d1-a80a-0ab74a5eae99",
        "indexUid": "books",
        "primaryKey": "id"
      }
    }
  ]
}

Federated Search with Metadata

Request: 
curl -X POST 'http://localhost:7700/multi-search' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer masterKey' \
  -H 'Meili-Include-Metadata: true' \
  -d '{
    "queries": [
      {"indexUid": "movies", "q": "batman"},
      {"indexUid": "remote_movies", "q": "matrix", "federationOptions": {"remote": "remote-1"}}
    ],
    "federation": {"limit": 5}
  }'
Response: 
{
  "hits": [
    {
      "id": 1,
      "title": "Batman Begins",
      "overview": "After training with his mentor, Batman begins his fight to free crime-ridden Gotham City from corruption.",
      "_federation": {
        "indexUid": "movies",
        "queriesPosition": 0,
        "weightedRankingScore": 0.9848484848484849,
        "remote": null
      }
    },
    {
      "id": 101,
      "title": "The Matrix",
      "overview": "A computer hacker learns from mysterious rebels about the true nature of his reality and his role in the war against its controllers.",
      "_federation": {
        "indexUid": "remote_movies",
        "queriesPosition": 1,
        "weightedRankingScore": 0.9242424242424242,
        "remote": "remote-1"
      }
    }
  ],
  "processingTimeMs": 256,
  "limit": 5,
  "offset": 0,
  "estimatedTotalHits": 2,
  "requestUid": "0199a428-54d6-7a73-91bb-cb2f7e8c2d5a",
  "metadata": [
    {
      "queryUid": "0199a428-56d2-7873-a505-f88093d8d68e",
      "indexUid": "movies",
      "primaryKey": "id"
    },
    {
      "queryUid": "0199a428-56d2-7873-a505-f89683a44df9",
      "indexUid": "remote_movies",
      "primaryKey": "id",
      "remote": "remote-1"
    }
  ],
  "remoteErrors": {}
}

Implementation Notes

  • Backward Compatible: No metadata included by default
  • Header Forwarding: Remote requests automatically forward the header
  • Query Ordering: Federated search metadata array matches query order

< 8000 path d="M15 8a7.002 7.002 0 00-7-7" stroke="currentColor" stroke-width="2" stroke-linecap="round" vector-effect="non-scaling-stroke" />

@ManyTheFish ManyTheFish marked this pull request as draft October 2, 2025 09:39
@ManyTheFish
feat: add metadata field with queryUid to search responses
194ace6 
- Add SearchMetadata struct with queryUid field (UUID v7)
- Add metadata field to SearchResult for /search route
- Add metadata field to FederatedSearchResult for /multi-search route
- Update perform_search to generate queryUid and set metadata
- Update federated search to generate queryUid for each query
- Update multi-search non-federated path to include metadata
- Fix pattern matching in analytics and other code

The metadata field contains:
- For /search: single object with queryUid
- For /multi-search: array of objects, one per query
- For federated search: array of objects, one per query

All queryUid values are generated using Uuid::now_v7() for time-ordered uniqueness.
@ManyTheFish
feat: add indexUid to SearchMetadata
b23f66d 
- Add indexUid field to SearchMetadata struct
- Update perform_search to include indexUid in metadata
- Update federated search to include indexUid for each query

The metadata field now contains both queryUid and indexUid:
- For /search: single object with queryUid and indexUid
- For /multi-search: each result has metadata with both fields
- For federated search: array of objects, each with queryUid and indexUid
@ManyTheFish
WIP: Add metadata field with queryUid, indexUid, primaryKey, and remote
5205393 
- Add SearchMetadata struct with queryUid, indexUid, primaryKey, and remote fields
- Update SearchResult to include metadata field
- Update FederatedSearchResult to include metadata array
- Refactor federated search metadata building to maintain query order
- Support primary key extraction from both local and remote results
- Add remote field to identify remote instance for federated queries
- Ensure metadata array matches query order in federated search

Features:
- queryUid: UUID v7 for each query
- indexUid: Index identifier
- primaryKey: Primary key field name (null if not available)
- remote: Remote instance name (null for local queries)

This provides complete traceability for search operations across local and remote instances.
@ManyTheFish
Add header-based metadata opt-in for search responses
4b66c48 
- Add Meili-Include-Metadata header constant
- Modify perform_search to conditionally include metadata based on header
- Modify perform_federated_search to conditionally include metadata based on header
- Update all search routes to check for header and pass include_metadata parameter
- Forward Meili-Include-Metadata header to remote requests for federated search
- Ensure remote queries include primaryKey metadata when header is present
@ManyTheFish ManyTheFish added < 8000 a id="label-56452d" href="/meilisearch/meilisearch/issues?q=state%3Aopen%20label%3A%22no%20db%20change%22" data-name="no db change" style="--label-r:140;--label-g:252;--label-b:144;--label-h:122;--label-s:94;--label-l:76;" data-view-component="true" class="IssueLabel hx_IssueLabel d-inline-block v-align-middle"> no db change The database didn't change Analytics labels Oct 6, 2025
ManyTheFish
ManyTheFish commented Oct 6, 2025
View reviewed changes
@ManyTheFish
Fix remote index collision in federated search metadata
c754981 
- Use composite key (indexUid, remote) instead of indexUid only for remote metadata lookup
- Prevents collisions when multiple remotes have same indexUid but different primary keys
- Ensures each remote query gets correct primaryKey from its specific remote instance
@ManyTheFish
Fix remote index collision with HashMap-based lookup
af6d53c 
- Replace BTreeMap with HashMap for (remote, index_uid) -> primary_key lookup
- Prevents collisions when multiple remotes have same index_uid but different primary keys
@ManyTheFish
refactor: group perform_search parameters into SearchParams struct
4f4dc47 
- Create SearchParams struct to group related parameters
- Update perform_search function to use SearchParams instead of 8 individual parameters
- Fix clippy warning about too many arguments
- Update all callers to use new SearchParams struct
@ManyTheFish
Add comprehensive metadata tests with insta snapshots
a0ef800 
- Add 9 test cases covering single search, multi-search, and federated search
- Test metadata header opt-in functionality with case insensitivity
- Test header false value handling
- Test UUID format validation and consistency
- Use insta snapshots for reliable, maintainable test assertions
- Fix header parsing to properly handle 'false' values
- Add helper methods for testing with custom headers
@ManyTheFish ManyTheFish marked this pull request as ready for review October 7, 2025 09:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
Analytics no db change The database didn't change
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@ManyTheFish
0