8000 NEW BACKEND! MongoDB Atlas by caseyclements · Pull Request #1883 · docarray/docarray · GitHub
[go: up one dir, main page]
Skip to content

NEW BACKEND! MongoDB Atlas #1883

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
JoanFM merged 37 commits into from
Apr 29, 2024
Merged

NEW BACKEND! MongoDB Atlas #1883

JoanFM merged 37 commits into from
Apr 29, 2024

Conversation

@caseyclements
Copy link
Contributor
@caseyclements caseyclements commented Apr 23, 2024

Description

This pull request introduces MongoDB integration as a document index backend, enhancing search capabilities within the application. Below are the details of the implementation and its supported functionalities:

Simple Usage

from docarray.index import MongoAtlasDocumentIndex
import numpy as np

class MyDoc(BaseDoc):
    text: str
    embedding: NdArray[10]

docs = [MyDoc(text=f'text {i}', embedding=np.random.rand(10)) for i in range(10)]
query = np.random.rand(10)
db = MongoAtlasDocumentIndex[MyDoc](host='localhost')
db.index(docs)
results = index.find(query, search_field='embedding', limit=10)

Supported Functionality

  • Find (vector search): Enables vector-based search.
  • Filter: Allows filtering on textual and numeric data using MongoDB syntax.
  • Text Search: Supports text search using regex match.
  • Get/Del: Retrieve and delete operations.
  • Subindex: Ability to create subindexes for better organization.

Integration Tests and documentation

  • tests/index/mongo_atlas
  • docs/API_reference/doc_index/backends/mongodb.md

Coming soon

  • Implementation of QueryBuilder and Hybrid Search.

@JoanFM
Copy link
Member
JoanFM commented Apr 24, 2024

Hello @caseyclements ,

Thanks a lot for this amazing contribution. May I ask if you could please sign off the commits so that we can pass the DCO and merge the PR if accepted?

JoanFM
JoanFM requested changes Apr 24, 2024
View reviewed changes
WaVEV and others added 28 commits April 24, 2024 12:36
@WaVEV @caseyclements
Mongo backend index initial.
98dda1c 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
find unit test.
3150375 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
unit test of configuration, find, del, get and persist data.
386b5ab 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
query builder refactor.
08cbc90 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
Rename function _collect_query_args_required_args => _collect_query_r…
6d45d1f 
…equired_args

Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
filter by parent id and query builder refactor.
12c7216 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
Add nested schema fixture.
df0e461 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
refactor test find, moving text and filter test.
f0ff913 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
Add test filter.
028f1be 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
Add test text search.
d2f1f03 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
Add query builder test.
2cafd61 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
Set collection name by schema name.
61b9943 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
Fix unit test.
b29290e 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
Add index name property.
b3b35cf 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
set scope for mongo_fixture_env.
fdbb334 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
add subindex test.
4cb814d 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
subindex find.
00de14f 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
Update readme and manage exception when an Index is missing.
2c083f9 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
test find without index.
b52cd14 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
refactor fixtures import.
63ede54 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
Importing all fixtures to avoid importing dependencies.
1b1ab1c 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
fix poetry lock.
04cd2a6 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
QueryBuilder: hybrid search implementation.
de1d74f 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
Hybrid search: fix score.
df57c22 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
Refactor: project now takes extra fields to project scores.
53e93f4 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
Moved query builder to query-builder-implementation branch.
addac74 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@WaVEV @caseyclements
Refactor and clean up.
6880
95bf4f3 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@caseyclements
Rename MongoAtlasDocumentIndex to MongoDBAtlasDocumentIndex
78a670d 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@caseyclements
Update env variable name
09c6e70 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@caseyclements
Add MongoDB Atlas setup instructions.
b5b73d1 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@caseyclements caseyclements force-pushed the mongodb-atlas-backend branch from d74b174 to b5b73d1 Compare April 24, 2024 16:36
@caseyclements
Updates in response to maintainer comments
62be05c 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@caseyclements
Added detailed README in tests/ for setup of indexes
aed03a2 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@prakul
Copy link
prakul commented Apr 24, 2024

All the comments have been addressed @JoanFM

JoanFM
JoanFM requested changes Apr 25, 2024
View reviewed changes
Copy link
Member
@JoanFM JoanFM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please check how to pass the black check? You can find in our CONTRIBUTING guidelines the steps to find

@caseyclements
black formatted
9df653b 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@caseyclements
Changed typing tuple > Tuple for python<=3.8
c2178de 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@JoanFM
Copy link
Member
JoanFM commented Apr 26, 2024

We are gettting these errors in the tests caused by the type annotations , maybe my requests were not ggood enough.

docarray/index/backends/mongodb_atlas.py:197: in MongoDBAtlasDocumentIndex
    ) -> Tuple[list[dict], list[float]]:
E   TypeError: 'type' object is not subscriptable

@caseyclements
black formatting ellipsis to pass
30d2b40 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@caseyclements
Copy link
Contributor Author
caseyclements commented Apr 26, 2024

Hi @JoanFM. Thank you for your working with me on this. I created a python 3.8 poetry environment for better coverage. black appears to have changed its mind between versions with regards to the ellipsis .... Running from 3.8, it moved it to the following line, where the day before, my 3.11 black moved it to the same one! :) I changed it to pass. ¯_(ツ)_/¯

-    class RuntimeConfig(BaseDocIndex.RuntimeConfig): ...
+    class RuntimeConfig(BaseDocIndex.RuntimeConfig):
+        ...

And now

-    class RuntimeConfig(BaseDocIndex.RuntimeConfig): ...
+    class RuntimeConfig(BaseDocIndex.RuntimeConfig):
+        pass

I'll turn to the typing issues, mypy with py3.8.

@caseyclements
Updated typing of Lists for backward compatibility
5c01811 
Signed-off-by: Casey Clements <casey.clements@mongodb.com>
@caseyclements
Copy link
Contributor Author
caseyclements commented Apr 26, 2024

I'm sorry that I missed list to List. I recognized the required change while updating tuple, but it was a busy morning. I pushed a new commit.

@codecov
Copy link
codecov bot commented Apr 26, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 41.11675% with 116 lines in your changes are missing coverage. Please review.

Project coverage is 44.75%. Comparing base (febbdc4) to head (5c01811).

❗ Current head 5c01811 differs from pull request most recent head 3b03f06. Consider uploading reports for the commit 3b03f06 to get more accurate results

Files Patch % Lines
docarray/index/backends/mongodb_atlas.py 40.10% 115 Missing ⚠️
docarray/index/__init__.py 75.00% 1 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff             @@
##             main    #1883       +/-   ##
===========================================
- Coverage   84.69%   44.75%   -39.95%     
===========================================
  Files         136      137        +1     
  Lines        9263     9459      +196     
===========================================
- Hits         7845     4233     -3612     
- Misses       1418     5226     +3808
Flag Coverage Δ
docarray 44.75% <41.11%> (-39.95%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@caseyclements
Copy link
Contributor Author
caseyclements commented Apr 26, 2024

Hi @JoanFM . I can address the code coverage by adding a new action to .github/workflows/ci.yml. @prakul We can set up the correct credentials and share a google doc.

@JoanFM
Copy link
Member
JoanFM commented Apr 26, 2024

Hi @JoanFM . I can address the code coverage by adding a new action to .github/workflows/ci.yml. @prakul We can set up the correct credentials and share a google doc.

no need to worry about code coverage

@caseyclements
Copy link
Contributor Author
caseyclements commented Apr 26, 2024

Hi @JoanFM . I can address the code coverage by adding a new action to .github/workflows/ci.yml. @prakul We can set up the correct credentials and share a google doc.

no need to worry about code coverage

Cool. What remains then?

In the next two weeks (I'm in London this coming one) when we add the QueryBuilder, we'll also set up the testing on your end. We are already running on our CI against Atlas on our end. Maybe we could set up a face-to-face meeting in a couple weeks. We can optimize to get the most of MongoDB's API once we know the scope of the use cases for the Indexes, and the data types.

@JoanFM
Copy link
Member
JoanFM commented Apr 26, 2024

Hi @JoanFM . I can address the code coverage by adding a new action to .github/workflows/ci.yml. @prakul We can set up the correct credentials and share a google doc.

no need to worry about code coverage

Cool. What remains then?

In the next two weeks (I'm in London this coming one) when we add the QueryBuilder, we'll also set up the testing on your end. We are already running on our CI against Atlas on our end. Maybe we could set up a face-to-face meeting in a couple weeks. We can optimize to get the most of MongoDB's API once we know the scope of the use cases for the Indexes, and the data types.

there seems to be a test timing out but not sure if it comes from your changes.

So what is the plan for the upcoming changes feom your side?

@JoanFM JoanFM merged commit f5c9ab0 into docarray:main Apr 29, 2024
@blink1073 blink1073 mentioned this pull request Jul 30, 2024
Closed
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JoanFM JoanFM JoanFM approved these changes

+1 more reviewer

@prakul prakul prakul left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@caseyclements @JoanFM @prakul @WaVEV
0