-
Notifications
You must be signed in to change notification settings - Fork 234
feat: add filter capability to DocumentArray #1051
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
4ce0576
refactor: instances of parametrized tensors are no longer parametrize…
Jackmin801 6d52497
feat(v2): rich display for doc and da (#1043)
c006fb3
feat: add filter capability to DocumentArray
eaab724
test: add tests on operations on out new types
18756e6
test: add equality operator for Text Document
399d58b
feat: support python 3.7 (#1055)
samsja c7a7332
test: remove some useless tests
98a75c2
docs: add better docstring for text
2af8306
8000
test: add bones for filter test
fdbba68
test: full test driven development
cd0a8ac
refactor: remove useless class
3378567
fix: blackify
d3b1080
fix: fix the mypy
07fb69f
fix: apply black and some ruff
3b96083
feat: filters ready
c1ef8ea
Merge branch 'feat-rewrite-v2' of https://github.com/docarray/docarra…
f278c0a
fix: blackify text document
3ea2dcc
refactor: fix str
1acb0ec
docs: add docstring to filter
8757648
fix: apply suggestions
bd4f797
fix: apply suggestions
63ce5d8
refactor: create query language package
cb91273
test: add tests for query language package
5830839
fix: fix type hints
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| import json | ||
|
|
||
| from typing import Union, Dict, List | ||
|
|
||
|
|
||
| from docarray.array.abstract_array import AnyDocumentArray | ||
| from docarray.array.array import DocumentArray | ||
|
|
||
|
|
||
| def filter( | ||
| docs: AnyDocumentArray, | ||
| query: Union[str, Dict, List[Dict]], | ||
| ) -> AnyDocumentArray: | ||
| """ | ||
| Filter the Documents in the index according to the given filter query. | ||
|
|
||
|
|
||
| EXAMPLE USAGE | ||
|
|
||
| .. code-block:: python | ||
|
|
||
| from docarray import DocumentArray, BaseDocument | ||
| from docarray.documents import Text, Image | ||
| from docarray.util.filter import filter | ||
|
|
||
|
|
||
| class MyDocument(BaseDocument): | ||
| caption: Text | ||
| image: Image | ||
| price: int | ||
|
|
||
|
|
||
| docs = DocumentArray[MyDocument]( | ||
| [MyDocument(caption='A tiger in the jungle', | ||
| image=Image(url='tigerphoto.png'), price=100), | ||
| MyDocument(caption='A swimming turtle', | ||
| image=Image(url='turtlepic.png'), price=50), | ||
| < 8000 /td> | MyDocument(caption='A couple birdwatching with binoculars', | |
| image=Image(url='binocularsphoto.png'), price=30)] | ||
| ) | ||
| query = { | ||
| '$and': { | ||
| 'image.url': {'$regex': 'photo'}, | ||
| 'price': {'$lte': 50}, | ||
| } | ||
| } | ||
|
|
||
| results = filter(docs, query) | ||
| assert len(results) == 1 | ||
| assert results[0].price == 30 | ||
| assert results[0].caption == 'A couple birdwatching with binoculars' | ||
| assert results[0].image.url == 'binocularsphoto.png' | ||
|
|
||
| :param docs: the DocumentArray where to apply the filter | ||
| :param query: the query to filter by | ||
| :return: A DocumentArray containing the Documents | ||
| in `docs` that fulfill the filter conditions in the `query` | ||
| """ | ||
| from docarray.utils.query_language.query_parser import QueryParser | ||
|
|
||
| if query: | ||
| query = query if not isinstance(query, str) else json.loads(query) | ||
| parser = QueryParser(query) | ||
| return DocumentArray(d for d in docs if parser.evaluate(d)) | ||
| else: | ||
| return docs |
Empty file.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.