Description
Is your feature request related to a problem? Please describe.
The current in-memory retriever uses naive keyword matching, which can limit search relevance and effectiveness, especially if the session interaction grows for complex multi-agent tasks.
Describe the solution you'd like
I propose adding a new type of InMemoryService for the BM25-based retriever (via the rank_bm25 library) called InMemoryBM25RetrievalMemoryService. This would improve the relevance of search results using a weighted termed based algorithm.
Key details:
- BM25 retriever is built for better results
- Can be used using the InMemoryBM25RetrievalMemoryService
- Default retrieval of top 3 with max_results
- Cache is invalidated on memory/session updates to ensure accuracy
Describe alternatives you've considered
Offloading search to an vector retrieve but that's already done with VertexAI RAG. Out of scope for in-memory prototyping.
Additional context
- Code is available and ready to share if requested (This has now been submitted along with unit tests)