Docs: Design document for semantic memory #377
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR introduces a design document for refactoring profile memory into a generalized semantic memory subsystem. The system uses LLM-based fact extraction with specialized prompts to build, update, and search structured knowledge across various domains.
Key Changes:
- Design document outlining architecture for semantic memory extraction and storage
- Definition of data models, ingestion flows, and consolidation strategies
- Documentation of background processing and caching mechanisms
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| invoke the `LanguageModel` to produce a list of `SemanticCommand`s that | ||
| are applied to the feature set. |
There was a problem hiding this comment.
The search flow description appears incomplete or incorrect. Lines 88-89 describe invoking the LanguageModel to produce SemanticCommands applied to the feature set, but in a search operation, the system should be querying the vector database with the embedding, not modifying features. This should describe the semantic search process using the embedding to find relevant features.
| invoke the `LanguageModel` to produce a list of `SemanticCommand`s that | |
| are applied to the feature set. | |
| use this embedding to query the vector database for similar feature embeddings, | |
| retrieving the most relevant features as search results. |
There was a problem hiding this comment.
I kind of agree with Copilot here. I don't fully understand why we apply SemanticCommands to the feature set after turning the query into an embedding, unless maybe one of the SemanticCommands is some kind of wrapper for search?
|
Is this design document something we are implementing toward, or something that has already been implemented? I ask as if it has been implemented, I need to open a separate Activity to update the Architectural design documentation on docs.memmachine.ai. |
It is something we are working towards over the next weeks. |
| To update a feature set, the feature set and the prompt associated with | ||
| a `SemanticMemoryType` are sent to the `LanguageModel`. The LLM | ||
| produces a list of `SemanticCommand`s that are applied to the feature | ||
| set. |
There was a problem hiding this comment.
It would be useful to provide a short example of what a "feature set" can be. I referenced some output that I got after performing some memory additions & searches in main, and I'm assuming that it means the list of features that come after "profile memory":
"profile_memory":[{"tag":"Hobbies & Interests","feature":"favorite_food","value":"apples","metadata":{"id":1,"similarity_score":0.3549225330352783}},{"tag":"Hobbies & Interests","feature":"liked_food","value":"persimmons","metadata":{"id":3,"similarity_score":0.2976627051830292}}]
| is to translate the application specific `SessionData` into a `set_id` | ||
| that will be used by the `SemanticManager`. | ||
| This includes the `persona_id` and the `session_id`, for both the profile based | ||
| semantic memory and the session based semantic memory. |
There was a problem hiding this comment.
I may be missing something. Is persona_id something new to be introduced in the refactor or something already in the code? On a first reading it seems maybe like a combination of the user_id and group_id that currently make up our session data.
|
I read through this to get a handle on #371. Overall I think the description gives me a good high-level overview of what the semantic memory is supposed to accomplish. I just have some Qs about the details therein |
There was a problem hiding this comment.
This is useful and, I think, OK to merge. The "user experience" part at the top makes clear how I can interface with the refactored code, and then the rest follows...
Now, I understand "feature sets" not to be characteristics you glean from the data (e.g. "hobbies and interests" or "food preferences") but more like a set of identifiers that the message is associated with (group X, organization Y, etc.)
| metadata is folded into the text for better grounding. | ||
| 2. **Dirty-Set Tracking** – `SemanticUpdateTrackerManager` counts | ||
| messages and elapsed time per set to decide when updates should run. | ||
| 3. **Background Loop** – `_background_ingestion_task` wakes on a fixed |
There was a problem hiding this comment.
I feel running in the background would be very hard to let the caller know when an error happens. It think it is more straightforward to let the caller deside when to update/ingest, and the function/API call should let the caller know whether the result is success or failure.
If running in the background, a failure could cause the data/message to be gone forever.
Running in the background makes the user experience better in some way. It works as long as we figure out a good way to handle errors.
There was a problem hiding this comment.
Yes. We need an API to report the status of those background tasks.
|
I think feature set should be defined. |
| produces a list of `SemanticCommand`s that are applied to the feature | ||
| set. | ||
|
|
||
| Then when a search is being conducted with a query, the `SemanticProfileManager` |
There was a problem hiding this comment.
| Then when a search is being conducted with a query, the `SemanticProfileManager` | |
| Then when a search is being conducted with a query, the `SemanticManager` |
| - **Feature Entry** – `(set_id, memory_type, feature, value, tag, embedding, metadata, | ||
| citations)` tuples persisted by the storage backend. Metadata tracks | ||
| provenance, timestamps, and arbitrary annotations. | ||
| - **Set Config Entry** – `(set_id, llm_model_name, embedder_name, []semantic_memory_type_names)` |
There was a problem hiding this comment.
Go-like notation is confusing.
Description
We are looking to refactor profile memory into a general knowladge and fact extractor.
Where given a set of specialized prompts, each prompt can extract a set of facts.
This can then be used both for extracting facts about a user persona as well as extracting facts on other topics.
Type of change