When I started building agent memory, I expected retrieval to be the difficult part: which database to use, how to embed a record, and how to rank the results.
While building Luthn, a more basic question came first: who is allowed to see a memory?
A memory can be useful to everyone in a workspace without every related record being shared with everyone. That became the design boundary I needed to make explicit before tuning search.
A memory record crosses several boundaries
Luthn separates the original record from the projection an agent may use. The flow is not “save, then search.” It is closer to this:
| Boundary | Decision | Agent-facing consequence |
|---|---|---|
| Intake | Classify the complete candidate, including title, summary, content, and tags | Sensitive signals cannot hide in metadata |
| Storage | Keep private or restricted material behind the storage boundary | A stored record is not automatically visible |
| Projection | Return only a policy-approved safe projection | Agent APIs do not expose unrestricted source bodies |
| Ownership | Derive workspace and owner from authenticated server state | Caller-supplied identity cannot select another scope |
| Publication | Make external publication a separate approval | Agent visibility never becomes publication permission |
Search ranks eligible projections; it does not decide whether a record is eligible in the first place.
This is why the system keeps agent-safe knowledge, collection provenance, sensitive-access requests, and the original record as different things. The raw record stays behind a storage boundary. After classification and policy checks, the agent receives only a bounded safe projection.
Owner isolation is a server decision
The dangerous shortcut is to trust a field supplied by the caller: a user ID in JSON, an agent name, a session label, or provenance metadata. Luthn’s API documentation treats workspace and owner as server-derived authorization state. The request can carry context about the work, but it cannot choose the boundary that will authorize the request.
That difference is visible in the test suite. A multi-user test creates Alice’s memory while supplying a different claimed user in provenance. Bob’s read returns NotFound, Bob’s search returns no result, and the stored owner remains the authenticated Alice identity. The test also checks that identical turn-summary idempotency keys do not collide across personal workspaces.
Shared workspaces are a different case, not an escape hatch. A public-safe memory can be readable by two agents in the same workspace, while provenance remains outside the other agent’s scope. The policy is therefore not “private or public.” It is owner, workspace, visibility, sensitivity, retention, and agent eligibility considered together.
Protected information needs another path
A safe projection is not a weaker version of the original record. It is a different output contract. If a user later needs a specific protected detail, the system creates a requester-bound access flow instead of adding the raw value to ordinary search.
Before approval, the result remains unavailable. After approval, the test suite checks that the response is no-store, that an unrelated agent cannot use the handle, and that the requesting principal receives only the allowed content and read count. A request that expires during classification cannot be approved afterward. If no public-safe redacted output exists, approval still does not make raw content available.
A protected result is a short-lived, requester-bound decision—not a new shared-memory item.
This is also where audit design matters. Audit records can show that a request was created, approved, denied, expired, or read, but they should not become a backup copy of the protected content. The trail explains the decision without becoming another retrieval surface.
The research explains why the order matters
The concern is not theoretical. Research has demonstrated attacks that extract private queries from LLM agent memory, and browser-agent investigations have found assistants transmitting complete webpage content or collecting inputs related to banking and health data. MCP discussions raise the same implementation question in another form: a session ID or transport identifier is not automatically an authorization decision.
These sources do not prescribe Luthn’s exact design. They do explain why search quality cannot be the first acceptance criterion. A retrieval system that returns a highly relevant record to the wrong owner has improved ranking and failed its more important contract.
What I check before tuning retrieval
Before comparing embeddings or ranking changes, I want answers to six narrower questions:
- Who owns the record, and which workspace is the server authorizing?
- Which fields are classified together with the body?
- What projection may enter agent context?
- Which visibility and retention rules apply?
- Can another owner prove that the record is absent from read and search?
- Can a protected request expire, be denied, and remain content-free in audit output?
Luthn has not made memory risk disappear. It has made the risk easier to locate: first in classification and policy, then in ownership and projection, and only after that in retrieval. That order is the main lesson I am keeping. Search tells an agent what might be relevant. A permission model decides whether relevance is allowed to become visibility.
