Unified AI Context: One Memory, Many Agents

Here's a normal Tuesday morning.

The meeting agent takes a customer call at 9:30 and learns their migration is blocked on a schema change. At 10:15 I ask my coding agent to draft the fix, and it knows nothing about the call — so I re-type the context. At 11:00 I write a note about the decision, and the notes app has never heard of either. By the afternoon there are three systems that each hold a third of one situation, and the only thing joining them is me, retyping.

That's the n-agents problem, and it gets worse with every agent you add. Each one arrives with its own memory, and none of them share.

Bigger context windows don't fix it

The instinct is to reach for capacity: million-token windows, throw the whole corpus in.

That treats it as a storage problem. It isn't. Two things are actually broken, and neither is size.

Scope. Your coding agent's context is a repo. Your meeting agent's is a call. Your notes app's is a folder. Each is correctly scoped for its own job and structurally blind to the others. A bigger window makes each one deeper, not wider.

Identity. Even with everything in one place, the system has to know that [email protected] in a calendar invite, "Sarah" in a transcript, and @schen in a chat thread are one person. Without that, more context is more noise — the same person appearing three times as three strangers.

Retrieval doesn't fix identity either. Semantic search over an undifferentiated pile returns things that sound related. It can't tell you the customer who complained on Tuesday is the one whose renewal is next month, because that's a fact about entities, not about text similarity.

What a shared context layer has to do

Four jobs. I'll describe how we've built each, including where ours is still thin, because the gaps are more instructive than the wins.

1. Ingest from where the work already happens

If a context layer needs you to feed it, it's another app to maintain and you'll stop within a fortnight.

Ours pulls from the surfaces work already lands on: meetings run on the platform, notes imported from the five apps people actually keep them in — Notion, Apple Notes, Obsidian, Evernote, OneNote — notes written in-app, and completed todos.

Two honest limits on that. The note import is one-time, not a live sync — you run it, you get what's there, and new pages written in Notion tomorrow don't appear until you import again. And third-party meeting notetakers aren't sources today: if your transcripts live in Granola, Otter, Fireflies, or Fathom, there's no path into the graph, so the meeting context it holds is from meetings run on Waterr. Both are gaps I'd rather name than let you discover.

The loop is deliberately simple:

signal → look up nearby pages → LLM extracts entities + relations → write back → sync index

A new source arrives, the system pulls candidate pages that might be related, an LLM extracts entities and the relations between them, and that's written back into the graph.

2. Keep facts separate from the record of what happened

This is the design decision I'd defend hardest.

The store is SQLite, local, with a page per entity or topic. Each page holds a compiled_truth — the synthesized current state — kept separate from an append-only timeline of excerpts from the source material that produced it. Typed edges connect pages to each other. A counter tracks how many events have arrived since the truth was last compiled, which is what triggers recompilation.

Facts on top, audit trail underneath. It means synthesized state updates without losing provenance: you can always ask why the system thinks something, and get back the excerpt and the source it came from. An agent that asserts a fact it can't trace is an agent you'll stop trusting the first time it's wrong.

There's also an idempotency guard keyed on source type and ID, so re-ingesting the same meeting doesn't duplicate the timeline.

Where it's thin today: the read path is one coarse lookup that pulls candidate nearby pages for extraction context. No query pipeline, no embeddings, no full-text search yet. The write side is well ahead of the read side, and anyone building this should expect the same asymmetry — extraction is the fun part and querying is the part that determines whether it's useful.

3. Resolve identity — the actually hard part

Storage is a solved problem. Deciding two records are the same person is not.

Our People Brain does this over WhatsApp today: ingesting groups and DMs, resolving identities to a person record, tracking interactions and connections between people, compiling a per-person and per-group summary with an LLM, and bridging the high-signal ones into the central brain. The primitives are the giveaway — resolve-or-create a person, attach an identity to them, record an interaction, upsert a connection.

resolveOrCreate is doing the load-bearing work. Every new handle, address, or display name is either a new person or another identity for someone you already know, and something has to decide which. Get it wrong in one direction and one person fragments into five ghosts. Wrong in the other and two people merge into a chimera that has both their jobs.

There's no clean solution, only a stack of signals — shared groups, interaction history, name similarity, overlapping identifiers — and a compile step that revisits its own conclusions as more evidence arrives. That's why people records carry a recompile flag: identity is a standing judgement, not a one-time decision.

Anyone building a context layer should budget most of their time here. It's where the value is and where the difficulty is, and neither is obvious until you're in it.

4. Expose it over something standard

A context layer that only your own agents can read is a silo with better branding. The whole point is that the coding agent, the meeting agent, and whatever you adopt next all reach the same memory.

That's what MCP is for, and it's why we ship a server rather than a proprietary memory API. Any MCP client — Claude Desktop, Claude Code, ChatGPT, Cursor, Codex CLI, anything speaking the HTTP transport — connects to https://api.waterr.ai/v1/mcp with bearer auth and gets tools for listing scenarios, fetching a meeting, and pulling the scored analysis of a completed one.

Concretely: the coding agent can read the analysis of this morning's customer call while it writes the fix, without me retyping the call.

An open protocol matters more than a clever store here. Storage formats you can migrate. A protocol nobody else speaks means every new tool is an integration project, and you'll adopt fewer tools than you should.

Privacy is a boundary problem

Local-first is doing real work in this design, not just sounding good.

The graph is SQLite on the machine. That's not primarily a latency decision — it's that a unified context layer is, by construction, the most sensitive artifact you own. It's every meeting, every note, every relationship, joined up. "Just send it all to the model" is an answer that stops being acceptable the moment the layer is actually useful.

So there's a shield at the boundary. Queries going out through the agent get pseudonymized on the way — names, emails, phone numbers, chat identifiers detected and replaced before the request leaves — and the response is un-pseudonymized on the way back, so you see real names while the provider never did. It's built on Microsoft Presidio with a custom reversible operator and a small local NER model, so the detection cost is milliseconds rather than another network hop.

Two things worth stating plainly rather than glossing. It sits on the agent's LLM path — user queries, tool results, memory consolidation. Meeting transcript processing is explicitly out of scope. And pseudonymization is not encryption: it removes identifiers, not meaning. A masked query still describes your situation. It raises the floor considerably; it doesn't make the boundary disappear.

What this looks like when it works

Not magic. Just the absence of retyping.

The meeting agent takes the customer call and the entities from it — the person, their company, the blocked migration — land as pages with edges between them and excerpts underneath. The coding agent, connected over MCP, reads the analysis of that call while drafting the change. The note I write in the afternoon attaches to the same page as another event rather than starting a fourth island.

I stop being the integration layer. That's the whole pitch, and it's smaller and more useful than "AI that knows everything about you."

Frequently asked questions

What is unified AI context? A single memory layer that multiple AI agents read from and write to, so context gathered by one is available to the others. The alternative — the default today — is every agent maintaining its own memory and a human copying between them.

Is this the same as RAG? No, though they overlap. RAG retrieves passages that resemble your query. A context layer maintains entities, the relations between them, and a compiled current state with provenance. RAG can tell you which documents mention Sarah; a context layer knows Sarah is one person across four identifiers and that her renewal is next month. Most useful systems will do both.

Does MCP solve context sharing on its own? It solves the interface, which is the part that used to require a bespoke integration per tool. It doesn't decide what's worth remembering, resolve identity, compile facts from events, or keep provenance. MCP is how agents reach the layer; it isn't the layer.

Where does the context live? Locally — SQLite on your machine — with a pseudonymization boundary for what leaves. That's a deliberate trade: you give up effortless cross-device sync to keep the joined-up version of your working life on hardware you own.

Isn't a knowledge graph overkill? Why not one big markdown file? Start with the file — for a lot of people it's genuinely enough. The graph earns its cost at the point where you need to know that two mentions are the same entity, and where you need to answer why the system believes something. That's the threshold, and it arrives sooner than most people expect once more than one agent is writing.

Related reading: build your world model on why a digital brain matters at all, and your personal harness for a team of agents on running several of them at once.