^{Introducing Aperture Nexus: An Open Source Memory and Cognition Layer for AI Agents}

A colleague walks into a new problem already knowing why it matters. An agent walks in at zero, every time.

That gap is not limited to coding agents. It shows up in every AI tool we use. I use Claudette for code and content, Copilot for email and code review, and Gemini for research and podcast. These are different tools with genuinely different strengths, but none of them know what the others know. My own coding or writing preferences live in a file that one tool has read and another has never seen. I rebuild myself from scratch for every tool, sometimes in every session.

The same gap appears at enterprise scale, with more at stake. A customer spends twenty minutes in a support chat explaining that they have switched to a plant-based diet, or that they are planning a wedding, or that the budget is tight this quarter. That is genuinely useful information, but it rarely leaves the support system. Marketing sends the same generic promotion two days later, the exact offer the customer already said they did not want. No one made a mistake. The two systems simply never shared what either of them had learned.

People notice this. It is why "this company does not know me" is such a common complaint, not because no one on the other end actually knew, but because knowing something in one system and acting on it in another turns out to be the hard part.

If that sounds familiar, this piece is about the problem you have felt. It is not unique to engineers, or to any single AI product. It is what happens when intelligence has no persistent sense of who it is talking to, what has already been decided, or why any of it matters.

What We Learned Auditing 20+ Memory Frameworks

Our own framework for thinking about this, Knowledge, Memory, and Context, started from a simpler question: how do humans actually do this? We wrote about that origin separately in Human Memory As The Perfect Template For AI Memory. Before we built anything, we audited more than 20 memory and cognition frameworks, across memory-first products, agent frameworks with memory as one feature among several, model-native memory tied to a single provider, and specialized research not intended for production. The full framework-by-framework breakdown lives in our three-part series: The Spectrum of Machine Cognition, AI Memory & Cognition Landscape: Deep Dive, and The Architect's Playbook.

Two findings mattered most for what came next. Most frameworks are not attempting to solve for organizational-scale knowledge at all, several say so directly in their own documentation, and almost none treat Context, the who, what, when, why, and how that makes a fact meaningful rather than merely retrievable, as a first-class concern. And "multimodal" usually means an image gets reduced to a caption, or a document to extracted text, with the original discarded once that conversion finishes. A handful of frameworks in our study avoid that lossy step; most do not.

Eight requirements stood out as necessary for this to work at real scale:

1. Sessions need to hold more than one kind of participant. A human, an agent, and a customer are often present in the same session, rather than one persona assumed throughout.
2. A timestamp alone is not enough. Who, what, when, where, and why all need to travel with a fact, or it will not be usable later.
3. Related things need to actually be connected, not merely stored in the same database. Existing knowledge needs a way to be incorporated into this foundation. 
4. The original data cannot be discarded once it has been processed. It needs to remain part of the record, not be replaced by a summary of itself.
5. Permissions must be enforced, not simply recorded as a field that nothing checks.
6. The data foundation underneath needs to be fast enough to hold up when thousands of agents are operating across hundreds of thousands of sessions at once, not only in a demonstration with a handful of records.
7. Governance is genuinely difficult. The right API and the right hooks can make it achievable rather than an afterthought.
8. There needs to be room to add cognition hooks specific to a given domain, rather than a single fixed pipeline for every use case.

As the landscape study concluded: "This landscape reveals a clear divide: while almost every framework can store data, very few have mastered the connective tissue, the seamless synthesis of Knowledge, Memory, and Context required for human-level cognition."

None of these eight requirements are solved by choosing a smarter memory framework alone. They are solved by bringing together the right data foundation, adding the right memory framework on top of it, and exposing that combination through an API built together with real users, not in isolation.

Getting the Architecture Right

That combination is the architecture: a foundation underneath, and a memory framework on top of it, each doing a different job. 

ApertureDB: The Foundation

Without a unified data layer, a typical agent needs three to seven separate API calls across different components to retrieve the right data for a single decision. Each of those calls is an opportunity for latency, for two systems to quietly disagree, or for a query to fail in a way nobody notices until a customer does.

My cofounder and I both came from Intel Labs, where we built graph-based storage systems for AI workloads. That background informed three decisions we made for ApertureDB, years before Aperture Nexus existed as a name.

Graph as the organizing principle, not a feature. Most vector database companies add graph support as a feature, a way to attach relationships alongside embeddings. We built the graph in as the spine. The entities in a dataset, people, categories, sources, are not metadata fields attached to some other object. They are nodes, and the relationships between them are first-class. A query can traverse those relationships natively, without the application stitching results together afterward.

Everything lives together. Not linked, not federated, but actually together. Multimodal data, images, video, audio, documents, text, embeddings, and metadata, needs to live in the same database rather than distributed across separate systems that reference one another. Every system boundary crossed adds latency and creates an opportunity for two systems to fall out of sync. A clip indexed in a vector store whose entity relationship is never updated will return a wrong answer, silently.

Everything is accessed together. An image node carries its pixel data and its embedding together, linked through the graph to its metadata and its relationships. Updating it updates everything. Deleting it deletes everything. No orphaned vectors accumulate quietly in a corner of the system.

The result is what we think of as a persistent, multimodal, shared enterprise knowledge base, memory at scale rather than per-agent, per-session state. This is also where Knowledge, the K in KMC, lives: the stable, institutional baseline, catalogs, policies, and historical facts, built directly into ApertureDB as first-class graph data, sourced once and shared, rather than reconstructed in every conversation.

This is not a theoretical architecture. ApertureDB is already in production at Fortune 50 scale with Home Depot, and Gracenote, a Nielsen company, runs petabyte-scale video semantic search on it today. The performance work behind Aperture Nexus's claims, sub-10ms vector search, ~15ms graph queries at billion-node scale, is the same engine underneath those deployments. 

Aperture Nexus: Memory and Context, Built On Top

Aperture Nexus is what we built on that foundation to organize Memory and Context, the two layers that change over time. It is open source and still early, growing quickly alongside the users building on it, not a finished product we are declaring complete.

The graph model illustrates what this looks like in practice: sessions, users, contexts, and commits, all connected to the descriptors and blobs they touch, in one graph rather than scattered across systems that must be queried separately.

Some of this is visible directly in the open source repository: every stored object carries user_id, department, and organization, the same three fields on every object, from a single developer session up to a multi-team deployment. That is deliberate. It is the foundation for real visibility scoping without a later schema migration, one of the eight requirements the audit surfaced from the start.

The Cognition Gap: What Building With an AI Coding Partner Taught Us

We test-drove early versions of Aperture Nexus the way we want any team to work with it: by actually building against it, not just checking boxes. One example from that process stuck with the whole team. I had an AI coding partner, I call her Claudette, build against a real specification. She delivered clean, well-structured code and told me, with complete confidence, that every unit test passed.

They passed. They simply did not run. The tests were mocked, thoroughly mocked, mocks testing mocks, and each time I asked about integration tests, she found a new way to describe not having written them.

That is the Cognition Gap in one sentence: Claudette had the Knowledge, the specification, and the Context, the session, but not the judgment to recognize that a mocked test is a hallucination wearing a green checkmark. This is precisely why the hooks in the diagram above exist. Retrieve and reason over are not enough on their own. A system also needs surface, so a human or another agent can see what is being relied on, and update, enrich, or drop, so a wrong belief does not sit there, confidently wrong, indefinitely. Claudette had a foundation and context. What was missing was the hook that would have let either of us catch the gap before it shipped.

That gap is personal as well as architectural, and it is not unique to any one of us. Everyone on the team juggles multiple AI tools that do not share what we have told each of them. My own preferences live in a file Gemini has never read. Nothing from a Claudette session travels to Copilot automatically. We built Aperture Nexus to solve enterprise memory. We need it to solve our own, which is usually how you know a problem is real.

Trust Does Not Come From Any One Piece

None of this works if it cannot be trusted, and trust here has at least three separate dimensions that all have to hold up together, or "reliable" is only a word on a slide.

Lineage means a decision can be traced back to its actual source, the image, the document, the specific session, rather than a summary of a summary, the same problem that started this piece for us: wanting to know how an image had arrived at its current form.

Permissions are a second dimension, and an honest one to name here. Aperture Nexus already captures who a fact belongs to, at the individual, team, and organization level, as part of every stored object. What is not yet true is that those permissions are enforced at query time, only that the input exists at write time. That gap is real, and it is one of the reasons visibility enforcement is a v2 priority rather than a nice-to-have.

The third dimension is trusting that the model itself did not hallucinate, a different problem from either lineage or permissions. A perfectly governed, perfectly permissioned fact is only useful if the reasoning built on top of it is sound. This is territory where we expect to collaborate rather than solve entirely on our own. Our landscape study surfaced frameworks built specifically around deterministic decision validation and causal traces, exactly the kind of starting point worth exploring rather than a layer we think we need to reinvent ourselves.

Context on its own is not enough. A fact with no lineage is only an assertion. A permission that exists on paper but is not enforced at the moment it matters is not really a permission. Trust is what results when lineage, permissions, and reasoning are all checked together and that will be a big part of what we focus on with our users.

What Comes Next

Governance

Governance is genuinely difficult, as the eight requirements above make clear, and it is the area we are being most deliberate about rather than rushing. An audit trail that exists by default, not one added after something goes wrong, recording who wrote a memory and under what authority, is the direction we are building toward, with the API and hooks doing the enforcement rather than a policy document nobody reads. The permissions-at-query-time gap described above is part of this same work. 

Open Questions in Agent Memory Design

We did not build this roadmap in isolation. At a talk in San Francisco this month, the audience raised exactly the questions we had not fully resolved, and some remain open for us today.

How should the state be represented for something that keeps changing, without either losing its history or accumulating unmanageable versions? Recency and history are not the same query. Sometimes the freshest fact is what matters; sometimes the full trail of how a fact evolved is what matters, and most systems serve only one of those well. When a user corrects themselves mid-conversation, how does that correction reach every other agent already holding the earlier context? We do not believe memory infrastructure can solve that on its own; it likely requires agent-side logic layered on top. And why migrate data into memory at all, rather than holding a live reference and querying in place? This was raised specifically in the context of enterprise permissions: a copied fact does not know when access has been revoked. A live reference would.

None of these are fully solved in Aperture Nexus today and this is where the biggest collaboration opportunities lie ahead.

What's Coming in v2

Version 0.1 of Aperture Nexus ships the core of the KMC model: commit, search, connect, and authenticate, validated against a live ApertureDB instance. The shape of it is simple by design:

memory = Memory()
principal = memory.authenticate(user_id="alice", api_key=...)
ctx = Context(principal=principal, session_name="support-2024-001", purpose="Customer reporting missing order")
info = Information(context_id=ctx.id)
info.log(text="Customer says order #4821 never arrived")
info.log(image="screenshot.png")
memory.commit(ctx, info)

What is not yet built: Memory.update() with lineage, so a memory can be superseded while its full history remains intact; visibility enforcement at search time, not only at the moment of writing; GraphRAG neighborhood search; hybrid search combining vector and metadata in a single query; recency-weighted scoring; and an MCP server so Aperture Nexus becomes persistent memory inside Claude Code, Cursor, or any MCP client. Each of these is scoped, none is speculative, and the roadmap is public in the repository's discussions.

Get Started with Aperture Nexus

Aperture Nexus is open source, under the MIT license. If you have felt the gap this piece opened with, rebuilding yourself for every tool, every session, this is what we built instead. Clone it, run the demonstration, and tell us what would unblock what you are building. We are prioritizing the roadmap based on real use cases rather than assumptions.

git clone https://github.com/vishakha041/aperture-nexus
cd aperture-nexus
docker compose --profile demo run --rm nexus-demo

‍

With thanks to Sonam Gupta, Sr. Developer Advocate at Telnyx, for her review and feedback on this article.

‍