Let agent owners review answers without opening everyone's chats

We were getting an agent ready to go live on a department’s website when the client asked the question that stopped the project: how do we see the chats?

We could not give them that. So we did not launch.

That was the right call and I would make it again, but it left me with a problem rather than a principle. An office that puts its name on an AI agent needs to know what it is telling people. That is the floor for owning a service, and we were close to shipping without it.

The trouble was where the answers lived. Official agents and our general-purpose chat service both wrote to the same Azure Cosmos DB store. Give the office access to the store and it could read conversations that had nothing to do with its agent. Refuse, and I was asking it to put its name on answers it had never seen.

For a while I treated that as a dial: more visibility, less privacy, and back again. Every setting made someone reasonable unhappy, which should have been my first clue that I was holding the wrong tool.

The useful move was splitting one question into two. Who should be able to read a person’s chat history? And what does the owner of an official agent need in order to review its answers? Those are different purposes, different populations, and different people who get to decide. Sharing a database does not make them one governance decision, and I had been letting the storage layout write the policy.

Give the owner a narrower thing to own

Work backward from what an agent owner actually does. They hunt for wrong answers. They want to know which questions keep coming up, and which ones their source material never covered. None of that requires anyone’s general-purpose chat history, and most of it does not require knowing which account asked.

So the ask got narrow. Let the owner review its own agent’s interactions, inside a defined scope, and tell users that quality review is part of the service. Personal chat stays outside that view.

Privacy questions do not disappear at that point. People type personal things into any text box you hand them, whatever the agent was built to answer. What goes away is the fake tradeoff, the one where supervising an institutional service supposedly means opening everyone’s chats. Once the dataset and the purpose were specific, the questions that remained were ones we could argue about productively.

Adoption got easier to discuss too. Instead of asking an office to trust a service it could not inspect, I could show how review would work and exactly what access it required. The office owns answer quality. The platform team owns the controlled path to the evidence.

Make the data enforce the decision

I built a pilot on Microsoft Fabric, Microsoft’s platform for storing and querying analytical data, alongside nebulaONE, our agent platform. Fabric mirrors the Cosmos conversation store read-only, with no path back to the source.

The mirror is broad. The owner-facing view is narrow.

That view flattens conversation documents into question-and-answer segments and filters them by agent ID. A Fabric data agent, which turns plain-language questions into queries over data you select for it, reads only that view. MCP (Model Context Protocol), the standard that lets an AI agent call an outside tool, connects a nebulaONE analytics agent to the data agent. The owner reviews answers in a chat window, not a database client.

The filter in that view is doing the access work, and I want to be specific about why that matters. The alternative, the one that gets built when a deadline is close, is handing the model everything and instructing it to only discuss one agent. That leaves an authorization decision to a model, over data it can already reach, in a system where the next person to ask is free to ask differently. A query against a view that contains one agent’s rows cannot produce another agent’s rows for someone who asks more insistently.

The condition on that claim is the part worth testing. The tool must not have a broader route around the view. Fabric grants and the nebulaONE agent audience both have to agree about who may use the path, and in this setup view access, data-agent access and database connect access are three separate grants. A friendly chat window does not simplify the permission model underneath it.

The project page has the pipeline, the SQL shape, and the design choices I would defend.

Where this actually stands

A pilot on test conversations, demonstrated end to end by the person who built it. The owner has not driven it under their own identity yet, and until they do, I do not have a handoff. I have a demo. Those are different, and the gap between them is where most internal tooling quietly dies.

That question from the launch is the one I am working through now, and the work is getting the owner the permissions to use the analytics agent themselves. It does not get answered by me demonstrating the path to myself.

The privacy design has unfinished edges too. It calls for dropping name, email and user-ID columns and limiting the review window, and I still need to verify the deployed column list and that window against what I specified. Even then, dropping an email column does not anonymize a question that describes somebody’s circumstances. Conversation text needs disclosure, careful handling, and a named person who decides who may read it. The mirror holds the broader store, so its administrators stay inside the risk picture, and nobody has tried to break the design on purpose yet.

A scoped view also settles nothing about retention, records handling, or what an owner does when review turns up something alarming. Those need their own procedures and their own owners.

The operational limits are duller. Answers took roughly 30 to 90 seconds in testing, I have not measured how fresh the mirrored data is, and every additional agent needs its own view, data agent, permission setup and MCP connection. Fine for one pilot. A bad promise to make institution-wide without automating the setup.

What I would copy at another institution

Separate the populations and purposes before anyone opens a permissions dialog. Write down which conversations an owner needs for quality review, which ones are none of their business, and what users will be told. Name the person who has to act on what review finds, because a review path with no owner is a dashboard.

Then give that owner the smallest useful data object. Filter by agent in the data layer, drop the identity fields you do not need, pick a review window, and keep that window distinct from whatever retention rule governs the source. Read what is left in the free text before you call it minimized.

Before calling it ready, test the whole path as the consumer, not as yourself. Try to reach another agent’s rows. Inspect inherited grants, not just the ones you assigned this morning. Successful access and denied access are both acceptance tests, and the second one is the test people skip.

The access design is the transferable part. Give owners a defensible way to inspect the service they are accountable for, and let the data layer, not a prompt, keep everyone else’s conversations out of it.