Agent Conversation Analytics

The problem

This started with a question during a launch. We were preparing an agent to go live on a department’s website and the client asked how they would see the chats. We had no answer, so we held the launch.

An office responsible for an AI agent needs to see what it is telling people. Without that, it cannot check accuracy, find the gaps in its own source material, or decide whether the service is worth keeping. Asking an office to deploy an agent it cannot inspect is asking it to own answers it has never read.

Personal chat carries a different expectation. Nobody using a university’s general-purpose chat service agreed to let an unrelated department read their conversations. In our nebulaONE environment, both kinds of conversation landed in the same Azure Cosmos DB store, with no separation between agents.

That produced a deadlock. Broad database access exposed far too much. No access blocked responsible adoption. Read-only access did not help, because an owner with read access could still read conversations that were none of its business.

I wrote about how I got out of that deadlock in Let agent owners review answers without opening everyone’s chats. The short version: separate the access decisions before picking an interface. The decision worth defending was an owner reviewing its own agent’s interactions for quality, with disclosure to users and a named person accountable for acting on what review finds. That gave me something specific to build toward. It did not settle university-wide retention or incident response, which are still open.

What I built

A demonstrated analytics path on Microsoft Fabric, Microsoft’s platform for storing and querying analytical data, with the review happening inside nebulaONE rather than over a database connection.

The custody path, in order:

1. Unified chat and official agents write to the shared Cosmos store.
2. Fabric maintains a read-only mirror of that store, with no write-back.
3. A SQL view flattens the nested conversation documents into question-and-answer segments and filters them by agent ID.
4. A Fabric data agent, which turns plain-language questions into queries over data you select for it, reads only that view.
5. MCP (Model Context Protocol), the standard that lets an AI agent call an outside tool, connects a nebulaONE analytics agent to the data agent.

Two gates control access: Fabric permissions and the analytics agent’s audience. The pilot audience is two named people, including the agent owner. Fabric wants view access, data-agent access and database connect access as three separate grants, so being able to open the chat interface gets you nowhere on its own.

The diagram also shows the minimization design: drop the name, email and user-ID columns, and restrict the review period.

Status, honestly

This is a pilot on test conversations, demonstrated end to end by me. The owner has not yet used it under their own identity, so I do not have a handoff. I have a working demo, which is a real milestone and not the same thing. Getting the owner the permissions to use the analytics agent is the work in front of me now.

A few specifics I would not want anyone to assume:

• Two named people is the intended audience, not an audit of everyone with effective administrative access.
• I still need to verify the deployed column list and review window against the minimization design. Dropping identity columns would not anonymize a question that describes someone’s circumstances.
• The mirror holds the broader store, so its administrators and permissions stay part of the risk picture. Nobody has tried to break the design on purpose.
• Answers took roughly 30 to 90 seconds in testing, and I have not measured how fresh the mirrored data is.

Why you might care

If you lead AI adoption at another institution, your agent owners need a way to accept responsibility for answer quality. Give them a review path narrow enough to defend and they can find bad answers without becoming administrators of everyone else’s conversations.

That takes an ownership agreement as much as a query. Decide who reviews answers, what they may look at, what users are told, and who fixes the knowledge base or the service when review turns something up. A central team can run the shared infrastructure while local owners stay responsible for their own agents. This pipeline supports that arrangement. It will not make those decisions for you.

It should also change what you ask a vendor for. Agent-scoped analytics, audience controls and a documented conversation schema belong in the platform. Until they are there, somebody owns the manual setup for every department, and that somebody needs a maintenance plan.

Design decisions I’d defend

Put the row filter outside the prompt

The view filters on agent ID. A prompt asking for another agent’s conversations cannot widen it. That is a stronger boundary than handing the model every conversation and asking it to use discretion, because discretion is not an access control.

The grants have to match, though. If the integration identity can query broader objects, a scoped view is decoration. Before expanding access, test with the consumer’s identity and inspect inherited grants, not only the ones you just assigned.

Keep quality review in an interface the owner already uses

I started on a fixed report and could not design a useful one. Everything I sketched came out as a list of chats. The questions owners ask change week to week (what topics came up, which gaps recur, what the agent actually said behind a summary), and conversational review fits that better. Keeping it in nebulaONE also avoided maintaining another application. That choice needs validation through recurring real use, which this pilot has not produced yet.

Notes for the technical crowd

The view produces one row per question-and-answer segment and keeps a conversation ID so segments can be regrouped. It is a plain view, with no semantic model behind it. Personal chats are excluded at the consumer-view boundary, which is worth saying precisely: they are filtered out of what the owner can query, not removed from the mirrored storage.

Each additional agent needs its own view, data agent, permission set and MCP connection, and the flattening logic depends on the vendor’s document structure. Both are maintenance costs to solve before calling this a fleet-wide service.