Helping an insurer control AI agent behaviour as models change.

Evaluation and control work for an insurance company's in-house engineering teams building custom agents across the underwriting process.

Agent evaluationLLMOpsInsurance

Strengthen evaluation across your AI agent portfolio

Conceptual architecture

The challenge

The insurer's engineering teams were developing multiple AI agents, each focused on a different part of underwriting. As the underlying models changed, the organisation needed a way to evaluate each agent's behaviour rather than assuming a new model would behave like the one it replaced.

An agent is more than its language model. Its prompts, tools, retrieved information and workflow determine what it does. A model that improves a general benchmark can still produce a regression in a specific underwriting task. The client needed control and evaluation at the level where its agents actually performed work.

The solution

The engagement focused on giving the engineering teams a way to control and evaluate their underwriting agents as models evolved. The central idea was to assess the complete task behaviour of each agent, with model changes treated as changes to an application rather than a transparent infrastructure substitution.

This connects model lifecycle management to the agent portfolio. Each agent has a defined purpose and a relevant evaluation context; teams need to know which configuration they are testing and what behaviour is acceptable before rolling a change into operational use.

Technical evaluation focus

Agent identity and configuration

the agent being assessed, its task and the model configuration under test.

Task-level assessment

whether outputs and actions satisfy the underwriting workflow's requirements.

Change comparison

how a candidate model affects an existing agent's behaviour.

Evidence for decisions

examples and findings that help engineering and business owners decide whether to accept a change.

Portfolio control

keeping the evaluation of separate agents connected to an overall model-change process.

Conceptual architecture

Why evaluation must follow the workflow

For an underwriting support agent, a plausible answer is not enough. Relevant tests can examine whether required information is omitted, whether evidence is correctly represented and whether an exception reaches the appropriate reviewer. Agents with action tools also require assessment of what they attempt to do, not just the text they generate.

These concerns differ across tasks, so a single aggregate score cannot replace agent-specific evaluation. The useful control is the ability to connect a proposed model change with evidence about the actual agent affected.

What the work addressed

The engagement addressed the insurer's need to maintain control over a growing set of custom underwriting agents while underlying models changed. It connects bespoke AI development with the evaluation discipline needed to manage that capability over time.

The case demonstrates an operational AI challenge that persists after an initial agent is built: keeping behaviour understandable and assessing change in the context of the business process.

Related services

A model catalogue and managed endpoint workspace

PreviousProject experienceEnterprise AI Model Governance on Azure

Example environmental assessment copilot with source documents and a reviewable report draft

NextProject experienceEnvironmental Assessment Copilot

Strengthen evaluation across your AI agent portfolio

Evaluation and control work for an insurance company's in-house engineering teams building custom agents across the underwriting process.

Strengthen evaluation across your AI agent portfolio