agentic evals are not an afterthought
Let's take a break from AWS content for today! (hurrah) Today I will answer some critical questions related to a topic I'm interested in which is AI evaluations i.e. ensuring agentic/LLM-generated outputs are reliable and dependable enough within the healthcare landscape.
In a recent DS meeting I was co-leading, someone asked about why agentic evaluations are important as we have great models nowadays. Every new model hits new benchmarks and these great models are now great at producing excellent SQL and not straying from the plan, etc. Why are evaluations still important then?
It's a good question and to answer this, we have to separate the model from the system the model is deployed within. Although model scores are improving, this doesn't translate directly to 100% reliable production and system behaviour at all times. The benchmarks for top of the line models do not capture systemic metrics such as tool failure, API failures, context-changes or changes in user variability. Only system level evaluations can surface these. Even changing something as simple as a prompt could have regressions but these won't be caught unless evaluations are implemented (for example, running evaluations as a pre-merge check).
To capture these metrics, we must collect traces (i.e. step records, tool calls, outputs of a run) to evaluate such workflows. Only then can we measure metrics such as costs, latencies, tool success, task success. It's imperative to continue doing this even after launch. We also need to monitor for drift continuously so we can be proactive with changes to our products and systems and catch regressions early. These metrics can also be tied directly to business outcomes:
- collecting token costs means you can have better intuition about system scalability
- task completion metrics can help you understand operational reliability (especially critical in health)
- latency can help with understanding user experiences
Overall, evaluations matter greatly and must be a key counterpart to building agentic systems, not an afterthought.