Measure Agent Quality and Safety with Azure AI Evaluation SDK and Azure AI Foundry

A practical evaluation pipeline for GraphRAG agents with quality metrics, safety scans, and observable runs. Introduction In Part 4, we orchestrated multiple agents. This article (Part 5) answers a harder question: can we prove that the system is reliable enough for production workloads? For AI Engineers, answer quality alone is not enough. You also need: Repeatable quality checks before release. Safety evidence for security and compliance reviews. Traceability when behavior changes after prompt, model, or tool updates. This part adds an evaluation module under src/evaluation with three goals: Quality: task completion, intent resolution, tool-call behavior, graph-grounded correctness. Safety: adversarial probing with red team strategies and risk categories. Observability: telemetry and artifacts that support debugging and regression analysis. How the three goals are measured Goal Primary signals Current evidence in this article Quality task_adherence, intent_resolution, relevance, cohe