Sort nightly test failures into bugs, flakes and infra problems
Rescale added an agent to its CI pipeline that classifies each failed end-to-end test as infrastructure, code bug, flaky test or environment problem, checks pod logs and past failures, and writes a remediation report for engineers. Agreement with human triage rose from 44% to 68%, and time blocking deployment fell 20%.
- Agent or tool
- Strands Agents on Amazon Bedrock
- Topic
- DevOps & Infra
- Reported on
- Blog (AWS)
- Added
- 2026-10-29
Why it matters
Keeping servers and pipelines healthy is constant, detail-heavy work. Agents that can read logs and run routine fixes are a big help, as long as a person keeps an eye on what changes. It is one of 45 sightings filed under DevOps & Infra on Agent Sightings (number 2 by interest), and one of 1 involving Strands Agents on Amazon Bedrock.
How you could try it
Here is a reusable prompt, written by us in our own words, that captures the task. Paste it into Strands Agents on Amazon Bedrock or any similar agent and adjust the details to your situation.
Our nightly browser tests failed. For each failure, look at the logs and past failures and label it as infrastructure, code bug, flaky test or environment. Give the evidence and the file or function to check. Do not change anything, just report.
The prompt breaks down into 4 steps you can check as the agent works:
- Our nightly browser tests failed.
- For each failure, look at the logs and past failures and label it as infrastructure, code bug, flaky test or environment.
- Give the evidence and the file or function to check.
- Do not change anything, just report.
Tip: Try it first on a staging system or with read-only credentials, and require approval for anything that changes production.
Source
Read the original first-hand account on Blog (AWS): https://aws.amazon.com/blogs/machine-learning/how-rescale-built-an-ai-triage-ag…. Our summary is a paraphrase; details are in the source.
Featured in these guides
Built an AI agent? Feature your agent on Agent Sightings, free submission or a featured spot.