Published release of whatbroke 0.3.0, a command line tool for comparing AI agent behavior between two runs. The code is written in TypeScript and distributed under the MIT license.
When changing the model, editing a prompt, or updating the framework, the agent can outwardly respond as before, but quietly stop calling the required tool or change the call arguments. Text diff does not see such failures. Whatbroke writes a trace log of the agent in JSONL format and shows information about which tool calls disappeared, which arguments changed, how costs and delays shifted, which final responses changed.
There is a proxy mode that does not require code changes: just point OPENAI_BASE_URL or ANTHROPIC_BASE_URL to the local proxy.
Since agents are non-deterministic, each scenario can be written several times – for each difference the frequency of repetition is displayed, and the noise that was present in basic runs is automatically reduced to the information level.
In version 0.3.0, the “import” command was added to convert ready-made OpenTelemetry logs (GenAI semantic conventions), Langfuse and LangSmith, as well as the “–fail-on” flag for integration into CI (return code 1 with disruptive changes). The tool is deterministic and works completely locally, without API keys, logs do not leave the user’s machine.