Black vs Pickle

A black-box evaluation of how AI-generated tests find functional bugs in live APIs.

Pickle Browser opens real pages, reads them, and reports back with its sources. All on your machine, in a window you can watch. Free, runs locally, no account.

About Black

A black-box evaluation of how AI-generated tests find functional bugs in live APIs. The harder question is whether those tests find bugs. Each system receives only a JSON schema and one valid sample payload, then must generate API test cases that expose failures in a live reference API. The evaluation uses APIEval-20 v1.0, a black-box benchmark contributed by KushoAI. Because KushoAI is also one of the evaluated systems, this report includes the methodology, workflow definitions, repeated-run setup, and robustness checks so readers can understand where the performance difference comes from.

About Pickle

Pickle Browser opens real pages, reads them, and reports back with its sources. All on your machine, in a window you can watch. Free, runs locally, no account. Pickle Browser: tell it what to do, watch it do it The agent-first browser Tell your browser what to do. It runs on your machine, in a window you can watch, so you see every step and can take the wheel at any point. Use a model that runs locally for free, or bring the Claude or ChatGPT you already pay for.