Black vs CodeTrain

A black-box evaluation of how AI-generated tests find functional bugs in live APIs.

CodeTrain is a Socratic AI coding tutor: it asks questions, reviews what you write, and runs your tests, but never writes the code for you. Free to start.

About Black

A black-box evaluation of how AI-generated tests find functional bugs in live APIs. The harder question is whether those tests find bugs. Each system receives only a JSON schema and one valid sample payload, then must generate API test cases that expose failures in a live reference API. The evaluation uses APIEval-20 v1.0, a black-box benchmark contributed by KushoAI. Because KushoAI is also one of the evaluated systems, this report includes the methodology, workflow definitions, repeated-run setup, and robustness checks so readers can understand where the performance difference comes from.

About CodeTrain

CodeTrain is a Socratic AI coding tutor: it asks questions, reviews what you write, and runs your tests, but never writes the code for you. Free to start. Two steps, about a minute, and the first one is going to catch you. The tutor's replies are pre-recorded, so it knows the handful of ways this lesson usually goes wrong. The real one reads whatever you actually wrote, and it is free for ten lessons a month.