API Testing: generated tests, chaos suites, CI gates · Evlio API Testing Break your API before an agent does Tests generated from your catalog, run on a schedule and in CI. Chaos suites throw malformed payloads, expired tokens, and retry storms at every endpoint, the way a confused agent actually will. Run your first suite See the catalog Tests written for you Scheduled runs Chaos suites Blocks the PR, not the release Test run #1284 2 failed triggered by PR #482 · main tests 8 passed 6 failed 2 p95 2.4s ✓ POST /payments · happy path 96ms ✓ POST /payments · missing amount rejected 42ms ✓ POST /payments · no token → 401 28ms ✕ GET /orders · pagination contract no cursor, 4,100 tokens ✓ GET /users/{id}/profile · PII redaction 88ms ✕ POST /orders/validate · retry storm 14 duplicate orders ✓ DELETE /builds/{id} · schema unchanged 74ms ✓ POST /auth/passkeys · expired token → 401 36ms 2 failures block this PR · contract policy: block-publish View diff 1,284 TESTS GENERATED, NOT WRITTEN 214 ENDPOINTS COVERED 6 CHAOS SCENARIOS PER ENDPOINT 9 BUGS CAUGHT BEFORE RELEASE 01 Generated coverage Nobody has time to hand-write 1,284 tests Evlio already knows every endpoint, its schema, and its auth. That is enough to generate the happy path, the boundary cases, the auth negatives, and the malformed payloads, per endpoint, without anyone maintaining a Postman folder. POST /payments generated suite 14 tests ✓ Creates a payment with valid body 201 · schema match ✓ Rejects missing amount 400 · field required ✓ Rejects negative amount 400 · boundary ✓ Rejects request with no token 401 · auth negative ✓ Rejects token missing write:payments 403 · scope negative ! Handles duplicate submit no idempotency key defined ✕ Survives 60 retries in 8s 14 duplicates created Assertions from the schema Status codes, required fields, types, enums, and nullability come from the spec Evlio already generated, so a schema drift fails a test instead of surprising a consumer. Auth negatives by default Every protected endpoint gets tested without a token, with an expired one, and with a token missing the scope. That is how the four endpoints with no auth got found. Editable, not magic Generated tests are readable files you can pin, edit, or delete. Add your own domain assertions and Evlio keeps them through the next rescan. 02 Chaos suites Agents are the worst client you will ever have A human reads the error and stops. An agent retries sixty times with slightly different arguments and reports success anyway. Chaos suites simulate that behavior on purpose, so you find the crash instead of an auditor. 01 Malformed payloads Wrong types, truncated JSON, extra fields, nulls where a string is required. What an agent sends when it half-understands your schema. found: 3 unhandled 500s 02 Retry storms The same call sixty times in eight seconds, arguments mutating slightly, no idempotency key in sight. found: 14 duplicate orders 03 Auth edge cases No token, expired token, valid token with the wrong scope, and a token for a revoked key. found: 4 endpoints with no auth 04 Schema drift Compares live responses against the contract on every run, so a removed field fails here rather than in an agent conversation. found: buildLog removed 05 Payload bloat Measures response size and token cost per call, because verbose output is a tax on every agent turn. found: 4,100 tokens on /orders 06 Latency under load Concurrent calls against the slowest tools, watching p95 and timeouts, because a slow tool multiplies traffic. found: sync_ledger p95 2.4s Chaos finding · retry storm on POST /orders/validate reproduced 3 of 3 runs WHAT THE SUITE DID 60 identical requests in 8 seconds, no idempotency key arguments mutated by one field each retry token refreshed mid-flight WHAT BROKE 14 duplicate orders created p95 climbed 138ms to 2.4s 500s returned with a stack trace in the body SUGGESTED FIX require an idempotency key rate limit per key at the gateway strip internal errors from responses 03 Automation In CI, on a schedule, and against production Suites run on every pull request, nightly against staging, and hourly as a smoke test on production. A failing contract test posts on the PR and, if your policy says so, blocks the publish. Schedules 4 active On every pull request contract + generated tests · ~90s blocking Nightly on staging full suite + chaos · 02:00 UTC report to Slack Hourly smoke on production read-only checks, no writes alerts on 2 failures Before every MCP publish tool schemas verified against live responses blocking github · PR #482 checks ❯ checks for PR #482 · payments-api ✓ evlio/contract-check 3 contracts verified ✓ evlio/generated-tests 1,282 passed ✕ evlio/chaos-suite 2 failed POST /orders/validate 14 duplicate orders on retry storm GET /orders unpaginated, 4,100 tokens per call ◷ evlio/mcp-publish blocked by policy refund_charge held back until schema verifies → review required · 3 stakeholders notified Closes the loop Testing that feeds the catalog, the contract, and the gateway A failing test is not just a red check. It lowers the endpoint score, blocks the contract publish, and keeps a broken tool out of your MCP server. Scores drop when tests fail Reliability is part of the grade. An endpoint that fails its chaos suite cannot sit at 92% in the catalog. API Catalog → Contracts stay honest A contract promises a shape. Tests verify the live API still returns it, and block the publish when it does not. API Contracts → Broken tools never ship Every MCP publish verifies tool schemas against live responses first. That is why refund_charge is still held back. MCP Management → FAQ How does Evlio generate API tests? From the same schemas and constraints in the catalog. A field capped at 1 to 500 in the schema editor is rejected at 501 by the generated tests, because tests, contracts, and MCP tool schemas all read one source. Can tests run automatically? Manual runs are on every plan. Scheduled runs, CI triggers, and chaos suites are included on Business and Enterprise. Why test APIs before agents use them? Agents exercise edge cases relentlessly and retry failures at machine speed. Generated tests find the gaps first, so the fix ships before an agent discovers the bug in production. From top startups to Fortune 500 companies, trusting Evlio SOC 2 Type II · certified ISO 27001 / 27701 · certified GDPR · compliant Encryption · TLS 1.3 / AES-256 SSO / SAML and SCIM · available Full security posture → RELATED READING Governance Aug 27 · 5 min API versioning strategies that survive breaking changes Read → Governance Aug 27 · 4 min An API governance checklist that fits in one pipeline Read → MCP May 22 · 7 min 23 tools, 4 agents, zero surprises Read → Find the bug on your terms, not an agent’s Connect a repo and your first suite runs in minutes, no test code required. Run your first suite See plans