Service6 min read

E2E test suites in CI that engineers actually run

The shape of a test suite that survives contact with a real engineering team: small enough to be fast, flake-budgeted, run on every PR, and trusted enough that on-call engineers follow its signal.

By the RanarTech engineering team · Published 2026-09-22

We have inherited dozens of test suites. The pattern is depressing and consistent: thousands of E2E tests, hundreds of which are flaky, a CI pipeline that takes 45 minutes, and a team that clicks 'merge anyway' when the suite goes red. Here is the shape of a test suite that survives contact with engineers.

The flake budget

We treat test flake like a production incident. Any test that flakes more than twice in a week is either fixed immediately, quarantined, or deleted. A test suite with a 5% flake rate is worse than no test suite because engineers learn to ignore it.

One test per user journey

The right unit of E2E coverage is a user journey, not a page or a feature. Sign up, complete onboarding, make a purchase, cancel a subscription. Twenty journeys, well-implemented, cover more business risk than 500 page-level tests.

Parallelize or die

A suite that takes 45 minutes runs once a day at best. A suite that takes 8 minutes runs on every PR. The difference between 'we ship with confidence' and 'we hope it works' is the suite runtime. We parallelize by data, by user, by tenant, by anything that lets us run independent slices.

Chaos in staging, not production

Every quarter we run a chaos drill in staging: kill a service, slow a dependency to 5 seconds, drop 30% of traffic. If the system does not degrade gracefully, that is a bug worth fixing before it surfaces in production. We document the failure modes and add a test for each one we have seen.

Load tests as a release gate

Continuous load testing is expensive and noisy. Scheduled load testing on every release candidate, against a staging environment with production-shaped data, catches the regressions that matter: N+1 queries, missing indexes, unbuffered writes. We gate releases on p99 latency staying within a budget.

Security scanning that is not noise

We separate static analysis, dependency scanning, and secret scanning, and we set thresholds that escalate the right things. A dependency with a known CVE in a transitive dep gets a PR automatically. A static analysis finding on a non-critical path does not block release but does open a ticket. The trick is calibrating the noise floor.

FAQ

How big should our E2E test suite be?

As small as possible. The ideal E2E suite has one test per critical user journey, runs in under 10 minutes, and covers every revenue path. Hundreds of E2E tests usually means hundreds of flaky tests and a suite nobody trusts.

What about load and chaos testing?

Both are release gates, not continuous tests. We run load tests on every release candidate against a staging environment with a representative data shape, and we run chaos drills quarterly. Continuous load testing is expensive and produces more noise than signal.

More from RanarTech Insights

Have a project like this?

Discovery call within 48 hours. NDA-friendly. Most engagements kick off within 1-2 weeks.

[email protected]