Replace tower RateLimitLayer with custom middleware returning HTTP 429.

Jforresterrequested to merge
T407588 into main

Tower's RateLimitLayer uses backpressure to handle rate limiting: when the limit is reached, poll_ready() returns Pending, causing requests to queue until the next time window rather than being rejected. This is unsuitable for a service running untrusted user code, where excess requests should be shed immediately with a clear signal to the k8s layer so it can be re-routed or fed back to the caller as appropriate.

Additionally, the previous code had a bug where the ServiceBuilder wrapping the router was constructed but its result was never assigned — get_app() returned the undecorated router, so rate limiting was not applied at all.

This commit replaces the tower layer with a custom fixed-window token-bucket rate limiter exposed via axum::middleware::from_fn. When a request arrives and no tokens remain in the current window, the middleware returns StatusCode::TOO_MANY_REQUESTS (429) immediately without forwarding to the handler. This matches the behaviour of the JavaScript evaluator's express-rate-limit middleware, which also returns 429 on the spot.

Specifically:

  • New module rate_limiter.rs: contains a RateLimiter struct (token bucket with configurable max_tokens and window duration) and an async middleware function that acquires a token or returns 429. Uses std::sync::Mutex (not tokio::sync::Mutex) because the lock is never held across an await point — only a fast integer check and decrement.

  • mod.rs: get_app() now constructs an Arc<Mutex> and applies it via Router::layer(middleware::from_fn(...)), correctly returning the layered router.

  • Cargo.toml: tower is moved from [dependencies] to [dev-dependencies] since only the test suite needs tower::Service for calling .call() on the router. The "limit" feature is replaced with "util".

  • test_service.rs: the expect_rate_limited branch now asserts StatusCode::TOO_MANY_REQUESTS instead of relying on timing-based delay assertions. A new test_rate_limiting_returns_429 exercises the path with max_requests=1, confirming the second request within the same window receives 429.

Bug: T407588
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com

Edited by Jforrester

Merge request reports