The limits of slow origins
“In which our first encounter with the scrapers goes poorly.”
Fastly’s analytics show some a surprising number of 503s popping up. I guess there are some limits to a slow origin - who knew?! - so I’ve tweaked some settings:
Increased the various timeouts Fastly supplies for connecting to origins and getting responses from them. I suspect, looking at the miss timings that Fastly reports, that this is the single most significant contributor.
Decreased the number of connections that each of Fastly’s cache nodes can make to the origin, although I doubt that’s a big factor.
Added a health check in hopes of getting slightly better info in the future.
Set up shielding in Sydney which should decrease the number of misses that Fastly can’t serve.
Stopped nginx logging 404s to the error log: lots of the security-scanner types hit a bazillion files that don’t exist in hopes of finding secrets. We might as well do less disk I/O on those, especially since we don’t currently do negative caching.
To be honest, I’m a little disappointed: I had hoped that we’d hold up better!
There are a few more things we can do at the Fastly end, but hopefully this is enough.
Updates
- November 2026: This seems to be working pretty well. I did also set up negative 404 caching, although I’m not sure how much that helps - I still seem to get a fair amount of security scanner traffic.