Commit 30aa13a

Daniel Axtens <dja@axtens.net>
2026-10-14 00:06:29
Wed 14 Oct 2026 00:06:28 AEDT
1 parent d041bea
Changed files (2)
blog
content
blog
scaling-limits
nginx
blog/content/blog/scaling-limits/index.md
@@ -0,0 +1,29 @@
++++
+title = "The limits of slow origins"
+date = "2026-10-13T23:38:40+11:00"
+description = "In which our first encounter with the scrapers goes poorly."
+
+tags = []
++++
+
+"In which our first encounter with the scrapers goes poorly."
+
+Fastly's analytics show some a surprising number of 503s popping up. I guess there are some limits to a [slow origin](/the-initial-tech-stack/) - 
+who knew?! - so I've tweaked some settings:
+
+ - Increased the various timeouts Fastly supplies for connecting to origins and getting responses from them. I suspect, 
+   looking at the miss timings that Fastly reports, that this is the single most significant contributor.
+
+ - Decreased the number of connections that each of Fastly's cache nodes can make to the origin, although I doubt that's a big factor.
+
+ - Added a health check in hopes of getting slightly better info in the future.
+
+ - Set up [shielding](https://www.fastly.com/documentation/guides/concepts/shielding/) in Sydney which should decrease the number of
+   misses that Fastly can't serve.
+
+ - Stopped nginx logging 404s to the error log: lots of the security-scanner types hit a bazillion files that don't exist in hopes of
+   finding secrets. We might as well do less disk I/O on those, especially since we don't currently do negative caching.
+
+To be honest, I'm a little disappointed: I had hoped that we'd hold up better!
+
+There are a few more things we can do at the Fastly end, but hopefully this is enough.
nginx/nginx.conf
@@ -21,6 +21,8 @@ http {
 
     keepalive_timeout  65;
 
+    log_not_found off;
+
     map_hash_max_size 65536;
     map_hash_bucket_size 128;
     map $uri $hash_etag {