Commit 5bc140f
Changed files (1)
blog
content
blog
blog/content/blog/tech-stack-1.md
@@ -0,0 +1,193 @@
++++
+title = "The initial tech stack"
+date = "2026-10-03T00:21:04+10:00"
+description = "The initial tech stack for intermittent.network: Self-hosted behind Fastly, FreeBSD jails, nginx"
+tags = []
++++
+
+# Goals
+
+I wanted to try self-hosting something, and I wanted to see how far I could get
+combining an old PC with a modern CDN.
+
+I also really don't want to get DoSed - or worse - have someone pop a shell on
+my webserver and then pivot to my entire home.
+
+# The architecture
+
+Building from the inside out, the architecture looks like this.
+
+## The old PC
+
+I have a Lenovo H30-05 mini-PC sitting in my office. (Several years back I
+walked into an OfficeWorks where it was a display model they were trying to get
+rid of - I think it set me back all of $60!) This is not a fast computer: it has
+a 2 core, 1.5GHz AMD processor, 4 GB of RAM, and spinning rust storage. It’s so
+low-power that its PSU is an external laptop-style brick rather than your usual
+desktop PSU.
+
+The computer has been running as my print server for some time. Out of a
+misplaced desire to broaden my experience from just Linux, I tried to set it up
+with FreeBSD. In hindsight, an AirPrint enabled print server was the wrong first
+FreeBSD project! But I really really did not want to fight with CUPS again, and
+I’ve wanted a good excuse to try jails for a long time, so I stuck with it. All
+of this would work just fine with Linux, just substitute containers in place of
+jails, and `nftables` instead of `pf` and so on.
+
+## Inside the PC: FreeBSD and nginx
+
+The web server that hosts the authoritative copy of the page you’re reading is
+an nginx server that runs in the `www` jail. The `www` jail has a virtual
+ethernet device connected to a local bridge that only connects to another jail,
+the `ingress` jail. There’s no internet access within the jail at all.
+
+The `ingress` jail is connected to the internal bridge, and has another virtual
+ethernet device which is bridged out to my home network. The external interface
+only has an IPv6 address - no IPv4. Given how huge the IPv6 address space is,
+I’m hoping that makes it basically unfindable by automated scanners.
+
+So to fetch this page, a connection comes in via IPv6, nginx does TLS
+termination, proxies the connection to the `www` jail, and returns the result.
+
+Outside the jail the `pf` firewall is set up to ensure that the `ingress` jail
+can’t make outbound IPv6 connections, only respond to inbound ones. So again, no
+internet access unless you can do a jail escape.
+
+## PC to outside world
+
+The next stop in our journey is my router, which needs a small pinhole in the
+firewall.
+
+IPv6, or rather, software support for it, makes this harder.
+
+As far as I know, my ISP is one of the few in Australia that offers a static
+IPv6 prefix. At least they used to: after several rounds of being eaten by
+larger ISPs, they’re pretty quiet about static IPv6 now… Hopefully it still
+works, because my router is very consumer-grade and will only let me specify
+individual IPv6 addresses to receive inbound connections. No wildcards here!
+I've specified the current public IPv6 address, so if the prefix changes that
+will break.
+
+This is definitely the most fragile part of this whole setup, which is a good
+opportunity to remind you that rather than bookmarking this website or hoping it
+shows up in a search engine in future, you should just [clone the git repo](https://www.intermittent.network/git/www/index.html)!
+
+Annoyingly this is a problem that dynamic DNS can’t solve: even if I could
+change a domain name, I don't know how to automate rewriting the IPv6 firewall
+rules.
+
+## Outside world to you
+
+Maybe this is unusual for self-hosting, but I decided to serve my site via a
+CDN. I picked Fastly because I work there and therefore I know the platform, but
+there’s no reason you couldn’t use another CDN.
+
+This gives me a few benefits:
+ - I can keep my origin’s IP address private, making it harder to DoS me.
+ - I can limit inbound connections to Fastly's IPv6 address space, meaning that
+ any traffic that comes to me has already run the gauntlet of Fastly’s various
+ protocol validation and other security measures.
+ - I can do TLS authentication with client certificates between me and Fastly,
+ meaning the traffic isn’t in plaintext between me and Fastly. I can do this
+ with self-signed certificates too, helping to keep my origin private.
+ - I can do caching, reducing the load on my ancient server.
+
+A have a theory that a lot of the risks and challenges of self-hosting become a
+lot more tractable if you design with a CDN. There are providers with free
+plans, or the ability to limit your spend.
+
+# The journey of a request
+
+So then, say you load this blog post.
+
+ - Your request goes to Fastly, which hopefully has the page cached and just
+ sends it to you without me doing anything. (Other than doing my job working
+ on the CDN there, I suppose!)
+
+ - If Fastly doesn’t have the content, it connects to me over IPv6.
+
+ - My router lets that through.
+
+ - It’s received by the old PC, where `pf` ensures that the connection is coming
+ from Fastly’s IPv6 space.
+
+ - It lands in the `ingress` jail, where nginx validates the client certificate,
+ strips off TLS, and forwards the request to the `www` jail.
+
+ - In the `www` jail, another nginx actually serves the request, and sends the
+ response back over the internal bridge to `ingress`.
+
+ - `ingress` sends the data back to Fastly, which is permitted by `pf` because
+ it’s in response to an established connection.
+
+ - Fastly takes the content and reencrypts it with the Fastly-managed public TLS
+ certificate.
+
+ - You get to read this webpage!
+
+## Does that mean there’s no dynamic content at all?
+
+Well no, I do have one little status thingy using
+[fastfetch](/cgi-bin/fastfetch.sh).
+
+# Do we meet our goals?
+
+## Self-hosting
+
+This machine in my house has half the cores of a Raspberry Pi 5, each running at
+less than two thirds of the speed. It’s not a quick machine. Fastly, on the
+other hand, has lots of very fast machines. I suppose time will tell, but I
+think this setup will work to keep the load on my machine manageable.
+
+Hopefully I will write a followup post on the simple things I do to optimise the
+caching that the CDN provides. None of it is complex but it’s a bit fiddly to
+bend nginx to your will!
+
+## Security
+
+### DoS
+
+I rely on my home internet working for such trivial niceties as “being able to
+work from home”, so I’m keen to avoid things that might allow someone to flood
+my fairly small pipe. Work here includes:
+
+ - Using IPv6 only makes the origin tricky to find by scanning.
+ - Using a self-signed certificate for me <-> Fastly means the it doesn’t show
+ up in certificate transparency logs (which would revail a domain name).
+ - Using a non-standard port between me and Fastly means that scans on ports 80
+ and 443 don’t find me.
+ - Using `pf` to limit traffic to Fastly’s IPv6 address space means that even if
+ you did find my IP, it would be difficult to confirm that it actually was my
+ IP.
+
+Ultimately it’s very difficult to prevent a DDoS - all of this is ultimately
+probabilistic - but hopefully it’s enough to avoid common attacks.
+
+### Popping a shell and pivoting
+
+Obviously it’s not ideal if someone defaces the website, but the thing I’m
+really most concerned about is pivoting to my home network.
+
+ - By virtue of the firewalls on the router and the host, you have to connect
+ via Fastly, which means hopefully you’re limited to HTTP, and everything that
+ is based on sending invalid HTTP requests is already out. My origin only
+ talks HTTP/1.1 over IPv6, so novel HTTP/2 or /3 attacks are also not my
+ problem. So your attack options are limited.
+
+ - Say you do manage to get into the `ingress` jail. A reverse shell is going to
+ be hard: you can’t make arbitrary outbound connections, and your only IPv4
+ address is on the internal bridge with `www`. If you manage to escalate your
+ privileges to root inside the jail, then yes, you have taken over the
+ website, but the firewall is managed outside the jail, so you’d need to do a
+ jail escape to get into my home network.
+
+ - Say I get inventive in the future and have more dynamic content and as a
+ result you manage to pop the `www` jail. You have the same problems as
+ `ingress` but worse - the only network interface is bridged only to the
+ `ingress` jail, which won’t forward packets for you. Plus, the source from
+ which `www` is built is all on git anyway, there’s no secrets there.
+
+ - If you do a jail escape, then yes, I guess at that point you win, you’re on
+ my local network. But at that point I think you’ve earned it! I’ve popped a
+ flag file in the root directory of the server: if you can capture the flag I
+ will give you kudos here and if practical buy you a beer or something.