Geo-distributed blog for $7/mo

Recently I got the itch to try and see if I could make this site load quicker, and even though there's not much to do from a content perspective (pages are ~5KiB of content, static files have long caches), the largest amount of time spent was purely latency based, as this site was served only from Germany, if you were not close then you are going to have to wait a bit.

The only way to reduce latency¹ is to put the devices that are communicating closer.

Moving this server to any other point wouldn't make much sense, the only option is to have multiple servers in different parts of the world.

As far as I know, the proper way to do this is to have a single IP shared between multiple DNS servers in different locations, then through Anycast, the client would route to the "closest" server.

I couldn't really find an operator that would sell me an Anycast "service" or even VPS with Anycast IPs. Building this myself would require buying a /24 IP block (255 IPv4 addresses) for $5-10k, find a few VPS vendors that also provide BGP services to announce my IP range.

That would be ideal, but it's a bit out of budget² for a side project; Sadly, I took the more pragmatic route: paying for a DNS service which handles this internally.

I tried a few options:

dnsimple promised a nice hobby tier, but gates the GeoDNS records behind $50/mo.

Gcore offers a nice free plan, to try it out, I made a script using fly.io to spawn machines all over the world and measure the perceived latency.

However, I kept seeing this pattern:

Enriched with response region:

Some of the requests are served from the wrong region (LAX in the previous table). I assume³ that Gcore's GeoDNS service uses some kind of reverse-lookup database to determine where the clients are, and which record to return.

I don't think this is a good way to implement location-aware DNS: those databases go out of date, and most importantly the concept of "to which physical location this address belongs to" does not make sense.

The right way to do this is to use the DNS server's geographical location for the decision, the user will be routed to the "closest" DNS server thanks to BGP anyway — given a sufficient number of POPs, the DNS server's location is a pretty good proxy for the user's location.

The only provider that offers this feature, as far as I know, is AWS' Route53, so that's what I am using now.

If you know of another DNS provider with this feature, please let me know.

For servers, I picked Greencloud for my VPS in Singapore ($25/year) and Racknerd for the VPS in Texas, USA ($12/year). I was already using Hetzner for my VPS in Germany ($36/year)

DNS is Route53, which costs $6/year.

The grand total is $7/month.

Was it worth it? Yes! Average latency is below 100ms⁴ from any⁵ location

This page should now load much faster; establishing a TLS connection requires 3–4 roundtrips, so for someone in San Francisco TTFB is now ~175ms, instead of ~650ms.

I use Caddy as my HTTP server, which will by default, ask Let's Encrypt for SSL certificates.

I was using the HTTP-01 challenge, which performs a GET request to /.well-known/acm-echallenge/<TOKEN>, but now that doesn't work anymore — all requests get directed to the Texas server, and that server does not know about requests performed by the other two servers.

The easy option would be to change and use the DNS-01 challenge, but that would require all 3 of my servers to have keys to manage my DNS records, and I don't want to trust a $10/year VPS provider with my keys.

Instead, I opted for disabling SSL certificate management in my "secondary" servers, and ship the certificates from the "main" server to them.

This is the caddy config for the secondary servers:

blog.davidv.dev {
    tls /var/lib/caddy/blog.davidv.dev.crt /var/lib/caddy/blog.davidv.dev.key
    header +region {$REGION}
    import blog_common
}

and a cron job on the primary server to distribute the certificates:

scp $CERT root@blog.sg.davidv.dev:/var/lib/caddy/
scp $KEY root@blog.sg.davidv.dev:/var/lib/caddy/
ssh root@blog.sg.davidv.dev "chown caddy /var/lib/caddy/blog.davidv.dev.*"
ssh root@blog.sg.davidv.dev "systemctl reload caddy"

Cloudflare offers a free (as in, $0) service for this kind of thing, but I don't believe they should control any more of the internet. Friends don't let friends use Cloudflare.