Posts
Running HTTPS and Gemini sites on Fly.io
As it's been way too hot this past week to close my office door and window in order to have a quiet environment for recording Always Developing videos, I had time to clean up the hosting of a whole host of otherwise inactive domains.
These domains are the usual mix of alternates for in-use domains that I've collected over the years, and generally use for development and testing, such as ianmjones.dev, ianmjones.ninja and the most awesome ianmjones.rocks π€. And also a long list of project names that either never took off, didn't make it to release, or are likely never even going to see a single line of code written π. In all, we're talking 46 domains.
However (un)likely it is that these domains are going to eventually become an actual thing, in the interim I wanted to make sure they were at least documented, and preferably reachable, mostly because I hate loose ends.
Previously...
Over the previous couple of months or so, I'd already dusted off a project that I'd created a while ago to deploy Caddy web server as a custom Juju Charm, with a companion Caddy Domain Charm that allowed for serving any number of domains as either proxied sites served via other Charms, or static resources.
It worked pretty well, and was fun to develop as I used the Python based Operator Framework for the Charms. Python is a language I don't have much experience with, but I can see why so many people like it as it's super powerful and has a relatively clean syntax.
However, even though I got the Charms to a useful state, and even deployed to a "manual cloud" at Linode with all these domains, it didn't feel quite right.
Firstly, it was way over-engineered for a project to host a bunch of otherwise unused domains. As part of the setup I deployed a 2Gb Linode for the Juju Controller, a 1Gb Linode (Nanode) for the Juju Machine that ran the Caddy Charm and subordinate Caddy Domain Charm, and a NodeBalancer to ensure I had a consistent IP address and could re-build or scale the back end.
Juju is very much a proper enterprise grade application management framework. It's quick to get going with, incredibly scalable, easy to build custom application charms for, and way too much for my needs! π
And of course, with a NodeBalancer, two Linodes, and backups enabled, I was looking at approximately $30 per month in bills. And while I was thinking that at some unspecified future date I'd be using these same machines for hosting other projects at no additional cost, that felt a little high for this particular project (a steal for any other project though given its flexibility).
sourcehut pages
Chances are that you're reading this on ianmjones.com, which is hosted on sourcehut pages.
I did briefly consider just hosting all these domains on sourcehut pages as I'm very happy with the service, it serves both HTTPS and Gemini sites, and it's free.
However, the way you deploy to sourcehut pages means you need to push out a tarball of the site's content to a unique path per domain and protocol. My build script already deployed to four endpoints to cover ianmjones.com and www.ianmjones.com for both HTML and Gemini content.
My previous .build.yml for deploying ianmjones.com
If I wanted to also host another 46 domains, that would have been a total of 96 copies of the site being stored on sourcehut pages' infrastructure, and a whole heap of curl requests to get it there on each update to my site.
I simply couldn't bring myself to do that, it didn't feel like a responsible or kind thing to do to a fledgling service like sourcehut.
Fly.io
And then I listened to a couple of Ship It! podcast episodes that really piqued my interest in Fly.io.
Episode 50: Kaizen! We are flying βοΈ
Episode 59: Postgres vs SQLite with Litestream
Episode 60: Kaizen! Post-migration cleanup
It was "Episode 50: Kaizen! We are flying βοΈ" that I think first introduced me to Fly.io, pretty sure I hadn't heard of Fly.io before then. And then along came episodes 59 and 60 which further solidified my interest in the service just as I was thinking about switching away from Juju, especially episode 59 with Ben Johnson, the creator of Litestream, who now works for Fly.io.
Litestream: Streaming replication for SQLite
What's interesting about Fly.io is that it takes a Dockerfile and then converts it to run in a very lightweight Firecracker MicroVM. With the Dockerfile format being such as popular and well understood method of deploying applications in the world of software development, this is pure genius. And on top of that, Fly.io's flyctl CLI and a whole bunch more is open source, and they have a very generous free tier that is way more than I need for mucking about with a dinky little static site and exploring the platform for some future projects I may have in mind.
I just had to have a play and see how easy it might be to host all these sites with them.
My core requirements were:
- Deploy a single copy of this static site that all domains use.
- All domains served over HTTPS using a Let's Encrypt certificate.
- The site content gets updated at the same time builds.sr.ht updates ianmjones.com.
Bonus requirement:
- All domains also served over Gemini using same content as ianmjones.com.
Initial Deployment for HTTPS
One of the first docs I found in Fly.io's documentation was a quick start for serving a static site.
Deploy a static website on Fly
This was absolutely perfect, not only because it was a well written guide for doing what I wanted to do, but also because it used a super neat minimal web server written in Go. These days, I'm a big fan of the Go language.
I followed that guide, substituting in my site's own public.html build output as the source of the Dockerfile's COPY command, and voila, I had a copy of my site hosted at ianmjones.fly.dev.
The Dockerfile was very small:
FROM pierrezemb/gostatic COPY ./public.html/ /srv/http/
The fly.toml file needed to deploy the app was generated with the flyctl launch command just as specified in the quick start guide, and only needed the internal_port setting to be updated to change the default value of 8080 to 8043 as used by goStatic. This was all mentioned in the guide.
It was very simple, and the flyctl CLI client was easy for me to install as there's a version in Nixpkgs that I could add to my system's config and have installed with a quick nixos-rebuild.
After that, it was just a case of updating the DNS A and AAAA records for a couple of my domains to point them at the IP address of the Fly.io app, and running a command similar to the following:
flyctl certs add ianmjones.dev
That dutifully added a certificate for the domain to the Fly.io app, and a quick test proved to me that all was well. Then, as I already had a file in my site's repository that listed all the domains, after updating their A and AAAA DNS records, all I had to do was loop over the file and run that flyctl command for each domain, with a little sleep thrown in to be a little nicer. Something like the following:
cat domains.txt | while read DOMAIN do flyctl certs add $DOMAIN sleep 3 done
Because I'd already manually added the certs for a few of the domains, I just made sure they weren't in the actual file I used for that automated run.
Automating Deployment via builds.sr.ht
So at this stage I had all the domains hosted on Fly.io with auto-renewing certs set up, great, but what about automating site content updates?
First I just made sure that the Dockerfile and other bits and bobs were squirrelled away in the repo, and then I added a new "secret" to my SourceHut account for a Personal Access Token I generated in Fly.io especially for use with builds.sr.ht.
Integrating Flyctl: Environment Variables
The builds.sr.ht secret was set as a "file" type, with name "~/.fly_api_token", and contents similar to the following:
set +x export FLY_API_TOKEN="thesupersecretapitoken" set -x
Then in the .builds.yml file in the repo I added the new secret's UUID to the secrets section, and a new task to the end that sourced the .fly_api_token file and used flyctl to deploy the site with the contents built by an earlier task.
image: nixos/unstable
oauth: pages.sr.ht/PAGES:RW
packages:
- nixos.kiln
- nixos.gmnitohtml
- nixos.flyctl
...
secrets:
- d6dddce0-cce3-4ffd-ae12-e08687dc13ba
- a634b767-e6bb-4efe-be10-0105161d7334
tasks:
...
- deploy: |
. ~/.fly_api_token
cd $site
flyctl deploy
...
When you create a secret in your account, SourceHut assigns a UUID that only your account can use in builds, ensuring that the otherwise public nature of build manifests doesn't compromise your secrets.
With that in place and a small change to my site's content to ensure a successful deployment could be recognised, I committed and pushed the changes. Within a few minutes I had a notification from SourceHut that a successful build had completed, and I could see the changes in the Fly.io hosted sites.
I now had all my core requirements covered, with a single deployment of site content to Fly.io being served via HTTPS and updatable via builds.sr.ht. π
Serving Gemini Content
Given the title of this post, you know I couldn't stop there though, I had to enable serving the sites over Gemini too! π
At first I played with getting a Gemini version of my site up and running locally, as up until then I'd only ever seen my site rendered in Amfora (a Gemini browser) once I'd pushed it to sourcehut pages.
Amfora: A fancy terminal browser for the Gemini protocol
I spent quite a bit of time working with Adnan Maolood's Nova Gemini server.
It worked great when I compiled and ran it locally, but I really struggled to get it working within a Docker container. I wanted to get it working in a container so that I could then easily use it with with Fly.io, but while I could get it to build, including as a scratch container after a builder stage, it kept on wanting Amfora to reconnect on a much higher port than the initial Gemini port 1965. I presume this is something to do with certificate generation, as like with most Gemini servers you get TOFU (Trust On First Use) TLS certificates generated for you so that a third party certificate authority isn't required. It may not even have been that, I really don't know, but either way, I couldn't get it to work in a container and it seems to only generate certificates for a domain that you specify as an arg on startup too, which wouldn't scale well when I have 46 domains each therefore requiring separate processes. That was a shame as it would have been nice to have used a minimal Gemini server written in Go along with a minimal HTTPS server also written in Go. π€·
Then I took a look at Drew Devault's gmnisrv program that I believe is used for the sourcehut pages Gemini service.
sourcehut pages Gemini hosting
This again is a simple Gemini server, but this time written in C. After a bit of investigation I found that gmnisrv was already packaged for the distro I tend to use in containers, Alpine Linux. This wasn't too much of a surprise as I know Drew not only likes Alpine Linux but contributes a lot to it, including packaging software. What was a nice surprise though was that gmnisrv is also in Nixpkgs and therefore available on my NixOS machines, as well as many other distros.
gmnisrv also does TOFU certificates, and crucially, has a nice simple ini file that specifies where certificates are to be stored (I'll come back to this in a minute), and also which domains are to be served and where their files are to be found. This ini file format is simple enough that I could easily create what amounts to a template that specifies the core settings at the top, and then I could append entries for each domain pointing to the same files with a for loop very similar to how I added certs to Fly.io.
Here's what the gmnisrv.ini.tmpl file looks like:
# Space-separated list of hosts listen=0.0.0.0:1965 [::]:1965 [:tls] # Path to store certificates on disk store=/data/gemini/certs # Optional details for new certificates organization=Ian M. Jones [localhost] root=/srv/gemini
And here's the gmnisrv.ini.sh script that is used generate a usable gmnisrv.ini file:
#!/usr/bin/env bash
CURR_DIR=`dirname $0`
INIFILE=${CURR_DIR}/gmnisrv.ini
# Start gmnisrv.ini with a clean slate.
cat ${CURR_DIR}/gmnisrv.ini.tmpl > ${INIFILE}
# Let's do the simplest thing possible to append domains ...
cat ${CURR_DIR}/domains.txt | while read DOMAIN
do
echo "" >> ${INIFILE}
echo "[${DOMAIN}]" >> ${INIFILE}
echo "root=/srv/gemini" >> ${INIFILE}
done
exit 0
It was then a cinch to create a Dockerfile for serving the Gemini content:
FROM alpine
RUN apk update && \
apk add gmnisrv && \
mkdir -p /data/gemini/certs && \
chown gmnisrv:gmnisrv /data/gemini/certs
COPY ./gmnisrv.ini /usr/etc/
COPY ./public.gmi/ /srv/gemini/
EXPOSE 1965
USER gmnisrv
ENTRYPOINT ["gmnisrv"]
And that could be brought up with something like:
docker build -t ianmjones/gmi:latest -f Dockerfile.gmi . docker run -d -p 1965:1965 --name ianmjones-gmi ianmjones/gmi
That worked fine, I could use amfora to navigate to gemini://localhost/ and view the Gemini version of my site locally, yay! π
Then came the head-scratcher moment, how do I get Fly.io to serve both HTTPS and Gemini on the same domain names?
My first instinct was to try and see if I could add a second app using my new Dockerfile.gmi, and then tell Fly.io to route port 1965 (gemini) to that app rather than the current ianmjones app that handles ports 80 (http) and 443 (https). However, I couldn't find an easy way to do that without building a new version of the ianmjones app that acts as a proxy server. Fly.io does have the concept of a "replay header" that allows your app to respond to a request to tell Fly.io to replay the request on another app or instance, but that just seemed like overkill for this use-case.
The only sane option really was to update the current app that serves the HTTP site with a goStatic process, to also run a second gmnisrv process for serving Gemini content.
So that's what I did.
I created a Dockerfile that built goStatic (as it's not packaged for alpine) in a builder stage, and then in the final stage copied across goStatic from the builder stage, installed gmnisrv, copied in the static files and configs for both the html and gmi based sites, and then used a very naΓ―ve shell script as the entry point that started both goStatic and gmnisrv.
The Dockerfile:
# Build
FROM golang:alpine AS builder
WORKDIR /go/src/github.com/PierreZ/
RUN apk update && \
apk add git && \
git clone https://github.com/PierreZ/goStatic.git
WORKDIR /go/src/github.com/PierreZ/goStatic
RUN go build -ldflags="-s" -tags netgo -installsuffix netgo -o ./goStatic
# Dist
FROM alpine
WORKDIR /
COPY --from=builder /go/src/github.com/PierreZ/goStatic/goStatic /usr/bin/goStatic
RUN apk update && \
apk add gmnisrv && \
mkdir -p /data/gemini/certs && \
chown nobody:nobody /data/gemini/certs
COPY ./entrypoint.sh .
COPY ./gmnisrv.ini /usr/etc/
COPY ./public.html/ /srv/http/
COPY ./public.gmi/ /srv/gemini/
EXPOSE 8043
EXPOSE 1965
USER nobody
ENTRYPOINT ["./entrypoint.sh"]
The entrypoint.sh shell script:
#!/usr/bin/env sh /usr/bin/goStatic & /usr/bin/gmnisrv & # Wait until first background process exits. wait -n # Exit with status from first background process to exit. exit $?
After testing it locally with something like ...
docker build -t ianmjones/static:latest . docker run -d -p 8043:8043 -p 1965:1965 --name ianmjones-static ianmjones/static
... having previously stopped and removed the docker containers I'd been using for testing, I was ready to commit and push the new setup to git.sr.ht to trigger a build and deploy to both sourcehut pages and Fly.io.
Well, first I need to make sure that Fly.io knew how to handle requests on the Gemini port of 1965, as up until then it only knew about ports 80 and 443.
To accomplish that, all I had to do was duplicate the [[services]] section in the fly.toml file that deals with internal port 8043, and update the new section to use internal port 1965, and handle any tcp connection on external port 1965.
[[services]]
http_checks = []
internal_port = 1965
processes = ["app"]
protocol = "tcp"
script_checks = []
[services.concurrency]
hard_limit = 25
soft_limit = 20
type = "connections"
[[services.ports]]
port = 1965
[[services.tcp_checks]]
grace_period = "1s"
interval = "15s"
restart_limit = 0
timeout = "2s"
With that in place, committed and pushed, I had a working Fly.io app that served my static site via both HTTPS and Gemini protocols.
One little thing that I then improved was to make sure gmnisrv saved the certificates it generated on first use to a volume mounted into the MicroVM.
This was easy enough. All I had to do was create a 1Gb volume on Fly.io:
flyctl volumes create ianmjones_data --region lhr --size 1
A 1Gb volume is way bigger than I need, I could have got away with just 100Mb, maybe less. But from what I can see, it's only possible to create volumes with an integer for the size, so 1Gb is the minimum.
Then I told the app to mount the volume as /data by adding the following to the fly.toml file:
[mounts] source="ianmjones_data" destination="/data"
In the gmnisrv.ini file that I talked about earlier you may have noticed that it didn't use the standard path of /var/lib/gemini/certs for the tls store, instead it used /data/gemini/certs, and the Dockerfile made sure its path existed and was owned by the user that runs gmnisrv. Now you know why! π
You can check out the current config for my site on git.sr.ht.
The most important parts for the Fly.io setup are the Dockerfile and fly.toml files.
https://git.sr.ht/~ianmjones/ianmjones.com/tree/master/item/Dockerfile
https://git.sr.ht/~ianmjones/ianmjones.com/tree/master/item/fly.toml
Future Improvements
The Dockerfile could do with a bit of optimization, there really isn't any need for running apk update, at least not twice, and it could likely install gmnisrv in the builder stage and just copy it into the final stage if it's a static binary (I'll have to check). Even without moving the gmnisrv install step into the builder stage, I think it can be optimized a little by not running the apk update at all, and not caching during the apk add step. I'll have to test those things out locally.
Talking of local dev, I could likely make the local dev experience better by not rebuilding the entire Docker image when wanting to refresh the local content. I'll probably change things up so that local dev just mounts the public.* output dirs into the running container for instant refresh.
Another thing I could improve in the local dev experience is to mount a local dir into the container for gmnisrv to save certificates to. At present whenever the image is rebuilt I get a warning when reconnecting to the local Gemini site that the certificate has changed. This isn't the case in the version deployed to Fly.io because of the mounted volume.
That Was Fun
All in all, that was a fun little diversion for a few mornings. And I'm pretty sure that I'll be deploying my next hosted project to Fly.io as the experience was awesome, and there's a lot more to the platform than I've used so far.
---
"Running HTTPS and Gemini sites on Fly.io" was published on July 20, 2022.