Welcome to what is, most likely, the most niche project I've ever undertaken: surtoget.no. What is it about? It is about my dissatisfaction with this one train line in Norway. The one called Sørlandsbanen, aka Sørtoget, aka Surtoget.
Why am I dissatisfied? Because it is late every time I try to visit my family, or my family wants to see me. There's always something that goes wrong, often resulting in lengthy delays.
This is not just my opinion. According to BaneNor, it seems to be one of the least reliable lines in Norway. This line alone makes it harder for me to visit my family with my children, and it makes it more of a hassle for my family to visit me, which is why I built surtoget.no - to let off some steam and hopefully get a conversation going.
This post is not about my frustration with sørlandsbanen, though. Instead, it is a technical write-up on the components behind the site and its architecture. Luckily, you don't need to take my word for it, because the entire thing is open source and available on Github!
Guess what? Surtoget is built much like this very website: it is powered by Gleam, with Tailwind making it pretty, and the site is hosted on Fly. It has become my go-to stack for creating nearly static sites. If you look at the technologies powering this site, you'll find that Surtoget is built nearly the same way (though I dropped Clojure. It was a fun thing for my personal website, but it complicates things a little too much for sites that don't have a complicated frontend).
This might be a big "duh" moment for me. Whenever someone visits this website, a Gleam function is invoked; the call is routed, and a response is built using Lustre. This response is then converted into an HTML string and returned as a response in Wisp. This is simple enough, but it is somewhat wasteful for purely static pages, like my contact page.
For Surtoget, I remembered this waste and bothered to do something about it! Now we pre-render purely static pages:
let ctx =
Context(
image_cache_subject: cache.data,
delayed_subject: delayed.data,
about_page: render_page(about.render()),
faq_page: render_page(faq.render()),
news_page: news.get_news_articles() |> news.render() |> render_page(),
)
Whenever we need to route a request, we return the pre-rendered pages:
fn route_request(req: Request, ctx: Context) -> Response {
case wisp.path_segments(req) {
//...
["om-surtoget"] -> ctx.about_page
["faq"] -> ctx.faq_page
["news"] -> ctx.news_page
//...
}
}
While it doesn't seem like a huge thing (and it isn't), it is still a small improvement that cuts off some extra milliseconds, and I'll incorporate it into this website as well.
In Surtoget, the news is static, which might seem a bit odd. Remember, Surtoget does not have a database, which was a deliberate decision. In my opinion, we should strive to avoid adding to infrastructure unless it is absolutely necessary. Adding a database brings so much more complexity to a project, and I try to avoid it whenever I can.
This is why the news is hardcoded, which is fine. After all, this is one train line in Norway, and it can go days (if not weeks) between newsworthy events happening. Regardless of a database existing, I'd still be forced to update something somewhere whenever I need to add a new article, and keeping it just in the codebase makes it easier.
The bigger challenge, however, was how to deal with images. The images come from the articles, but I don't want to hotlink directly to the source images. I want to be a good citizen of the web, and therefore, I don't want to impose the monetary burden of hosting content used by my users on other sites. At the same time, I don't want to simply steal images and incorporate them into my project without a proper license.
The solution is a compromise. When the application starts up it starts fetching the images for the articles:
let _ = case pid_actor {
Ok(pid) -> {
news.get_news_articles()
|> list.each(fn(article) {
process.spawn(fn() { load_image(pid.data, article) })
})
Ok(Nil)
}
Error(_) -> Error(Nil)
}
This means that for each instance of the service, the image is queried once. We avoid the issue where each visit creates extra load for someone else, but we also don't permanently store someone else's content within our project. It only exists within a cache and is destroyed with the application.
I hope this is a fair compromise, as the images are not presented as my own either. They are clearly labelled as connected to the article, which is linked to:
As the service starts, it loads each new image in a different process, which populates the cache. In the meantime, a placeholder image is displayed if any visitors visit the page before the cache is populated, or if an image fails to load for some reason.
We also utilise an FFI (Foreign Function Interface) to Elixir, where we use Vix to resize the images to a shared size, as Surtoget rarely displays images as large as the news sites do:
@external(erlang, "Elixir.Vix.Vips.Operation", "resize")
fn resize_ffi(img: Image, scale: Float) -> Result(Image, String)
//...
fn cache_image(article: news.NewsArticle) -> Result(bytes_tree.BytesTree, Nil) {
case fetch_image_from_external_source(article.external_image_url) {
Ok(image) -> {
let image_type = AVIF(80, False)
// Just defaulting to 600 as that is the max size an image will take up with our layout anyway.
let image_max_size = 600
let scale_ratio =
int.to_float(image_max_size) /. int.to_float(get_width(image))
case resize_ffi(image, scale_ratio) {
Ok(resized_image) -> {
to_bit_array(resized_image, image_type)
|> bytes_tree.from_bit_array()
|> Ok()
}
//..
Surtoget has support to check whether Sørtoget is delayed. This is mostly just a fun feature to make someone roll their eyes and sigh if they happen to visit the site whenever the train is delayed (which is frequent):
During the development of Surtoget, I discovered that entur has this funky API that allows me to query for this huge XML file, which contains information about various trips and their status.
The only limitation of this API is that it allows no more than four queries per minute. This limitation is fine for Surtoget, as we don't need to be updated on the second. We only need to be up to date within a few minutes or so.
The main challenge here is that Gleam does not have an XML library which can parse the result, which was both strange and interesting. It can parse JSON just fine and generate HTML images, but XML libraries that can parse XML are non-existent?
The great thing about Gleam is that when it lacks something, we can often utilise something from the JS, Elixir or Erlang ecosystem! And in this case, we have Erlang's Xmerl library. Some minor Erlang wrapper code combined with an FFI, and boom, we got XML parsing!
To keep the delay notification relatively up to date, we need a scheduled job that could execute asynchronously from the rest of the service. Normally, this can be complicated. In the world of Java, we'd start pulling in a library that could do this for us (like Spring) and configure it. Perhaps even involve Shedlock, in case we have a database and multiple instances. Async work can get messy real quick. For Surtoget, I kept it simple: sleep and recursion!
fn scheduler(subject: process.Subject(DelayMessage)) {
wisp.log_info("Running delayed update check...")
// Spawning a new unlinked process to avoid any issues propagating
process.spawn_unlinked(fn() {
update(subject)
wisp.log_info("Update check ran successfully")
})
process.sleep(wait_time_ms)
scheduler(subject)
}
The above is running as a separate process that is spawned when the system starts up.
There are some weaknesses to this approach. For example, if the process crashes, the cached result is never updated. And this can go unnoticed for quite some time. I should probably rewrite this to use a supervisor rather than rolling my own, but this works for the time being.
Time will tell. Currently, it is a highly niche site with a niche audience, where I complain about my issues with a single train line in Norway. Nothing more, nothing less.
In any case, if you're also upset with this one train line in Norway, do head over to surtoget.no and voice your opinion as well!
At least the logo is pretty kick-ass:
