When the outage detector goes offline
Downdetector is one of those sites you probably only think about when something is broken. It tracks outages for big online services like Discord and lets you see if a problem is on your side or if half the internet is on fire.
But this week something pretty ironic happened. A major Cloudflare outage hit a bunch of services and in the chaos Downdetector itself went down. The site that tells you when things are broken was also broken.
So what do you do when your outage detector is offline? You turn it into a joke and build an outage detector for the outage detector.
That is exactly what happened and the result was a very nerdy and very funny chain of websites that feels like watching the internet look into a mirror over and over again.
The birth of Downdetectors Downdetector
Gus Owen, the creator of the web game TimeGuessr, decided to have some fun with the situation. TimeGuessr is a browser game a bit like GeoGuessr but instead of just guessing where a photo was taken you also guess what year it was taken in. Clearly he already likes creative twists on simple ideas.
His idea this time was simple. If Downdetector can go down then we obviously need a separate site to tell us when Downdetector is down. So he launched a site called Downdetectors Downdetector. Its entire purpose is to tell you whether Downdetector itself is up or not.
The site is mostly satirical but that was the whole point. It landed on the front page of the Hacker News forum where people happily joined in on the joke. Whether it perfectly tracks the real status of Downdetector is not really the main concern. The fun is in the concept itself.
Of course the internet was not going to stop there. Once Downdetectors Downdetector existed someone quickly created Downdetectors Downdetectors Downdetector. This one checks whether Downdetectors Downdetector is up.
Then someone else stepped in and added another layer with Downdetectors Downdetectors Downdetectors Downdetector. That one tells you if Downdetectors Downdetectors Downdetector is up or down.
At the time the article was written the status of the stack of detectors was:
- Downdetector: up
- Downdetectors Downdetector: up
- Downdetectors Downdetectors Downdetector: up
- Downdetectors Downdetectors Downdetectors Downdetector: also up probably
The only problem is that there is no fifth level site to tell us if the fourth one is really working worldwide. So we just have to trust it. Very uncomfortable for anyone who lives on uptime graphs and status pages.
How far can you take the joke
The Hacker News crowd did not just laugh at the stacking detectors. They also nerd sniped themselves with a very real technical limit. One commenter pointed out that you cannot just keep adding more and more Downdetectors into the same domain name forever.
There is a rule in the domain name system. The part between two dots in a domain name can only be up to 63 characters long. For example in www.example.com the word example is the part that is limited.
If you try to register a domain like:
www.downdetectorsdowndetectorsdowndetectorsdowndetectorsdowndetector.com
you run into that limit. That specific string is exactly one character too long. So the most excessive version of the joke domain cannot actually exist in that exact form.
There are workarounds though. You could drop one letter to shorten it or use subdomains. For example you might have something like:
- downdetectors.downdetectorsdowndetectorsdowndetector.com
Then the chain continues but it is split across different levels of the domain structure instead of being one huge word in front of dot com. It is a fun reminder that even internet jokes are still bound by low level technical rules made decades ago.
The serious outage behind the silliness
All of this started because Cloudflare had a serious problem. Cloudflare sits in front of a massive number of websites handling traffic routing security and performance. When Cloudflare has an issue the ripple effects are huge.
In this case the outage was not caused by a cyberattack or anything dramatic like that. According to Cloudflare it came from a change to database permissions. In other words someone updated who or what is allowed to do certain things in a database system and that change accidentally broke important parts of their infrastructure.
Cloudflare published a detailed write up on its blog explaining what went wrong and how they fixed it. It is full of technical detail for people who like postmortems. The bigger takeaway for everyone else is a little unsettling.
Our everyday lives lean very heavily on a stack of online services that all depend on each other. A lot of the time everything works and we forget about it. But every now and then a single misconfigured setting somewhere inside a huge provider can make a lot of the internet stumble at once.
The Downdetectors Downdetector chain is funny because it turns that fragility into something playful. It is the internet collectively saying if everything is going to break sometimes we might as well have fun watching the monitors together.
So the next time your favorite app refuses to load you might end up checking Downdetector. And if that ever fails again do not be surprised if an even more ridiculous Downdetectors Downdetectors Downdetectors Downdetectors Downdetector pops up to keep score.
Original article and image: https://www.pcgamer.com/software/platforms/after-downdetector-itself-went-down-someone-made-a-downdetector-downdetector-and-then-someone-made-a-downdetector-downdetector-downdetector-and-then/
