2024/11/29

Problem

Nobody can post or follow on TootCat, though the page seems to reload and remote updates are coming in. When trying to post, a "500" error appears in the lower right corner.

Diagnosis

• 08:50 EST Tentatively, the web servers and the database are fine -- but I need to discuss this with our web engineer (Dan) to confirm my understanding and work out a fix. I don't want to go making things worse with a poorly-conceived kluge that doesn't actually work. It looks like the load-balancer/firewall is the only thing affected, but I don't have a firm enough grasp of our architecture to be sure that this is the case or to be sure of how to temporarily work around it.
• 08:14 EST The problem may be a service outage at Hetzner. We're checking possibilities.
  • This was originally "API failure (datacenters & locations)" but later changed to "Failures on API and Console".
  • Updates from Hetzner:
    • 2024-11-29 14:08 UTC+0 It is currently not possible to access the Cloud Console. This is intended to help us recover.
    • 2024-11-29 12:41 UTC+0 Currently it is not possible to create or modify resources in the Hetzner Cloud. Already running resources are not affected.
      • ...except it looks like our firewall/load-balancer is out of commission. -W.