Hey everyone, so as I’m sure everyone is aware Lemmy.World has been experiencing several outages throughout the last few days.

We have been investigating the root cause of these outages but believe that they are related to our current hosting provider (Hetzner) blocking access from ClouldFlare as (we think) they believe that our CDN is a DDoS’er, and is causing these disconnects to our backend server, problematic for sure.

We’ve opened support tickets with our current provider and are awaiting a response. We have no issue with being as transparent as possible with downtime. Anyone that is curious, can feel free to check out https://status.lemmy.world and https://dash.lemmy.world for up to the minute outage information. We are also looking into other fediverse friendly methods of posting status and outage updates

In the meantime, we are evaluating alternative hosting options and solutions to provide a high level of reliability to you, our users. Really, we want to say thanks to everyone for soldiering through all our technical growing pains.

Cheers

  • LW Infra Team
  • Cthuwu@lemmy.worldEnglish
    1372·
    1 year ago

    As always, the transparency is appreciated. Some growing pains are certainly to be expected

  • Machefi@lemmy.world
    421·
    1 year ago

    I don’t blame you for this, but the uptime records are incomplete at best. I’ve experienced the site being down (and confirmed with Down for Everyone or Just Me), yet status.lemmy.world showed all systems operational. As I’m writing this, status.lemmy.world is missing most data up to yesterday and dash.lemmy.world shows 16 days uptime.

    I have lots of respect to you for even having these. I also remember status.lemmy.world work mostly fine some time ago. But as of right now, both uptime monitors fail to serve their purpose.

    • lwadmin@lemmy.worldOPMEnglish
      361·
      1 year ago

      You need to hover over the status bar to see if there is any down time for that day. We can enable it to log incidents every time there is a burp, but we are still tuning alerts as we only have it create a incident when we ACK it in PagerDuty. You can always check the dashboard for up to the minute stats, as well as https://lemmy-status.org/endpoints/_lemmy-world We’ll add this info to make things clearer <3

  • drewthejedi@lemmy.world
    20·
    1 year ago

    Everyone talking about the downtime including lemme and me just enjoying lemmy like never before. I’ve experienced no downtime so far

  • webhead@lemmy.world
    9·
    1 year ago

    Could look into Dacentec if you need more cheap servers. I use them for my stuff. YMMV since you’re getting a hell of a lot more traffic than I am but they haven’t blocked Cloudflare on mine yet so that’s a plus. :)

  • BellaDonna@mujico.org
    2214·
    1 year ago

    Let me be real. I never noticed outages stopping. It feels like it’s daily, I’m used to it, but I think it happens so often that lemmy.world has lost its growth opportunity, and we alienated the normies. I’m still going to stay on Lemmy, and I believe you’re doing the best you can, but we lost for the time being, the migration to Lemmy from Reddit is stunted.

    • PM_ME_YOUR_ZOD_RUNES@sh.itjust.works
      93·
      1 year ago

      We shouldn’t be trying to grow a single instance. That defeats the whole point of Lemmy. I started on Lemmy world and switched once I got fed up with the constant connection issues. Plus, Lemmy world blocked piracy communities so fuck that. I’m happy that I am able to quickly create an account on another instance.

    • HTTP_404_NotFound@lemmyonline.comEnglish
      12·
      1 year ago

      I never noticed outages stopping

      I am in the same boat. If it were not for these posts, I’d have never noticed lemmy world was down.

      If I post to, or put a comment on something from lemmy world, it will just federate over when it’s back online.

      If someone posts to something on lemmy world, it will eventually federate over my way.

      But, hey, everyone is gonna fuss when they decided to put all of their eggs in one basket and now, that one basket gets targeted. (metaphor for lemmy world.)

    • HEISENBERG@lemmy.world
      52·
      1 year ago

      This is about what’s happening the last few days I think? Lemmy World has been a lot more stable compared to even a week ago but still has some daily downtime. But not for hours every day anymore