With forewarning about a huge influx of users, you know Lemmy.ml will go down. Even if people go to https://join-lemmy.org/instances and disperse among the great instances there, the servers will go down.

Ruqqus had this issue too. Every time there was a mass exodus from Reddit, Ruqqus would go down, and hardly reap the rewards.

Even if it’s not sustainable, just for one month, I’d like to see Lemmy.ml drastically boost their server power. If we can raise money as a community, what kind of server could we get for 100$? 500$? 1,000$?

    • Dessalines@lemmy.ml
      38·
      2 years ago

      SQL. We desperately need SQL experts. It’s been just me for yeRs, and my SQL skills are pretty terrible.

      • Valmond@lemmy.ml
        2·
        2 years ago

        Put the whole DB in RAM :-)

        Makes me remember optimization, lots of EXPLAIN and JOIN pain, on my old MySQL multiplayer game server lol. A shame I’m not an expert …

    • poVoq@slrpnk.net
      231·
      2 years ago

      There are some SQL database optimisations being discussed right now and apparently the picture resizing on upload can be quite CPU heavy.

      • itsmikeyd@lemmy.ml
        43·
        2 years ago

        SQL dev here. Happy to help if you can point me in the direction of said conversation. My expertise is more in ETL processes for building DWs and migrating systems, but maybe I can help?

      • veroxii@lemmy.world
        211·
        2 years ago

        I’ve been helping on the SQL github issue. And I think the biggest performance boost would be to separate the application and postgresql onto different servers. Maybe even use a hosted postgresql temporarily, so you can scale the db at the press of a button. The app itself appears to be negligible in terms of requirements (except the picture resizing - which can also be offloaded).

        But running a dedicated db on a dedicated server - as close to the bare metal as possible give by far the best performance. And increase it for more connections. Our production database at my data analytics startup runs a postgresql instance on an i9 server with 16 cores, 128GB RAM, and a fast SSD. We have 50 connections set up, and the run pgbouncer to allow up to 500 connections to share those 50. And it seamlessly runs heavy reporting and dashboards for more than 500 business customers with billions of rows of data. And costs us less than US$200pm at https://www.tailormadeservers.com/.

        • Cadende@lemmygrad.ml
          7·
          2 years ago

          And I think the biggest performance boost would be to separate the application and postgresql onto different servers.

          I think hexbear.net (an older lemmy fork-ed site) is working on this in conjunction with moving back to a modern lemmy version

        • poVoq@slrpnk.net
          8·
          2 years ago

          Probably not, but it does mention a more general CUDA based solution that might be interesting to add to Pictrs. I could for example move my Pictrs instance onto a server that does have an older Nvidia GPU to accelerate stuff (to use for Libretranslate and some other less demanding ML stuff).

          Edit: Ok looks like the resizing is anyways only supported on Pictrs 0.4.x which most Lemmy instances are not using yet. However this seems to use regular ImageMagick in the background, so chances are quite high that it can be made to work with OpenCL: https://imagemagick.org/script/opencl.php

    • esturniolo@lemmy.ml
      51·
      2 years ago

      And may be the bandwidth. Serve thousands and thousands need at minimum 1gbps.