• Nymphioxetine@beehaw.org
    8·
    2 years ago

    Did you post this from Mastodon? I wish I could tell where this came from.

    Basically if I understand this right, if you have an instance with a very popular community on it. It is likely that it will need some massive infrastructure scaling if it wants to handle the enormous amount of world wide traffic?

    • dan@upvote.au
      6·
      2 years ago

      I wish I could tell where this came from.

      Isn’t that what this colourful icon in Lemmy is for? It appears to link to the original source of the post or comment:

    • veaviticus@lemmy.one
      5·
      2 years ago

      Yes. If you run the server, then you are the source of truth of that community. All other servers that federate your community query your server to access the community and show it to their users.

      So if you run a server and a community explodes there, you might only have 500 users on your instance, but you might have 50k users reading that community and interacting with it from other Lemmy instances, thus your server needs to scale to 50k users worth.

      And ever more essential, your server is the source of truth of that community. So if your server is hacked or corrupted or deleted, that community is gone. Other instances don’t mirror it (except for temporary caching), so the Lemmy network essentially is a trust network of other people maintaining servers long term (and each inventing a monetary system to pay for it). I still think the network might be better than a centralized system like reddit, but it definitely has a lot of growing and policies that need to be sorted out very soon

      • TheAmorphous@lemmy.world
        2·
        2 years ago

        So are these other servers just routing requests from their users to your server’s community? Or are they actually copying everything over every so often (caching) and serving up the requests themselves? How real time is it, I guess is what I’m asking?

        • veaviticus@lemmy.one
          4·
          2 years ago

          Yeah, apparently I was wrong about this (still learning lemmy and fediverse stuff…). Text content of posts and comments are “synced” to your server and stored in your database there. Then future requests for that content are served from your instance. So its not as bad as I thought it was (the network load should be lower since you aren’t acting as entirely a proxy, more like a cache), but database bloat will be a huge problem (its already a big problem in other federated things like mastodon and matrix, where every server ends up saving everything they want into their own database).

          I’m not sure what happens when the original server goes down, does the federated servers discard that data? Or do we each maintain a forever copy until we want to get rid of it ourselves? There’s also some notes I’ve seen about how servers only incrementally cache federated content (only posts and comments that are viewed by someone are fetched, and new comments may not be fetched until someone wants to see it)… so not everybody has a “pure and full” copy of posts necessarily.

          But overall I wonder how all the various sysadmins hosting these lemmy instances will deal with the expotential growth they’re going to see, or if smaller instances will start defederating to save on hardware costs (no reason for my tiny instance that only talks about blue shiny rocks to federate with lemmy.world and store all that content)

        • aardA
          2·
          2 years ago

          Information about post changes on the originating instance come in pretty much in real time, and get saved in the local database.

          If the local instance is configured with pictrs support images are also cached locally.

      • Corfiot@pleroma.elementality.org
        1·
        2 years ago

        Why do these communities need long-term persistence? You could use a separate archive based on plain web server mirrors for anything worth preserving. Maybe it’s good that communities disappear and coalesce elsewhere, maybe it’s evolution. Maybe being forced to pick and choose what to archive and what to let go is a good thing.

        AP is a very chatty protocol and to handle large world-scale groups requires additions like compressed digest distribution, mirrors and sharding. Threads are already fragmented by design so in the end it may be unworkable to follow large group threads.