Yes, I understand all of that. I know that it helps all the various instance owners. But that’s a problem that has already been solved. Building for scale is not specific or special to Lemmy. There are already entire automation toolsets—things like K8s or Docker Swarm, Terraform and Ansible, and endless documentation and examples on how to use and implement all of this. You’re talking about the greater whole, and what I’m trying to talk about is Lemmy.ml.

I do agree we’re probably talking past each other, though, and that’s alright, that’s how it goes on the Internet sometimes.

I’m referring specifically to Lemmy.ml, which is what the admins (of that instance) have been discussing and posting links to GitHub issues for. You can’t just take ‘everyone’s’ instance and spread it out into one giant working install of Lemmy. Every single instance that wants to handle scale is going to have to be built, managed, and maintained for it. If Lemmy.ml isn’t built to handle scale, then it’s going to go down when traffic spikes. They’re already having problems with their SQL database and traffic levels are basically nothing. You’ll end up with a bunch of users attempting to access any of the communities on Lemmy.ml and being unable to. They will need to go to a different Lemmy instance, which will have all of the same issues that Lemmy.ml will have regarding traffic load, and interact with threads there. The good thing about federation is that they’ll be able to keep using Lemmy on other instances, even if they don’t have access to Lemmy.ml specifically.

I promise I understand what I’m talking about, building for scale on a global level is what I do for a living. I also know something about open source projects, having co-founded Rocky Linux and the Rocky Enterprise Software Foundation and serving as its Director of Operations.

That’s not how this works. Lemmy itself may be open source, but the instance it runs on is not. All the work in work in the world on the Lemmy codebase won’t mean anything if its actual deployment is not built for scale, and that’s not anything anyone but the admins can do anything about.

I want Lemmy to succeed, but I’m highly skeptical of the ability of the instance operators to be able to do so. There’s a great deal of technical sophistication that is required to support a large number of users, and from what I’ve seen, they don’t have it. This isn’t a slight against them in any way, but they freely admit that they lack SQL expertise, and I think I’ve seen some significant gaps in their knowledge on how to horizontally scale. This instance, for example, is all hosted on a single virtual server. There are no load balancers, no database sharding, no fanning out of services onto different servers…security is as well also likely in a shoddy state.

Again, no hate from me, nothing but praise so far. But there are some significant technological gaps here, and I worry their team isn’t large or technically deep enough to fill them. What’s in place at the moment is just waiting to tip over when any amount of traffic starts coming over. For what it’s worth, I have offered my expertise to the admins around networking, security, scale, and automation.

That’s helpful to think of it that way, thank you. Perhaps I will reconsider :)

I’m interested in getting into this, but I think I’d probably end up abandoning it and having it feel like a chore, then feel guilty about not getting it ‘done.’

I just edited the post to include a quote from The Verge on exactly this (the…odd, almost dystopian nature of these headsets), and I agree completely.