I think the Rust code is holding people back, and the queuing for federation and the overhead of HTTP connects for each individual comment and posting is too much.

Major Bypass Surgery is an option here. “Lemmy to Lemmy” MTA.

Two Birds with One Stone

  1. The Client API is supposed to be high performance for loading 50 postings at a time, 300 comments at a time, Exercising the Client API is a good thing, identify the performance problems and log how it performs on multiple servers during replication.

  2. Avoiding the Federation path that is currently being used in the Rust code for outbound sending will likely have an immediate improvement in server performance and stability.

  3. Third bird: this tool build an API for the foundation of bulk back-fill between servers.

 

Changes to Rust code

  1. New Client API to query “raw0” instead of “hot” and other sorting options. “raw0” sorting would be the local servers latest INSERT into the posting table without concern for pinned/sticky posts, votes, etc. Can we ORDER BY post_id itself, does that make sense?

  2. Same with comments, can we ORDER BY coment_id itself on a posting?

  3. Can we get the RAW JSON of the POST and COMMENT tables as output. Even if we need a api/v4 or api/bulk0/ path for a different output format?

  4. Do we want to use Rust? Are we confident in the code not binding up the monolithic Rust application and should we consider having an api/bulk0 path into another app that directly accesses PostgreSQL?

  5. Comments and Posts are the bulk concern. Federation of user profiles, community announcement, etc leave alone. Have a switch in the code to turn off SEND/OUTBOUND Comments/posts/votes - but leave everything else flowing through federation for a specific peer instance.

 

api/bulk0

  1. A Query for a community to return a list of all peer instances that needs to have comments and posts delivered. At lest one subscriber on that remote instance.

  2. A Query for an instance and all Communities that need comment and post delivery, have at least 1 subscriber on that remote instance.

  3. Edited posts and Comments, Query all newly edited post/message from a specific post_id and comment_id forward. Perhaps limit query to 300 of each, but allow higher number in a list return format (without the body of he posting/comment).

Incomplete…