It’s the same argument when discussing why people need a pickup truck as their daily driver for the one time a year they need to haul a trailer or move a couch. When faced with the possibility of switching from a half-ton to a sedan, suddenly everybody needs to carry their refrigerator with them everywhere.
Matrix has the unfortunate problem right now where all the big clients have matrix.org set as the default homeserver. Yes, it is a decentralized and federated protocol, but I wonder how many users are registered on matrix.org vs other servers.