Edit: obligatory explanation (thanks mods for squaring me away)…
What you see via the UI isn’t “all that exists”. Unlike Reddit, where everything is a black box, there are a lot more eyeballs who can see “under the hood”. Any instance admin, proper or rogue, gets a ton of information that users won’t normally see. The attached example demonstrates that while users will only see upvote/downvote tallies, admins can see who actually performed those actions.
Edit: To clarify, not just YOUR instance admin gets this info. This is ANY instance admin across the Fediverse.
The fact that this can be abused is quite alarming to be honest. Of course one could create a throwaway just to avoid this, but imagine if some admin of a different instance decide to ban you with a pretext for seeing a downvote on a post saying trans people are not people (which they might have created just to attract people and see who does downvote). Downvoting/Upvoting detailed info should not be visible to anyone imho to keep it transparent.