The fact that this can be abused is quite alarming to be honest. Of course one could create a throwaway just to avoid this, but imagine if some admin of a different instance decide to ban you with a pretext for seeing a downvote on a post saying trans people are not people (which they might have created just to attract people and see who does downvote). Downvoting/Upvoting detailed info should not be visible to anyone imho to keep it transparent.