Before tractors, almost the entire food chain involved animal slavery, since farms used draft animals. For that matter, even transportation was based on animal slavery (horses). Many of the sustainable and high performance fabrics like wool, silk, and leather are now replaced by synthetic fossil-derived plastics. But petroleum is also an animal product.
To be vegan, is to choose fossilized animal products and services over fresh.
My issue pertains more to forcibly breeding animals into existence for the sole purpose of exploiting them, and also the often terrible conditions they are kept in. I consider hunting and fishing to be more ethical than industrial animal ag, so long as it is done in a responsible manner (yes, I know this is subjective).
Nature is brutal, but industrial animal ag has absolutely heinous abuses.