Most of the video games I’ve played were pretty good. The only one I can think of that I didn’t like was MySims Kingdom for the Nintendo DS. Dropped that pretty quickly. It was a long while ago, but I’ll guess it was because there were too many fetch quests and annoying controls.
Lots of games had duel modes without downtime, when your duel ends, you get paired up with another player whose duel ended recently. It’s a few seconds at most usually.
I never felt particularly stressed during the collection segment, just bored, and from the other guy who wrote that he likes that time to mess around with his friends, I don’t know that your experience is universal.
That feeling of tension that you describe was absolutely present for me playing traditional deathmatch. The drive to want to win was strong enough to make me give a shit about the game. Especially if it was like, a clan match or something.