I read once that people tend to fall into one of two camps, and it's a recipe for a bad vacation if people from the "do nothing" camp vacation with "do ALL the things" people. I kind of think of it more as a spectrum, though I'd agree you're probably up for some friction if you vacation with people who aren't close enough to you on the spectrum.
Where do you think you land?
I think I prefer a structured enough vacation that I know what I'm doing when (e.g. on Tuesday, I'm going to go hiking at this park) but empty enough that there's still downtime and you're not running from activity to activity. I don't think I'm interested in paying for an actual "do nothing" vacation, because I can do nothing at home, for free.