I have killed several post-mature bucks during the rut that weren't participating in the activities at all. I've always believed that some older, mature bucks opt out of rutting activities in favor of longevity. But yes, on free range properties does have their core areas and bucks have theirs, most of which do not overlap. These areas change, especially for the bucks, when the rut kicks in. So you could have an area devoid of bucks one week and crawling with them in a few weeks.
Either that, or your bucks bred all your does and went to the neighbors to clean up his