I don't believe that's correct.

Me either, otherwise I want some royalties for planes flying 20k feet over my house....lol

There are various interpretations and regulations and some specific instances, but you basically own the air rights the to area above your land that you can reasonably use. FAA doesn't allow for normal flying within 500 feet of structures and such. Consideration of air rights in urban areas differ widely than in rural areas. Simply put, you don't own all the air above you or the space that it occupies.

We have Border Patrol that fly over ours quite often.

The regulations are there for the safety of the aircraft as well as those on the ground. The 500 foot rule doesn't apply if an aircraft must make an emergency landing, for example, and must come within that 500 foot. In section 91.3 of the Federal Aviation Regulations, it states that in an in-flight emergency requiring immediate action, the pilot in command may deviate from any rule to the extent required to meet that emergency.

In general an aircraft must be at least 500 foot from any person, other vehicle, structure, etc. The rule doesn't just apply to a structure and doesn't give you ownership of said airspace. You can be sitting on a beach that you don't own but it doesn't make it legal for a guy to buzz you 20 feet over your head. In addition, if the aircraft isn't operating within 500 feet of a noticable structure, person, etc. then they can fly as low as they want over your land and it's perfectly legal. Not necessarily safe, but legal.

Whos? As long as they are high enough it's not illegal

Yep. One and the same.

Imagine them flying and hovering over your house Bill... same thing

Exactly!...If they are flying high enough then I doubt they will be very effective at driving or herding hogs off someones property, and that is the reason there are regulations.