One thing for sure is when he shoots a deer that don't drop in its tracks we know where it is most likely going to run. But that works both ways.
This is true. In most cases, a deer is going to attempt to seek safety in the cover from which it came.
Cover is one of the basic requirements of good deer habitat, and it's obvious that you have it while your neighbor doesn't. But with deer being the cash crop that's in relative high demand, there's no shortage of those willing to lease open pasture with the explicit intent of hunting fences of neighboring properties with good deer habitat.
Your only recourse is to inform your neighbors that it would be illegal for anyone, including a game warden, to retrieve any game or non-game animal that crosses the fence after being shot without landowner permission. It becomes your property, clear and simple. Just as your neighbor has the right to hunt near the fence, you have the right to keep anything that falls on your side.
If you have no interest in working out an agreement with your neighbors, I would suggest adding several posted signs.
Such things brings into question the ethics of those who will hunt a stand and feeder with such a high likelihood that breaking the law will be required to retrieve a deer.