I know what you mean, I experienced the same thing with mine and then the hogs started owning it. I learned when you think of the time and expense we put into our hunting opportunities it's well worth the time and money to put the odds much more in your favor. I know your area well as I had a 1000 acre lease between Graham and Woodson for 5 years.
Since adding the pen and doing it right this time, I always have shooters consistently visiting the feeder.