My experience is deer do not like being around hogs. A feeder on my place without a pen, will guarantee hogs and few deer at those feeders. The penned feeders have very few hog sightings. I don't want hogs screwing up my deer hunting, so I pen the feeders and don't let the corn reach the outside of the pen. As mentioned, a larger pen is more comfortable for the deer. More will get in a larger pen.
On a side note, I learned to keep corn from being thrown outside the pen years ago to keep sheep and goats from stampeding to the feeder on a lease I was on. Cattle can behave the same way, and they can be very destructive to your pen if they get a taste of corn. If you have livestock, by all means keep the corn inside the pen. If you want to see more deer, I'd also keep the corn inside the pen to eliminate the hogs. If you want to hunt hogs, use a different location with no pen, or hunt them in the off season.
Another conundrum is the hog panels do not keep 100% of the hogs out. My neighbor and I have what we call "jumpers". We have had a few hogs that are agile enough to jump the 34" hog panel. I ended up adding two strands of barbed wire around the top to deter the "jumpers". So far, so good, but makes stepping over the pen a little more risky.