My family owns 670 acres. We fenced off 2 sides bc of neighbors shooting anything and everything. We tried that for 5 years. Never saw a huge increase in age structure. We fenced the whole place in 1994-5. The biggest hurdle for us was genetics. The deer in the area (eastland cty) aren't known for large antlers. A large buck in our area is a 140 class type buck. To date, we have killed a 169, 3-4 in the 150s several in the 140s and culls. It takes a lot of time. You won't be able to change the genetics in 3-4 years. In 20 years, we still have junk bucks and spikes. Most, if not all, of the spikes are bc of late fawning dates. We have had poor fawn crops over the last 5-6 years which hurts buck recruitment.
No matter what you put out to help keep the deer around, they will come and go especially when the rut hits. I wouldn't worry about "culling" the herd until the fence is finished. Don't expect to see monster deer in 1-2 years. You will want to keep the better bucks to breed. We have left some to die of old age to give them every chance to breed. I would be hesitant to shooting ANY bucks until you know what your buck to doe ratio is and what the overall numbers are on the ranch when fence is done.
Just my 2 cents worth with 20 years experience of managing a "smaller" HF place with native deer.
If you shoot a young deer because a neighbor will shoot it, you are that neighbor.