We had a few whitetail and axis skinned and hanging in the cooler, you couldn't hardly tell the difference when they were side by side. If you looked close you tell a difference in color, but it wasn't true for all of the deer, maybe their diet or how well they bled, who knows.
We have always kept the hides in the cooler or at least photos and we have had some insist on keeping a head.
Although I have never heard of anyone getting questioned one way or the other, I am sure that it has happened.
"Man is still a hunter, still a simple searcher after meat..." Robert C. Ruark