Be not concerned about the "light" bullet of a 25-06.
The load I use is 52 grains of IMR 4831 or 53 grains of RL 19 under the Barnes 100 grain TSX BT. It's devastating on game. In my Browning B-78 either load will give me a five-shot average right at 3200 fps over the chronograph. The IMR groups are typically around one-inch for five-shot at 100 yards, with the RL 19 loads typically sub one-inch.
Below is a picture of a 100 grain Barnes X-Bullet (the predecessor of the TSX) recovered from a 170 lb. buck shot at 190 yards. The deer was North bound. The bullet entered the left ham and was recovered under the skin of the right shoulder. All of the deer I've shot with the TSX were broadside. None of those bullets were recovered because they were all pass throughs.
Bullet Options: http://www.barnesbullets.com/bullets/tsx/
Load Data: http://www.barnesbullets.com/files/2014/11/25-06Remington.pdf