Have FL sized thousands upon thousands of pieces of rifle brass and have never once lubed the inside of the neck. I use the same case lube as you by the way.
If it would make you feel better, and if you have a bench grinder, you can polish the expander ball. Cotton buffing wheel and some polishing compound. It'll make the steel shine like chrome. Slick as a VS model's azz.
800 Yard Steel Range
Precision Rifle InstructionMemberships and Classes Available