If just a small area try a pencil eraser
This is the trick an old dude at my hometown range told me when I first started shooting. And also the graphite in the pensil is a very mild abrasive so it can really only remove superficial things, like light rust. I usually use a drop of gun oil and rub it gently with a pencil and wipe it off. Then repeat as necessary. It can be a painfully slow process sometimes.