Colt, I'm allergic to Cocobolo dust myself. Breaks me out worse than poison ivy which is why I never use it.
A lot of the Rosewoods are "sensitizer woods" and will make you react worse each time you encounter them.
I know a couple of knifemakers who ended up in the emergency room with constricted airways from inhaling the dust.
My experience has been that it's just the dust that will do that though.
Here's a link to a list of toxic woods...
http://www.bwwt.org/bwwt_toxic_wood_chart.html