Some of the most fertile ground is rocky, Arkansas is like that. But there is a limit. Usually harder on the equipment too. Size of rock can make a difference I have seen, small is better, table rock type stuff sucks.
You can try what Nav said and see what happens, probably more dirt there than you think....Maybe
Nice thing about rocks though, they tend to help hold moisture well underneath, so you may cone out better than most during dry times.