I honestly don't think there's a big difference between 380 and 9mm. 9x17 vs 9x19 in measurement. Same bullet width, slight speed/weight difference.
If a 100% jump in kinetic energy is not a big difference to you, what is?
.380 -> 9mm ~= 100% KE difference
9mm -> .40 ~= 30-40% KE difference
9mm +P -> .40 ~= 10-20% KE difference.
I hope this posts well, but here's a little table comparing defensive loads of all 3:
CAL GR MV KE (J) KE (ftlbf)
9+p+ 127 1250 597.31 440.57
9+p 124 1180 519.71 383.34
9+P 115 1350 630.87 465.33
9+P+ 115 1400 678.47 500.44
40 155 1205 677.46 499.69
40 165 1150 656.83 484.48
40 180 1050 597.35 440.60
40 180 990 531.03 391.69
40 165 1140 645.46 476.09
380 95 1000 285.96 210.92
While I'll grant that KE heavily favors light and fast, it's a good illustration with real numbers. The top two 9mm, bottom two .40 and the 380 are all from the same Winchester Ranger T line.