Chalk me up to "practice" at proper pressures. You absolutely, without any doubt, can ride gravel and dirt at higher pressures....I ride gravel a LOT and never air down. It's technique.
I spend a lot of time on a light 250 dual sport and even aired down it's as skittish on gravel as a Strom aired down to 20lbs, so the feeling of not feeling quite "planted" is 95% mental. You feel like you don't have traction, and you convince yourself you don't, but in actuality if you point that front end and gas it you will NOT go down.
Mud & sand are a different matter but gravel and hard pack don't require lower pressures to maintain traction, IMHO.
Never met a dog I didn't like, a machine I couldn't break, or a woman I couldn't tick off.