I think that's probably the case for me also. But for me it's hard to predict that 10% of the time that I need approach angles, clearance, locking differentials and off-road tires. Not to mention the times when I don't need the help, but have to pull others out of the ditch. It makes more sense for me to pay for one loan/insurance/license than try to have two vehicles. So I suck it up and pay for the 20 mpg to always have the option to get somewhere nasty.
My vehicle always has chains, shovel, axe, a place to sleep, food etc. It's nice being able to hop in the rig and know you're good to go no matter the situation.