I think it depends on what and when, and also, I want to say I mix them all the time!
I almost always where patagonia capilene long john, best by far, then a lightweight fleece over that. I"ll have another fleece in my pack that will serve as backup for real cold, then if I put it on and plop my goretex over the top I'm snug as a bug in a rug, regardless.
In fall/winter, I always where wool pants. I have several different styles of surplus that I where, depending on the weather outlook. Some are heavy, some are light. LOVE wool pants, no way to beat them. I'd love to be able to afford some King of the Mountain wool. I'd still have my fleece on under it though.