I recently spoke with some horse folks here in central Wyoming who had issues with people destroying cairns that marked a trail moving across a high, windswept plateau. Late season, during general rifle, near-whiteout conditions make the trail impossible to follow without these large (~5 ft+) cairns. Of course, these cairns mark an existing, maintained USFS trail, as opposed to an unofficial route.
Personally, when I am off of maintained trails and find cairn-marked routes, I tend to dismantle them as I go...for the same reasons many have expressed on this thread - to avoid increased use that might affect wildlife or the general landscape in a negative way. I think it's important to leave things as we find them. I also think that unofficial trails encourage folks to get into more rugged, remote stuff without necessarily having the navigation skills to back it up.
In Washington state, there is a famous set of hiking guidebooks (100 Hikes in...) that were written way back when by these two fogies named Harvey Manning and Ira Spring. Interesting duo, and really groundbreaking books...at the time, there weren't really guides for things like backcountry travel in the Northwest. The duo eventually ended their business partnership because one of the guys, Harvey, couldn't bear to see what the increased exposure was doing to the Cascades and the Olympics. He'd grown up hiking and climbing in there and seen them go from a literally forgotten wilderness into something that was being irrevocably altered with considerable help from the books they had written. He was eventually unable to morally justify continuing to write the books. Interesting guy...kind of iconoclastic, didn't give a shit what people thought. Not a saint, by any means (though neither was Cactus Ed), but he understood our capacity to destroy what we love, and when enough was enough.