First off, just because it's limited entry, does not mean it'll be less crowded. CPW does a really good job of collecting their revenue every year. For example, the Flattops 1st rifle season is a CF. Do your research and look and see how many tags are issued.
Second, they're completely different hunts, most years. 1st rifle is generally better weather, more people, and the elk are still high in their nasty little hidey holes. 4th rifle, the elk are moving (usually, I have hunted in a t-shirt in 4th before, and had to go high for elk), and more visible, but the weather can be disastrous for those unprepared, and you can run into more private land issues.
Of course these are all generalizations, and each unit can be completely different. Pick your poison.
Personally, I hunt 4th rifle because it's less crowded, and I like using optics w/spot and stalk opportunities, vs. still hunting aspen/timber areas. Also, the NR seem to like 4th rifle less.
1st can be a beautiful time of year to be out there, catching the tail end of the aspens turning, and maybe some bugling. Here is a picture of opening day of 1st rifle back in 2014, the last year I hunted that season.
If you go 4th you gotta be prepared. Towing a trailer in somewhere is risky, camping in a tent is insanity. 4 sets of chains w/ 4wd are a must, hell last year I could have used a snowmobile. I either wall tent it or get lodging, which the longer I do this the more I appreciate lodging in town.
Hope this helps, good luck!