As I said concerning culling of spike deer, I don't think a bad year or a bad start means the animal will forever be small. Sometimes the fawn was born late, due to breeding of a yearling that came into estrus late, or an adult doe that just wasn't bred earlier. I saw a calf elk that was shot in early November that weighed less than 100 pounds, but was perfectly healthy. Whether such an animal survives a really bad winter is questionable, but just the same, it illustrates the vagaries of breeding among big game animals.
As someone said above, going into the winter in good condition makes a great difference, as the better the body condition the more energy can be put into horn or antler growth. But, an animal with the wrong genetics for horn length and mass probably isn't going to be a Boone and Crockett trophy even in the good years.
Many hunters, concentrating on horn or antler size, come to believe there is a relationship between horn size or antler size and genetic fitness, so the larger the horns or antlers the better the animal is genetically. Maybe yes, maybe no. I've seen smaller, more aggressive bucks run off bucks with larger horns and take over a herd, and in at least one case, a smaller buck with straight horns gored a Boone and Crockett buck behind the shoulder, punctured a lung, and killed the buck with larger horns.
Without knowing the nutritional content of the foods the antelope are eating in the Red Desert, we can't know whether those bucks should have bigger horns than the ones growing up where food is consistently good. Historically the genetics for big horns have been present in the Red Desert, but during the drought years of the 1990s the biggest bucks I heard of were coming from areas where there hadn't been much hunting and alfalfa was present. I tend to agree with Big Fin that you can find an antelope with big horns nearly anywhere if conditions are right, but history still shows that the big ones tend to come from the same places year after year.
One other thing we don't know is whether part of the genetic story is a tendency to produce larger horns at a younger age in some places and later in others. During the drought years I certainly saw a lot of bucks at check stations that were over 4 years old but had horn size I'd expect on two and three year old bucks.