Does it matter? I don't think it usually matters to me, based on how I've made some of my shoot/don't shoot decisions in the past.
Does the number change my mind? Not the number, but the maturity of the animal )which is usually expressed by large antlers that usually equates to a larger number) does. Or, at least if I'm honest about how many younger bucks I've let walk in such a situation.
I can draw from my personal past behaviors to convince myself either Yes or No. I've shot 120" whitetails in a very similar situation, so I struggle to convince myself I'd pass a more mature mule deer in the same situation, though I don't know that I've ever been in that situation with a mule deer over 3 years old.
I passed a couple very nice mule deer bucks while elk hunting central Montana this year. I really can't explain what was going through my head, other than I was enjoying watching them act goofy and rationalized that if I shot one it might mess up the elk situation. Taking one of those would have been more satisfactory to my "earned it" feeling than the situation you pose the question of. But, for some curious reason I'm still reflecting on, I passed.
To me, it doesn't matter what someone else would do. I'd be happy for whatever decision they arrived at.
Right now, at this point in my life, I suspect in most instances I'd pass just due to my pleasure of being out there and waiting for that perfect situation (not necessarily perfect buck). If I get back to a time when I am not hunting as much as I currently do, I suspect I'd be more inclined to take advantage of that situation, even with a smaller buck, as my freezer may be more likely to have space at that time than is the current situation where I get to hunt a lot.
I know the last two years my MT deer tag, and this year's elk tag, went unfilled when opportunity presented and I mostly attribute it to having good luck already in my freezer from hunts in other states. That causes me to honestly answer your question with, "It depends."