I did this year and it worked out fine. The jacket I wear hunting is camo so I usually wore it but I didn't wear camo pants until it got colder later in the season. I shot a doe wearing a black rain jacket and grey Walmart outdoor pants this year.I think I am going to start going this route as well, even for whitetail hunting here at home.