I've only had milspec armor on one character, and that was primarily for background purposes, or the rare case when they were going in expecting a drekstorm. He had other armor for his day-to-day work.
Really, even in the pinkest of Pink Mohawk campaigns, wearing milspec in any situation where you're not A) on a military base, B) storming a stronghold where 'shock and awe' are necessary to even have a prayer of coming out alive, or C) taking on magical threats like toxic shamans or bugs, it just draws too much attention.