Yeah, I've undoubtedly been guilty of that too, by using a flap disc for the initial clearing of older aluminum stock surface debris, and also for on-the-fly tungsten sharpening.

Now that you mention it, it's undoubtedly not a best-practice.

Otherwise, I'll always have a fairly fresh stainless steel bristle brush handy.