I do this too, but nobody’s ever noticed. I do it a lot less when I have my headphones on.