Huh. I kinda have that problem with left and right, or used to: if someone just asks me where left is, or right is, I can answer. But if I have to think about it, I get muddled, though not nearly so much anymore. I also don't drive much, if ever; when I started to seriously do so to work on getting my license, I mostly solved the problem, because I just never needed to worry about such directionals otherwise.

I do tend to flip numbers around at work (like telling someone they have $10.40 instead of $10.04, or telling them their paid amount instead of their change), but I still usually give them the correct money. But I'm also horrible at math, so there you go.