It is indeed a point to be considered. As I said, I've never done it, but when you think about it, if you have enough time to use the bathroom pass to get into the 2nd panel it probably wasn't that crowded anyway (considering that the validity of the bathroom pass is the lenght of a panel, which is 45 minutes). If you stay in Balroom 20 you'll still be taking the place of a real fan being bored during a panel you couldn't care less about, that's how comic con works.
So I agree, it's not the most correct thing to do and everyone should consider that before doing it, but if they do attempt it I believe they'll probably only succeed with panels that you can basically just walk right in.