Just because I can't enter your house with a bag over my head is no reason why I shouldn't be allowed to wear one.

But I agree you have the right to deny me entry

... and I think you can extend that principle as far as it will go.