While I agree with your sentiment, there is a difference between not understanding and actively disparaging. The former is fine - there’s plenty of stuff I don’t understand, and I just don’t comment on it because I have no business doing so. Where I take objection is when the lack of understanding transforms into bigotry and disparaging remarks.
By all means be ignorant (and I don’t mean that in a derogatory manner - we are all ignorant about various things), but don’t let your ignorance manifest into negativity.
Part of why cats like you is probably because you don’t like them and seek them out. Most cats are wary of people directly trying to approach them without invitation or greeting. In general cats will find the person ignoring them the least threatening and most approachable.
It never fails that cats tend to flock toward the person that is allergic to them, because that person is typically entirely disinterested in them lol. Not being interested in the cats is likely why you make friends with them so quickly!