Think you’re immune to judgment? You might be surprised. One study found that, subconsciously, people tend to perceive those who buck stereotypes as threats.

First, the researchers tested participants for racial stereotyping and found that they tended to connote being white with having a high socioeconomic status and being Latino with having a low socioeconomic status. They then tested what happened when participants encountered the opposite pairings. Participants met with confederates who either fulfilled stereotypes or countered them (meaning they were white and poor, or Latino and rich), gave a speech about working together, and did a word-finding task with their confederate partners.

Overall, participants with surprising confederates had greater “threat” responses. (In a “threat” state, people believe they can’t deal with the challenge at hand, and experience a series of related cardiovascular reactions.) They also performed more poorly and gave the confederates lower ratings, too.

But were participants really responding to surprise, or were they fighting against a change in the familiar social hierarchy? To find out, the researchers ran another experiment with confederates who had surprising accents (Asians had southern accents, for example) instead of socioeconomic statuses. The effect remained: people paired with surprising confederates had greater threat reactions and performed more poorly than those paired with stereotypical confederates (whites with local accents).

The good news, though: the more familiar we become with people who go against our stereotypes, the less threatened we feel—and the more positively we judge them.