Apparently according to players that it doesn't matter if you have toggled hats on or off, hats are toggled on for them on their original team colors.
As a dedicated skinner player, it's not's very bushido of people just running at my non-obvious disguise and hitting me.