This is the point of Heather McGee's book "The Sum of Us" -- white people have long been willing to vote against their own interests if they fear people of color might get to enjoy a fair shake.