Trump is bleeding support from the only voters who have stuck with him since 2016