'Warning sign for Trump' as key voting group 'lags' in crucial swing state