'The last thing we need': Core part of Trump’s base threatens to walk away over key issue