It is a fact that Southern troops numbering several thousand would not cross the Potomac into Maryland during the Gettysburg Campaign. It was of such grave concern for General Lee that he issued strict orders on desertion and straggling, which included execution. 

They only wished to protect their own homes, not invade the homes of Northerners…All they wanted was to be left alone.