It is a fact, Southern troops numbering several thousand would not cross the Potomac into Maryland in September of 1862. It was of grave concern for General Lee, so that he issued strict orders on desertion and straggling, which included execution. 

Confederate soldiers refused to join in the invasion because they declared that they were fighting to defend their homes and not to go north and take war to other men’s homes, they did not wish to leave Virginian soil. 

This also happened to a lesser degree during the Gettysburg campaign.