I don't think it would hurt the environment to let them go.... (Don't quote me though)
However, if you value your relationship with your neighbors, I would take them far away from your neighborhood before letting them go.
They will find someones garage and set up shop in there.