These temp files and folders should be cleaned up, they are only required while an instance is active.
These files can be cleaned up without much trouble with the Agent shut down. If there is a file that it cannot find it will recreate the temp files the process requires. This causes the process to be slightly slower, this increase should be slight enough that it is unnoticeable.
If the problem persists please open a support case and we can get the SE team to review.