Methods for optimizing multcore processing in ArcGIS

Here is an example of a multicore arcpy script. The process is very CPU-intensive so it scales very well: Porting Avenue code for Producing Building Shadows to ArcPy/Python for ArcGIS Desktop?

Some more general info in this answer: Can concurrent processes be run in a single model?

In my experience the biggest problem is managing stability. If you do six weeks of processing in a single night you will also have six weeks of inexplicable errors and bugs.

An alternative approach is to develop standalone scripts that can run independently and fail without causing problems:

• Split the data into chunks that a single core can process in <20 minutes (tasks).
• Build a standalone Arcpy script that can process a single task and is as simple as possible (worker).
• Develop a mechanism to run tasks. Lots of pre-existing python solutions exist. Alternatively you can make your own with a simple queue.
• Write some code to verify that tasks have been completed. This could be as simple as checking that an output file has been written.
• Merge data back together.