A starting point for learning how to implement MapReduce/Hadoop in Python?

I would recommend you start by downloading the Cloudera VM for Hadoop which is pretty much a standard across many industries these days and simplifies the Hadoop setup process. Then follow this tutorial for the word count example which is a standard hello world equivalent for learning Map/Reduce

Before that, a simple way to understand map/reduce is by trying python's inbuilt map/reduce functions:

x = [1, 2, 3, 4]
y = map(lambda z: z*z, x]
print y
[1, 4, 9, 16]
q = reduce(lambda m,n : m+n, y)
print q
30

Here the mapper transforms the data by squaring every element and the reducer sums up the squares. Hadoop just uses this to scale large scale computations but you need to figure out your own mapping and reducing functions.


For those who like MOOC as an option there is Intro to Hadoop and Mapreduce on Udacity, made in collaboration with Cloudera. During the course you have a chance to install Cloudera Hadoop Distribution virtual machine locally and perform some map/reduce jobs on sample datasets. Hadoop Streaming is used for interaction with Hadoop cluster and the programming is done in Python.


First, to use Hadoop with Python (whenever you run it on your own cluster, or Amazon EMR, or anything else) you would need an option called "Hadoop Streaming".

Read the original chapter (updated link) of Hadoop Manual to get the idea of how it works.

There is also a great library "MrJob" that simplifies running Python jobs on Hadoop.

You could set up your own cluster or try to play with Amazon Elastic Map Reduce. The later can cost you something, but sometimes easier to run at the beginning. There is a great tutorial on how to run Python with Hadoop Streaming on Amazon EMR. It immediately shows a simple but practical application.

To learn the Hadoop itself I would recommend reading one of the books out there. They say that "Hadoop In Action" is better in covering things for those who interested in Python/Hadoop Streaming.

Also note that for testing/learning things you can run Hadoop on your local machine without having an actual cluster.

UPDATE:

As for understanding Map Reduce (that is how to identify and express different kinds of problems on Map Reduce language) read the great article "MapReduce Patterns, Algorithms, and Use Cases" with examples in Python.