How to replicate a ddply behavior that uses a custom function with dplyr?

As shown in ?do, you can refer to a group with . in your expression. The following will replicate your ddply output:

iris %>% group_by(Species) %>% do(.[1:5, ])

# Source: local data frame [15 x 5]
# Groups: Species
#
#    Sepal.Length Sepal.Width Petal.Length Petal.Width    Species
# 1           5.1         3.5          1.4         0.2     setosa
# 2           4.9         3.0          1.4         0.2     setosa
# 3           4.7         3.2          1.3         0.2     setosa
# 4           4.6         3.1          1.5         0.2     setosa
# 5           5.0         3.6          1.4         0.2     setosa
# 6           7.0         3.2          4.7         1.4 versicolor
# 7           6.4         3.2          4.5         1.5 versicolor
# 8           6.9         3.1          4.9         1.5 versicolor
# 9           5.5         2.3          4.0         1.3 versicolor
# 10          6.5         2.8          4.6         1.5 versicolor
# 11          6.3         3.3          6.0         2.5  virginica
# 12          5.8         2.7          5.1         1.9  virginica
# 13          7.1         3.0          5.9         2.1  virginica
# 14          6.3         2.9          5.6         1.8  virginica
# 15          6.5         3.0          5.8         2.2  virginica

More generally, to apply a custom function to groups with dplyr, you can do something like the following (thanks @docendodiscimus):

iris %>% group_by(Species) %>% do(mm(.))