Calculate multiple aggregations on several variables using lapply(.SD, ...)

You're missing a [[1]] or $mpg:

mtcars.dt[, lapply(.SD, function(x) list(mean(x), median(x)))[[1]],
            by="cyl", .SDcols=c("mpg")]
#or
mtcars.dt[, lapply(.SD, function(x) list(mean(x), median(x)))$mpg,
            by="cyl", .SDcols=c("mpg")]
#   cyl       V1   V2
#1:   6 19.74286 19.7
#2:   4 26.66364 26.0
#3:   8 15.10000 15.2

For the more general case, try:

mtcars.dt[, as.list(unlist(lapply(.SD, function(x) list(mean=mean(x),
                                                        median=median(x))))),
            by="cyl", .SDcols=c("mpg", "hp")]
#    cyl mpg.mean mpg.median hp.mean hp.median
# 1:   6    19.74       19.7  122.29     110.0
# 2:   4    26.66       26.0   82.64      91.0
# 3:   8    15.10       15.2  209.21     192.5

(or as.list(sapply(.SD, ...)))

Realized right after clicking "ask" :) The solution is to list the lapplys:

mtcars.dt[, list(mpg.mean=lapply(.SD, mean), mpg.median=lapply(.SD, median)), 
          by="cyl", .SDcols=c("mpg")]
# Solution:
    cyl mpg.mean mpg.median
|1:   6    19.74       19.7
|2:   4    26.66       26.0
|3:   8    15.10       15.2