Passing C++ vector to Numpy through Cython without copying and taking care of memory management automatically

I think @FlorianWeimer's answer provides a decent solution (allocate a vector and pass that into your C++ function) but it should be possible to return a vector from doit and avoid copies by using the move constructor.

from libcpp.vector cimport vector

cdef extern from "<utility>" namespace "std" nogil:
  T move[T](T) # don't worry that this doesn't quite match the c++ signature

cdef extern from "fast.h":
    vector[int] doit(int length)

# define ArrayWrapper as holding in a vector
cdef class ArrayWrapper:
    cdef vector[int] vec
    cdef Py_ssize_t shape[1]
    cdef Py_ssize_t strides[1]

    # constructor and destructor are fairly unimportant now since
    # vec will be destroyed automatically.

    cdef set_data(self, vector[int]& data):
       self.vec = move(data)
       # @ead suggests `self.vec.swap(data)` instead
       # to avoid having to wrap move

    # now implement the buffer protocol for the class
    # which makes it generally useful to anything that expects an array
    def __getbuffer__(self, Py_buffer *buffer, int flags):
        # relevant documentation http://cython.readthedocs.io/en/latest/src/userguide/buffer.html#a-matrix-class
        cdef Py_ssize_t itemsize = sizeof(self.vec[0])

        self.shape[0] = self.vec.size()
        self.strides[0] = sizeof(int)
        buffer.buf = <char *>&(self.vec[0])
        buffer.format = 'i'
        buffer.internal = NULL
        buffer.itemsize = itemsize
        buffer.len = self.v.size() * itemsize   # product(shape) * itemsize
        buffer.ndim = 1
        buffer.obj = self
        buffer.readonly = 0
        buffer.shape = self.shape
        buffer.strides = self.strides
        buffer.suboffsets = NULL

You should then be able to use it as:

cdef vector[int] array = doit(length)
cdef ArrayWrapper w
w.set_data(array) # "array" itself is invalid from here on
numpy_array = np.asarray(w)

---

Edit: Cython isn't hugely good with C++ templates - it insists on writing std::move<vector<int>>(...) rather than std::move(...) then letting C++ deduce the types. This sometimes causes problems with std::move. If you're having issues with it then the best solution is usually to tell Cython about only the overloads you want:

 cdef extern from "<utility>" namespace "std" nogil:
    vector[int] move(vector[int])

When you return from doit, the WhyNot object goes out of scope, and the array elements are deallocated. This means that &WhyNot[0] is no longer a valid pointer. You need to store the WhyNot object somewhere else, probably in a place provided by the caller.

One way to do this is to split doit into three functions, doit_allocate which allocates the vector and returns a pointer to it, doit as before (but with an argument which receives a pointer to the preallocated vector, anddoit_free` which deallocates the vector.

Something like this:

vector<int> *
doit_allocate()
{
    return new vector<int>;
}

int *
doit(vector<int> *WhyNot, int length)
{
    // Something really heavy
    cout << "C++: doing it fast " << endl; 

    // Heavy stuff - like reading a big file and preprocessing it
    for(int i=0; i<length; ++i)
        WhyNot->push_back(i); // heavy stuff

    cout << "C++: did it really fast" << endl;
    return WhyNot->front();
}

void
doit_free(vector<int> *WhyNot)
{
    delete WhyNot;
}