What is a CUDA context?

The cuda API exposes features of a stateful library: two consecutive calls relate one-another. In short, the context is its state.

The runtime API is a wrapper/helper of the driver API. You can see in the driver API that the context is explicitly made available, and you can have a stack of contexts for convenience. There is one specific context which is shared between driver and runtime API (See primary context)).

The context holds all the management data to control and use the device. For instance, it holds the list of allocated memory, the loaded modules that contain device code, the mapping between CPU and GPU memory for zero copy, etc.

Finally, note that this post is more from experience than documentation-proofed.

essentially, a data structure that holds information relevant to mantaining a consistent state between the calls that you make, e.g. (open) (execute) (close)

This is so that the functions that you invoke can send the signals in the right direction even if you don't specifically tell them what that direction is.