How does a vector as a key works internally in C++?

There is an overloaded operator < for the class template std::vector.

template <class T, 
class Allocator>
bool operator< (const vector<T, Allocator>& x, const vector<T, Allocator>& y);

that is based on the standard algorithm std::lexicographical_compare.

Here is a demonstrative program.

#include <iostream>
#include <iomanip>
#include <vector>
#include <iterator>
#include <algorithm>

int main() 
{
    std::vector<int> v1 = { 1, 2 };
    std::vector<int> v2 = { 1, 2, 3 };
    std::vector<int> v3 = { 2 };

    std::cout << std::boolalpha << ( v1 < v2 ) << '\n';
    std::cout << std::lexicographical_compare( std::begin( v1 ), std::end( v1 ),
                                               std::begin( v2 ), std::end( v2 ) )
             << '\n';                                              

    std::cout << std::boolalpha << ( v1 < v3 ) << '\n';
    std::cout << std::lexicographical_compare( std::begin( v1 ), std::end( v1 ),
                                               std::begin( v3 ), std::end( v3 ) )
             << '\n';                                              

    std::cout << std::boolalpha << ( v2 < v3 ) << '\n';
    std::cout << std::lexicographical_compare( std::begin( v2 ), std::end( v2 ),
                                               std::begin( v3 ), std::end( v3 ) )
             << '\n';                                              

    return 0;
}

Its output is

true
true
true
true
true
true

So the class can be used as a key in map.

By default the class template map uses the function object std::less that in turn uses the operator <

template <class Key, class T, class Compare = less<Key>,
class Allocator = allocator<pair<const Key, T>>>
class map 
{
    //...
};

However there is no overloaded operator << for the class template std::vector.

Name of an object and content of that object are always unrelated things.

operator == for std::vector will first compare length of vectors and then each of it's elements using operator == as well.

operator < compares elements in vector lexicographically, i.e. it returns x[i] < y[i] for the first non-equal element in vectors x and y.

These are the requirements std::map has for a type used as Key. Since std::vector satisfies both, it can be used by as Key. Note that type managed by vector must also has these operators overloaded for this to work (because std::vector relies on those operators to implement it's own operators).