Use of undefined type

In State::State, you are using Core before it is actually defined. You can fix this easily in your example by moving the constructor implementation out of the class definition:

class State{
public:
    State(Core* core);
};

class Core{
   // This stays the same...
};

State::State(Core* core)
{
   core->setState();
}

It's much more common in practice to have the implementation of these functions in a separate implementation (.cpp) files, in which case the forward declarations would work as you've expected.

In that case:

// State.h
class Core;

class State{
public:
    State(Core* core);
};

And

// Core.h
#include "State.h"
#include <iostream> //This is probably a good reason to further separate
                    //Core.h into Core.h and Core.cpp

class Core{
public:
    Core(){
        State state(this);
    }

    void setState(){
        std::cout << "setting state" << std::endl;
    }
};

And the implementation file:

// State.cpp
#include "State.h"
#include "Core.h"

State::State(Core* core)
{
   core->setState();
}

You can forward declare a type when you only need to name, e.g., to form pointers, references, function value arguments, or return types. If you substantially use it, e.g., in a function definition or by dereferencing a pointer, you need the definition. The way to deal with the problem about is to declare the member function but not to define it in the class definition of State. Instead, you'd define it once the definition of Core was seen:

State::State(Core* core){
    core->setState();
}