When to use Provider.of<X> vs. Consumer<X> in Flutter

It doesn't matter. But to explain things rapidly:

Provider.of is the only way to obtain and listen to an object. Consumer, Selector, and all the *ProxyProvider calls Provider.of to work.

Provider.of vs Consumer is a matter of personal preference. But there's a few arguments for both

Provider.of

• can be called in all the widgets lifecycle, including click handlers and didChangeDependencies
• doesn't increase the indentation

Consumer

• allows more granular widgets rebuilds
• solves most BuildContext misuse

For your questions:

1. Is this the correct way to distinguish Provider.of<X> and Consumer<X>. Former doesn't update UI, latter does?

Provider.of<X> depends on value of listen to trigger a new State.build to widgets and State.didChangeDependencies for StatefulWidget.

Consumer<X> always update UI, as it uses Provider.of<T>(context), where listen is true. See full source here.

2. If listen isn't set to false will the widget be rebuilt by default or not rebuilt? What if listen is set to true?

Default value is true, means will trigger a new State.build to widgets and State.didChangeDependencies for StatefulWidget. See full source here.

static T of<T>(BuildContext context, {bool listen = true}).

3. Why have Provider.of with the option to rebuild the UI at all when we have Consumer?

Pretty much covered by Rémi Rousselet's answer.

There should not be any performance concern by using it, moreover, we should use consumers if we want to change some specific widget only on screen. This is the best approach I can say in terms of coding practice.

 return Container(
    // ...
    child: Consumer<PersonModel>(
      builder: (context, person, child) {
        return Text('Name: ${person.name}');
      },
    ),
  );

Like in the above example, we only required to update the value of Single Text Widget so add consumer there instead of Provider which is accessible to others widget as well.

Note: Consumer or Provider update the only reference of your instance which widgets are using, if some widgets are not using then it will not re-drawn.