Drupal - How to create entity reference field from custom entity to specific content type

You're looking for the handler_settings setting, which provides configuration for the handler you've chosen, in your case, you want:

$handler_settings = [
  'target_bundles' => [
    'project' => 'project',
  ],
];

As always with fields, widgets and formatters, just create a configurable field in the UI, export it and look at the settings there. You can always copy the settings and other things almost 1:1 from the exported YAML files to a PHP array.

Also you can set it with ->setSetting('handler_settings',['target_bundles'=>['YOUR_CONTENT_TYPE_MACHINE_NAME'=>'YOUR_CONTENT_TYPE_MACHINE_NAME']] ) , you can pass multi value to it. I mean

->setSetting('handler_settings',['target_bundles'=>['YOUR_CONTENT_TYPE_MACHINE_NAME1'=>'YOUR_CONTENT_TYPE_MACHINE_NAME1','YOUR_CONTENT_TYPE_MACHINE_NAME2'=>'YOUR_CONTENT_TYPE_MACHINE_NAME2','YOUR_CONTENT_TYPE_MACHINE_NAME3'=>'YOUR_CONTENT_TYPE_MACHINE_NAME3']] )

So the final solution for your case is :

$fields['project_id'] = BaseFieldDefinition::create('entity_reference')
      ->setLabel(t('Project'))
      ->setDescription(t('The project ID, transaction was created for.'))
      ->setSetting('target_type', 'node')
      ->setSetting('handler', 'default')
      ->setSetting('handler_settings',['target_bundles'=>['project'=>'project']] )
      ->setDisplayOptions('view', array(
        'label'  => 'hidden',
        'type'   => 'project',
        'weight' => 0,
      ))
      ->setDisplayOptions('form', array(
        'type'     => 'entity_reference_autocomplete',
        'weight'   => 5,
        'settings' => array(
          'match_operator'    => 'CONTAINS',
          'size'              => '60',
          'autocomplete_type' => 'tags',
          'placeholder'       => '',
        ),
      ))
      ->setDisplayConfigurable('form', TRUE)
      ->setDisplayConfigurable('view', TRUE);