Drupal - How can I force Composer to install a dev branch over a stable release?

While it will install the latest dev release of the Drupal module/theme to simply do the following:

composer require drupal/bootstrap:3.x-dev

I would argue a best practice when installing a dev version of a module is to make sure you are going to get the commit you want (which might not be the latest), so I would do the following:

composer require drupal/bootstrap:3.x-dev#5670922dea05fffb6506a6cd5d53546b9fae8df9

Or you can manually add it to your require in composer.json (as below) and then on the command line run composer install

"drupal/bootstrap": "3.x-dev#5670922dea05fffb6506a6cd5d53546b9fae8df9",

To get the commit you should go to http://cgit.drupalcode.org/bootstrap/log/ and find the commit you want (the commit hash is the id on the commit's page). btw: the above hash is not a real commit on the Bootstrap project

You can keep the original prefer-stable settings. The @dev in 8.3.*@dev is the stability flag for the given package. The easiest way to force a dev-version of a package is to use a different version constraint. Use 8.3.x-dev instead, this version constraint references to a the 8.x-3.x branch and works independently from prefer-stable and minimum-stability.