Drupal - Proper way to create custom tables

hook_schema() is not used differently from how it was used on Drupal 7, the only difference is that it is not used to create database tables for entities. (See, for example, node_schema().) In particular, you don't need to call drupal_install_schema()) in hook_install() to create custom database tables, which was instead necessary in Drupal 6.

.schema.yml files (for example, user.schema.yml) are used to define the schema for the configuration used in the configuration files from the module. Its purpose is different from defining a database table schema, and they cannot be use for defining custom tables used from third-party modules.

For a database table, you use hook_schema to define it and if the module already exists, you can use hook_update to create the table manually as it hook_schema is only triggered during installation.

This is all like you describe.

The schema config is not used to create database tables, instead it's use to define the schema of configs. Basically you use it to explain the structure of configs your module will save, data types etc. I don't know what actually uses this as most things will work without it.

The two 'schema' have different contexts.

Your assertion of creating database tables is correct.

schema.yml files serve another purpose. From the docs:

Using the knowledge embedded in the configuration schemas about what is stored of a configuration entity, the default persistence implementation for configuration entities requires a configuration schema for the configuration entity, so the right properties are exported with the types defined. Although it is better to provide configuration schemas, if you really don't want, implement the toArray() method in your configuration entity implementation to not require schema for saving configuration entities of your type.

And that is just one reason. See: https://www.drupal.org/docs/8/api/configuration-api/configuration-schemametadata#use