Custom field validation in system.xml

As @Wojtek Naruniec writes, you have to create your own custom validation method in a javascript file and use it in your module configuration field in system.xml file.

Suppose your field as:

<field id="color" translate="label comment" type="text" sortOrder="3" showInDefault="1" showInWebsite="1" showInStore="1">
    <label>Color</label>
    <comment>Exadecimal value, without #: ex. FFFFFF</comment>
</field>

and you would like to check field length (exactly 6 characters).

Create your javascript file,

vendorName/moduleName/view/adminhtml/web/js/validation.js

for example:

require([
    'jquery',
    'mage/translate',
    'jquery/validate'],
    function($){
        $.validator.addMethod(
            'validate-exadecimal-color-length', function (v) {
                return (v.length == 6);
            }, $.mage.__('Field must have length of 6'));
    }
);

then load javascript file in admin configuration page so you have to generate the file

vendorName/moduleName/view/adminhtml/layout/adminhtml_system_config_edit.xml

<?xml version="1.0"?>
<page xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:noNamespaceSchemaLocation="urn:magento:framework:View/Layout/etc/page_configuration.xsd">
    <head>
        <link src="vendorName_moduleName::js/validation.js"/>
    </head>
</page>

Now you can use your validator adding <validate> tag into <field> tag of your system.xml file:

<field id="color" translate="label comment" type="text" sortOrder="3" showInDefault="1" showInWebsite="1" showInStore="1">
    <label>Color</label>
    <validate>validate-exadecimal-color-length</validate>
    <comment>Exadecimal value, without #: ex. FFFFFF</comment>
</field>

Basically, you need to register your custom validation method and then use it for your field in system.xml file.

Define your validation method:

jQuery.validator.addMethod(
    "validate-custom", 
    function (v) {
        return jQuery.mage.isEmptyNoTrim(v) || /^[1-4]+$/.test(v);
     },
    'Please use digits only (1-4) in this field.'
);

And use it for your field in system.xml:

<validate>validate-number validate-zero-or-greater validate-custom</validate>

Search for "validator.addMethod" in the Magento 2 core code, there are a bunch of examples there showing more complex use cases.