Custom system configuration fields

So the easy but not recommended way is to put your element under app/code/local/Varien/Data/Form/Element/. As Magento is simply using the order of include_paths and will first look into the local folder, before it goes to the lib folder.

But the wiring to Varien_Data_Form_Element is just a fallback which is defined in lib\Varien\Data\Form\Abstract.php.

public function addField($elementId, $type, $config, $after=false)
{
    if (isset($this->_types[$type])) {
        $className = $this->_types[$type];
    }
    else {
        $className = 'Varien_Data_Form_Element_'.ucfirst(strtolower($type));
    }

So the question is where can you defined that _types array? There is an addType method in the same class which can be called. When you search for that method you might stumble over an iteration of _getAdditionalElementTypes on the form block in app\code\core\Mage\Adminhtml\Block\Widget\Form.php

protected function _addElementTypes(Varien_Data_Form_Abstract $baseElement)
{
    $types = $this->_getAdditionalElementTypes();
    foreach ($types as $code => $className) {
        $baseElement->addType($code, $className);
    }
}

The default implementation of _getAdditionalElementTypes just returns an empty array, but there is already some working implementation in the Magento Core in Mage_Adminhtml_Block_Sales_Order_Create_Form_Abstract.

So that example looks like:

protected function _getAdditionalFormElementTypes()
{
    return array(
        'file'      => Mage::getConfig()->getBlockClassName('adminhtml/customer_form_element_file'),
        'image'     => Mage::getConfig()->getBlockClassName('adminhtml/customer_form_element_image'),
        'boolean'   => Mage::getConfig()->getBlockClassName('adminhtml/customer_form_element_boolean'),
    );
}

And even in your form which you need to extend: Mage_Adminhtml_Block_System_Config_Form

protected function _getAdditionalElementTypes()
{
    return array(
        'export'        => Mage::getConfig()->getBlockClassName('adminhtml/system_config_form_field_export'),
        'import'        => Mage::getConfig()->getBlockClassName('adminhtml/system_config_form_field_import'),
        'allowspecific' => Mage::getConfig()
            ->getBlockClassName('adminhtml/system_config_form_field_select_allowspecific'),
        'image'         => Mage::getConfig()->getBlockClassName('adminhtml/system_config_form_field_image'),
        'file'          => Mage::getConfig()->getBlockClassName('adminhtml/system_config_form_field_file')
    );
}

Based on that you should be able to implement your own _getAdditonalFormElementTypes by rewriting the System_Config_Form which can then handle new frontend types in your config.

€: Thinking about the question of Fabian it should also be possible by using an observer. The addType method on the form element is public, so you can call that and add custom types from and observer. You need to listen to an event in the form object and in the observer add your custom types. Just make sure this happens before _initFields, which is called from initForm. So a good point can be the prepare_layout events which are called before the initForm method (see Mage_Adminhtml_Block_System_Config_Edit->initForm and Mage_Core_Block_Abstract->setLayout)

I don't agree with above solution, you should not rewrite the System_Config_Form because if everybody follow a similar approach you will have a mess (aka: a lot of conflicts between modules).

A better practice is to use the xml tag <frontend_model> in your system.xml config file.

The following example cover the scenario where you want a full custom field otherwise the are

BLOCK FILE

<?php

class COMPANY_MODULE_Block_Adminhtml_System_Myfield extends Mage_Adminhtml_Block_Abstract
    implements Varien_Data_Form_Element_Renderer_Interface
{
    // this will look in adminhtml theme
    protected $_template = 'my/template/myfield.phtml'; 

    public function render(Varien_Data_Form_Element_Abstract $element)
    {
        return $this->toHtml();
    }
}

TEMPLATE FILE
Just put whatever html you want here

XML FILE ( the only thing special here is the <frontend_model> tag

<?xml version="1.0"?>
<config>
    <sections>
        <MYMODULE translate="label" module="translator">
            <groups>
                <general>
                    <label>GROUP TITLE</label>
                    <sort_order>999</sort_order>
                    <show_in_default>1</show_in_default>
                    <show_in_website>1</show_in_website>
                    <show_in_store>1</show_in_store>
                    <fields>
                        <my_custom_field translate="label">
                            <frontend_model>MODULE/adminhtml_system_myfield</frontend_model>
                            <sort_order>40</sort_order>
                            <show_in_default>1</show_in_default>
                            <show_in_website>1</show_in_website>
                            <show_in_store>1</show_in_store>
                        </my_custom_field>
                    </fields>
                </general>    
            </groups>
        </MYMODULE>
    </sections>
</config>

NOTE
If you need to save some values you will have to deal with the <backend_model> tag
Here some info: http://alanstorm.com/magento_system_config_validation