adding a custom icon to a tinyMCE button

this CSS based solution seems to work nicely:

 i.mce-i-[FONT-AWESOME-CLASSNAME]:before {   // FONT-AWESOME-CLASSNAME e.g. "icon-youtube"
    content: "[FONT-AWESOME-CONTENT]";       // FONT-AWESOME-CONTENT e.g. "\f166"
    font-family: FontAwesome;
    font-style: normal;
    font-weight: normal;
    text-decoration: inherit;
    color: #000;
    font-size: 1.5em;
    padding-right: 0.5em;
    position: absolute;
    top: 15%;
    left: 0;
 }

it is based on matt-royal's answer on this stack exchange wordpress thread

I know this is old, but I thought I would throw in my answer for anyone who is interested. I am using TinyMCE 4. I added this to my CSS

.mce-ico.mce-i-fa {
    display: inline-block;
    font: normal normal normal 14px/1 FontAwesome;
    font-size: inherit;
    text-rendering: auto;
    -webkit-font-smoothing: antialiased;
    -moz-osx-font-smoothing: grayscale;
}

then when I am setting the icon for the buttons I just use.

editor.addButton('adjust', {
    tooltip: 'Adjust Layout',
    icon: 'fa fa-adjust',
    onclick: function () {
        dialogLayout(editor, url, settings);
    }
});

basically this will let you use any font awesome icons without having to have a specific class layout for each icon.

Hope this helps someone.

For tinymce 5 you can use this sample:

editor.ui.registry.addButton('customButton1', {
    text: '<span class="fa fa-youtube"></span>',
    //icon: '<span class="icon-youtube"></span>',
    onAction: () => alert('Button clicked!')
});

From what I understand you trying to add a button with an image in the list of icons tinyMCE.

 tinymce.PluginManager.add("youtube", function (editor) {   
    editor.addButton('youtube', {
        tooltip: 'Add video',
        image: tinymce.baseURL + '/plugins/youtube/icons/youtube.gif',
        onclick: function() {

        }
    }); 
});

Create a folder (I named "youtube") in this folder create another folder (I named "icons" in which you put your image), then create your file youtube.js under youtube folder.