with jquery UI dialog, is there anyway to have a max height and use 'auto' if its smaller

You can achieve this by doing the following:

HTML

<div id="dialog" title="Dialog Title">
    Dialog content
</div>

JavaScript

$('#dialog').dialog({
    resizable: false,
    minHeight: 0,
    create: function() {
        $(this).css("maxHeight", 400);        
    }
});

Check out test case on jsFiddle.

I use this:

$('#dialog').dialog({
    maxHeight: $(window).height(),
    open: function() {
        $(this).dialog('option', 'maxHeight', $(window).height());
   }
});

Resetting maxHeight in "open" is useful when window has changed size.

You can do it like this:

 $('#testing').resizable("enable");
 $('#testing').dialog({ maxHeight: 400 });


<div id="testing" class="ui-widget-content">
<h3 class="ui-widget-header">Resizable</h3>
</div>