How to create visualforce page with iframe with auto resizing height?

The trick is to resize the iframe with JavaScript after you know the size of the browser viewport:

<apex:page sidebar="false">
    <apex:iframe src="https://www.salesforce.com/" id="theFrame" />
    <script>document.getElementById('theFrame').height = window.innerHeight - 220;</script>
</apex:page>

Leave about 220 pixels for the Salesforce header and footer.

For bonus points, you can trim the container cell padding if you need and play with the 220 figure.

<style>table#bodyTable {border-collapse: collapse;} table#bodyTable td {padding: 0;}</style>

Edit: You can additionally deal with when the window is resized:

<script>
    (function() { //this wrapper prevents pollution of the global scope
        var windowOnresize = window.onresize;
        window.onresize = function() {
            if (windowOnresize) windowOnresize(); //don't trample the handler; intercept it
            document.getElementById('theFrame').height = window.innerHeight - 220;
        };
    }());
</script>