How to include an HTML page into another HTML page without frame/iframe?

If you mean client side then you will have to use JavaScript or frames.
Simple way to start, try jQuery

$("#links").load("/Main_Page #jq-p-Getting-Started li");

More at jQuery Docs

If you want to use IFrames then start with Wikipedia on IFrames

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
        The material below comes from the website
        <iframe src="" height="200">
            Alternative text for browsers that do not understand IFrames.

You could use HTML5 for this:

<link rel="import" href="/path/to/file.html">

Update – July 2020: This feature is no longer supported by most major browsers, and is generally considered obsolete. See for the list of browsers which do still support it.

You can use an object element

<object type="text/html" data="urltofile.html"></object>

or, on your local server, AJAX can return a string of HTML (responseText) that you can use to document.write a new window, or edit out the head and body tags and add the rest to a div or another block element in the current page.

If you're just trying to stick in your own HTML from another file, and you consider a Server Side Include to be "pure HTML" (because it kind of looks like an HTML comment and isn't using something "dirty" like PHP):

<!--#include virtual="/footer.html" -->