Sharepoint - Detect if modal dialog box

Try this:

if (window.location.search.match("[?&]IsDlg=1"))

You could simply use javascript to identify if the page is loaded in an iframe: http://www.24hourapps.com/2009/01/check-if-page-is-loaded-in-iframe-using.html

var isInIFrame = (window.location != window.parent.location) ? true : false;

SharePoint dialog boxes are simply iframes.

When in Dialog mode the window.frameElement is NEVER null. use this as example on how to close it remotely

var parentObject = window.frameElement;
    if (parentObject != null) {
        var waitDialog = parentObject.waitDialog;
        if (waitDialog != null) {
            waitDialog.close();
        }
    }