How do I overwrite WordPress core functions?

In WordPress you can't owerride the whole core. You have two ways to modify the WordPress Core :

First option : Pluggable functions

Some functions can be redefined. This can be done in theme's functions.php file or in a plugin file. You can check the list of the Pluggable Functions.

Second option : Filter and Action hooks

• Filter hooks can be used to modify vars inside the core WordPress process.
• Action hooks can be used to fire custom function on some events.

Notice 1 : you can create your own filter and action hooks in your theme or plugin.

Notice 2 : you can use filter hooks to fire a function if you don't find a convenient action hook.

If you follow the WordPress coding standards, you should be able to deeply modify the behaviour of WordPress.

For the function your showing, you should look for the post_where filter hook, to do something like this :

add_filter( 'posts_where' , 'my_posts_where' );

function my_posts_where( $where ) {

    global $post;

    if ($post->post_type == 'video_photo' ){
        $where .= " AND meta_key like 'tqmcf_%'";
    }

    return $where;
}

Edit 1: The following may be more suitable even if this specific query is hard to target.

add_filter( 'query' , 'my_meta_form_query' );

function my_meta_form_query( $query ) {

    $pattern = "/SELECT(?:\W*)meta_key(?:\W*)FROM (.*)(?:\W*.*?)*LIMIT(?:\W*)([0-9]*)/g";

    if( preg_match($pattern, $query, $vars) ) {
        $postmeta = $vars[1];
        $limit = $vars[2];

        $query = "SELECT meta_key
                FROM $postmeta
                WHERE meta_key like 'tqmcf_%'
                GROUP BY meta_key
                HAVING meta_key NOT LIKE '\_%'
                ORDER BY meta_key
                LIMIT $limit";
    }

    return $query;
}

You have to digg into the code to find suitable hooks and how you can best filter variables. In this example, we check if the query is matching the one in the meta_form() function. We extract the existing query vars and we build a new query including the WHERE condition. I didn't test this piece of code and there may be some bugs but it can give you an idea of how we alter the core code.

There's no replacing core functions, unless they're defined in wp-includes/pluggable.php.

You can modify values and processes using hooks where available (you'll find them mainly by apply_filters and do_action functions in core files). IMO, there are more specialists and knowledge base at WordPress Answers, but I suggest focusing on the end result because you could hook on the wpdb or the postmeta queries, for example.
In you sample code, it's not clear what you're trying to achieve, but the only filter hook available doesn't seem adequate, hence the need of other hook and clear objective.

In the end of the day, the proper way of modifying WordPress behavior is through the Plugin API.