Mysql sub query returns more than 1 row

        (SELECT courses  FROM wp_category WHERE CatID =401) 
        OR 
        (SELECT  meta_value FROM wp_postmeta WHERE  post_id IN (SELECT courses FROM wp_category WHERE CatID =401) AND meta_key ='post_id' )

This is a condition, but you need n values.

This should work:

(
    ID IN 
        (
            (SELECT courses  FROM wp_category WHERE CatID =401) 
        )
    OR
    ID IN 
        (
            (SELECT  meta_value FROM wp_postmeta WHERE  post_id IN 
                (SELECT courses FROM wp_category WHERE CatID =401) AND meta_key ='post_id' )
        )
)

Also take care about subqueries with MySQL. In your case a sub- subquery.

Take a look here: https://stackoverflow.com/questions/12356784/mysql-dependent-sub-query-with-not-in-in-the-where-clause-is-very-slow