Why does Gradle build my module in Release mode when the app is in Debug

In the "Build Variants" panel window on the left, you should see both of your modules, and next to them the current "active" variants. For instance

app          debug
custom_lib   debug

When calling Build > Make Project we are building every modules in the project in their current variant.

However, due to a current Gradle limitation (https://code.google.com/p/android/issues/detail?id=52962), building app in debug will require building the release variant of custom_lib, and so you end up building both.

I would recommend to not use Make Project but instead use the option below that says Make Module app. This option will change from app to lib based on the current selection in the Project panel or based on the current editor, and will always do only what's needed to build the current module.

(Looking into this, we noticed that there isn't a shortcut for it, so we're adding one).

Put this in your app dependencies:

dependencies {
    debugCompile project(path: ':custom_lib', configuration: "debug")
    releaseCompile project(path: ':custom_lib', configuration: "release")
}

and in your library's build.gradle add:

android {

    defaultConfig {
        defaultPublishConfig 'release'
        publishNonDefault true
    }

}

Then the library will be built in the same mode as the app. Contrary to previous revisions of this answer, I've confirmed a flavor is not required on the library (this may be due to Gradle or Android plugin versions - I'm using Gradle 2.14 and Android plugin 2.1.0 and did not require it).

Edit: You may have trouble if you don't clean/rebuild after modifying the gradle files, as outlined in this answer here.

Gradle now supports Flavor-buildType-Compile directive, so KaneORiley's answer can now be enhanced as follows:

library's build.gradle:

android {
    defaultPublishConfig 'release'
    publishNonDefault true
    productFlavors {
        library {
    }
}

app's build.gradle:

configurations {
    devDebugCompile
    devReleaseCompile
    storeDebugCompile
    storeReleaseCompile
}

android {
    .....
}

dependencies {
    (...)
    devDebugCompile    project(path: ':path:to:lib', configuration: 'devDebug')
    devReleaseCompile  project(path: ':path:to:lib', configuration: 'devRelease')
    storeDebugCompile  project(path: ':path:to:lib', configuration: 'storeDebug')
    storeReleaseCompile project(path: ':path:to:lib', configuration: 'storeRelease') 
}