Use different build types of Library Module in Android App Module in Android Studio and Gradle

Update: publishNonDefault is deprecated and has no effect anymore. All variants are now published.

Documentation taken from Library Publication chapter in Gradle Plugin User Guide.

By default a library only publishes its release variant. This variant will be used by all projects referencing the library, no matter which variant they build themselves. This is a temporary limitation due to Gradle limitations that we are working towards removing.

Anyway, there are solutions for this problem.

Publish All - It is possible to publish all build variants of your library project by adding following line to your library project:

android {
    publishNonDefault true
}

Then you should modify your dependencies in app project as follows:

dependencies {
    releaseCompile project(path: ':yourLibrary', configuration: 'release')
    debugCompile project(path: ':yourLibrary', configuration: 'debug')

    // This is also possible
    customCompile project(path: ':yourLibrary', configuration: 'custom')
}

Change Default - You can change which varaint gets published from your library by adding following line to your library project:

android {
    defaultPublishConfig "debug"
}

And in this case you don't have to change app's dependencies because it will always get debug build variant.

As of Android Gradle Plugin v3.0.0, the plugin could choose the build type to compile for the sub-module library based on the build type being compiled for the app. Hence debug would compile debug type of library, and release would compile release type of the library. Furthermore there are even added extensions for resolving non-standard build types or flavors using matchingFallback or missingDimensionStrategy.

More info can be found here: Use variant-aware dependency management

Well, Gradle Android plugin simply can't build the debug version of dependent library modules. This is a well-known, old issue and this is not resolved yet.
You can try to use some workarounds from the discussion I mentioned, specifically take a look at posts #35 and #38.