Out of memory error: Java heap memory on Android Studio

This is old, but just add this to the build file

dexOptions {
    javaMaxHeapSize "2g"
}

Build => Clean Project

then

Build => Rebuild Project

When I do this, it always fixes for me.

Update 2022; DSL element 'dexOptions' is obsolete and should be removed. It will be removed in version 8.0 of the Android Gradle plugin.
Using it has no effect, and the AndroidGradle plugin optimizes dexing automatically.

I tested and with Gradle 7.0+, increasing below mentioned org.gradle.jvmargs option is all we need.

Original Answer

I had a similar issue on Android Studio 2.3.3. APK packaging would fail saying Java heap space when building from Android Studio and trying to process a 300MB .so library. In my case two things were needed to fix it. Your build.gradle should contain something like that:

android {
    dexOptions {
        javaMaxHeapSize "4G"
    }
}

At the root of your project you need to have a file named gradle.properties containing the following:

# That's needed for java to be able the create the APK with our 300MB native  library
# Basically we grant java a 4GB heap.
org.gradle.jvmargs=-Xmx4608m