How can I change default black dim background "color" (not the amount of dim) of Dialog?

This is a workaround but it's not really a pure solution because background touch is disabled and should be configured manually.

First, set custom dialog theme like this.

styles.xml

<style name="CustomDialogTheme" parent="android:Theme.Dialog">
    <item name="android:windowNoTitle">true</item>
    <item name="android:windowIsFloating">false</item>
    <item name="android:windowBackground">@android:color/transparent</item>
</style>

Setting windowIsFloating to false forces Dialog view to be expanded to full screen. Setting windowBackground to transparent removes default black dim background under Dialog. windowNoTitle option gets rid of the upper title bar.

CustomDialog.java

Apply the theme and construct your custom_dialog view as follows.

public HTCustomDialog(Context context) {
    super(context, R.style.CustomDialogTheme);
    setContentView(R.layout.custom_dialog);
}

custom_dialog.xml

<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:background="@color/main_solid_80">

    <RelativeLayout
        android:id="@+id/dialog_root"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_centerInParent="true"
        android:layout_marginLeft="50dp"
        android:layout_marginRight="50dp"
        android:background="@drawable/bg_popup"
        android:padding="16dp">

</RelativeLayout>

Now that CustomDialog view is a full-screen view, set background of your root layout to whatever color you'd like.

Sample result

I mosaiced the result a bit.

use custom style.

<style name="transparent_dialog_borderless" parent="android:Theme.Dialog">
    <item name="android:windowFrame">@null</item>
    <item name="android:windowIsFloating">true</item>
    <item name="android:windowIsTranslucent">true</item>
    <item name="android:windowNoTitle">true</item>
    <item name="android:background">#FF333333</item>
    <item name="android:windowBackground">@null</item>
    <item name="android:backgroundDimEnabled">true</item>
</style>

android:backgroundDimEnabled:control the black dim background