How can I restore the old ImageCompose behaviour?

The $over$ operator correctly implemented

According to the Wikipedia, when composing $A$ $over$ $B$, the output alpha channel value $\alpha_O$ and the output color channel value $C_O$ are calculated as follows:

$\begin{cases}\alpha_O = 1 - (1 - \alpha_A) (1 - \alpha_B) \\C_O = \frac{\alpha_A C_A + (1 - \alpha_A)\alpha_B C_B}{\alpha_O}, \text{if $\alpha_O \neq 0$} \\C_O = 0, \text{if $\alpha_O = 0$} \end{cases}$

where $\alpha_A$ and $\alpha_B$ are alpha channel values of $A$ and $B$, and $C_A$ and $C_B$ – color channel values of $A$ and $B$ correspondingly.

This can be directly implemented in Mathematica 11.1 or above as follows:

imageCompose[b_Image, a_Image] := 
 Module[{alphaA = AlphaChannel@a, alphaB = AlphaChannel@b, alphaO, 
         cA = RemoveAlphaChannel@a, cB = RemoveAlphaChannel@b},
  alphaO = 1 - (1 - alphaA) (1 - alphaB);
  SetAlphaChannel[(alphaA*cA + (1 - alphaA) alphaB*cB)/alphaO, alphaO]]

Let us check the associative property:

{{i0, i1, i2}} = ImagePartition[
  Import["http://i.stack.imgur.com/r13gh.png"], {Scaled[1/3], Scaled[1]}]

{i0~imageCompose~(i1~imageCompose~i2), (i0~imageCompose~i1)~imageCompose~i2}
Equal @@ %
ColorSeparate /@ %%

True

It holds! So what is the problem with ImageCompose of version 10 and later?

Current implementation of ImageCompose: the diagnosis

When writing the above implementation for the first time I unintentionally made a simple mistake: I forgot to divide the output value for the color channel by $\alpha_O$. Here is what happened:

imageComposeWrong[b_Image, a_Image] := 
 Module[{alphaA = AlphaChannel@a, alphaB = AlphaChannel@b, alphaO, 
         cA = RemoveAlphaChannel@a, cB = RemoveAlphaChannel@b},
  alphaO = 1 - (1 - alphaA) (1 - alphaB);
  SetAlphaChannel[alphaA*cA + (1 - alphaA) alphaB*cB, alphaO]]

{i0~imageComposeWrong~(i1~imageComposeWrong~i2), 
 (i0~imageComposeWrong~i1)~imageComposeWrong~i2}
Equal @@ %
ColorSeparate /@ %%

False

The output looks exactly the same as for the current ImageCompose:

{i0~ImageCompose~(i1~ImageCompose~i2), (i0~ImageCompose~i1)~ImageCompose~i2}
Equal @@ %
ColorSeparate /@ %%

False

Numerical comparison reveals tiny differences due to rounding off errors. But the final diagnosis is clear: the developer just forgot to divide the color channel by the alpha channel!

It is a great shame that during more than three years after the release of version 10.0.0 nobody noticed this in the company! Do they themselves use this functionality – or not?!

Please, do not be lazy to report this to the technical support, so that this shameful bug will be fixed as soon as possible! A high priority is given to bugs, which many users write about...

The remedy

From the above considerations the remedy is obvious: we must just divide the color of the ImageCompose output by the alpha channel:

icFix[img_Image] := img/AlphaChannel[img];

{i0~icFix@*ImageCompose~(i1~icFix@*ImageCompose~i2), (i0~icFix@*ImageCompose~i1)~icFix@*ImageCompose~i2}
Subtract @@ % // MinMax
ColorSeparate /@ %%

{-3.10689*10^-6, 3.08454*10^-6}

As one can see, there are still tiny differences due to rounding-off errors, but the associative property in fact is restored and the output is correct!

---

Original answer

Citing a comment by Rahul:

Well, that's certainly undesirable! Alpha compositing is supposed to be associative (i0~ImageCompose~(i1~ImageCompose~i2) should equal (i0~ImageCompose~i1)~ImageCompose~i2) and this doesn't do that. One could implement correct alpha compositing manually using ImageApply, but let's see if someone has a better way.

Indeed, in versions 8.0.4 and 9.0.1 ImageCompose is associative:

$Version

{{i0, i1, i2}} = ImagePartition[
  Import["http://i.stack.imgur.com/r13gh.png"], {Scaled[1/3], Scaled[1]}]

{i0~ImageCompose~(i1~ImageCompose~i2), (i0~ImageCompose~i1)~ImageCompose~i2}

Equal @@ %

ColorSeparate /@ %%

"9.0 for Microsoft Windows (64-bit) (January 25, 2013)"

True

... while starting from version 10.0 it is not:

$Version

{{i0, i1, i2}} = ImagePartition[
  Import["http://i.stack.imgur.com/r13gh.png"], {Scaled[1/3], Scaled[1]}]

{i0~ImageCompose~(i1~ImageCompose~i2), (i0~ImageCompose~i1)~ImageCompose~i2}

Equal @@ %

ColorSeparate /@ %%

"10.0 for Microsoft Windows (64-bit) (September 9, 2014)"

False

It is also worth to note that despite that Overlay[{i0, i1}] and Show[i0, i1] look the same as the old ImageCompose[i0, i1], they are not the same. But they do (approximately) equal to each other and do approximately hold the associative property:

$Version

overlayCompose[i0_, i1_] := Rasterize[Overlay[{i0, i1}], "Image", Background -> None];
showCompose[i0_, i1_] := Rasterize[Show[i0, i1], "Image", Background -> None];

{overlayCompose[i0, i1], showCompose[i0, i1], i0~ImageCompose~i1}

ColorSeparate /@ %

{i0~overlayCompose~(i1~overlayCompose~i2), (i0~overlayCompose~i1)~ overlayCompose~i2}

ColorSeparate /@ %

{i0~showCompose~(i1~showCompose~i2), (i0~showCompose~i1)~showCompose~ i2}

ColorSeparate /@ %

"9.0 for Microsoft Windows (64-bit) (January 25, 2013)"

As one can see from the above, Overlay introduces artifact at the top, while Show does not.

You can get the old ImageCompose behavior by using Overlay instead:

Overlay[{i1, i2}]

---

Edit:

As pointed out by the comment by ybeltukov the Head of an Overlay is "Overlay" and therefore doesn't match the Head of ImageCompose, which is "Image". I didn't realize this, because exporting to a .png file did handle the transformation.
One can use e.g.

ImportString@ExportString[Overlay[{i1, i2}], "PNG"]

to get an object with Head "Image", and that therefore can be used the same way as an object created with ImageCompose inside the notebook.

As well Karsten's solution using Overlay, technical support pointed out that Show can be used in the same way:

Rasterize[Show[i1, i2], "Image", Background -> None]

(Show converts the images to Raster expressions and overlays them in Graphics).

In both cases the alpha compositing is done by the front end, which uses the conventional associative "over" operator.