Area of sub-triangle inside a triangle

Hint: use ratio of areas of $2$ triangles. Let's use your diagram to calculate the red area and you can generalize it easily. The red area is half of the area of the triangle whose vertices at $0,2,2$ whose area is again is $\frac{2}{5}$ area of the triangle whose vertices are at $0,2,5$ and in turn whose area is half the area of the triangle whose vertices are at $0,4,5$. Hope it helps.

Without loss of generality we can assume that $F$ lies between $A$ and $E$. Recall that an area of a triangle is a half of a product of length of adjacent sides by the sinus of the angle between these sides. Therefore, since triangles $DEF$ and $DEA$ have a common angle $\angle E$ and common side $DE$, their areas relates as the sides adjacent to the angle $\angle E$, that is $\frac{\operatorname{area}(\triangle DEF)}{\operatorname{area}(\triangle DEA)}=\frac{EF}{EA}$. Also since triangles $DAE$ and $BAC$ have a common angle $\angle A$, $\frac{\operatorname{area}(\triangle DAE)}{\operatorname{area}(\triangle BAC)}=\frac{DA}{AB}\cdot\frac{AE}{AC}$. Finally, $$b=\operatorname{area}(\triangle DEA)=\frac{EF}{EA}{\cdot\operatorname{area} (\triangle DEA)}=\frac{EF}{EA}\cdot\frac{DA}{AB}\cdot\frac{AE}{AC}\cdot {\operatorname{area}(\triangle BAC)}= \frac{EF}{EA}\cdot \frac{DA}{AB}\cdot\frac{AE}{AC}\cdot a.$$

The points are canonically labeled such that $B \le D \le C$. This means if $D > C$, we swap the labels before applying the following logic.

Using Side-angle-side formula, we calculate area of $\triangle ABD$ as

$$x = { AB \times BD \times sin(\theta) \over 2}$$

Area of $\triangle ABC$:

$$y = { AB \times BC \times sin(\theta) \over 2}$$

Area of $\triangle ADC = y – x$, given by

$$y - x = { { AB \times BC \times sin(\theta) \over 2} - { AB \times BD \times sin(\theta) \over 2} } = {{AB sin(\theta)} \times (BC - BD) \over 2}$$

Ratio of Areas ($\triangle ADC$ / $\triangle ABC$) is given by

$$BC - BD \over BD$$

This is true when $BC - BD > 0$. When $BC - BD = 0$, the $\triangle ADC \cong \triangle ABC$ and the ratio is $1$.

---

I notice that I have used a different nomenclature for the triangles than what the OP had used in the problem statement and also the fact that the largest triangle has area $a$. For consistency in proof, let me call the outer triangle as $\triangle A'BC'$ (area $= a$). Using the same principles as outlined above for calculating the ratio of areas of triangles, the ratio Area of $\triangle A'BC'$ and Area of $\triangle ABC$ is given by

$$A'B \times BC' \over AB \times BC$$

Ratio of Areas ($\triangle ADC$ / $\triangle A'BC'$) is given by

$${BC - BD \over BD} \div {A'B \times BC' \over AB \times BC}$$

$\therefore$, Area $\triangle ADC$ is given by

$${BC - BD \over BD} \times {AB \times BC \over A'B \times BC'} \times a$$

Substituting $A'B = 4$ and $BC' = 5$, we get

Area $\triangle ADC$

$$ = \big( {BC - BD \over BD}\big) \times {AB \times BC \over 4 \times 5} \times a$$

As mentioned earlier, this is true when $BC - BD > 0$. When $BC - BD = 0$, the $\triangle ADC \cong \triangle ABC$ and the ratio (given inside the parenthesis) is $1$.

Note: Just a reminder, the sides are as denoted in the figure in this response (that are slightly different from the nomenclature used by the OP).