How to find the shape parameters of of a beta distribution given the position of two quantiles?

If you are fine with a numerical approximation of the values, FindRoot is the function you'll want to use:

sol = FindRoot[
  Quantile[BetaDistribution[Exp[a], Exp[b]], {1/2, 9/10}] == {1/10, 1/2},
  {
   {a, 0},
   {b, 0}
   }
  ]
BetaDistribution[Exp[a], Exp[b]] /. sol

{a -> -0.787898, b -> 0.717402}

BetaDistribution[0.4548, 2.0491]

In these equations I used Exp[a] and Exp[b], because those are always positive so FindRoot will never accidentally try illegal parameters of the BetaDistribution.

You could also try to "come close", e.g. minimize total quadratic error:

sol = NMinimize[
 {
   Plus[   
     (Quantile[BetaDistribution[α, β ], 1/2] - 1/10 )^2 , 
     (Quantile[BetaDistribution[α, β], 9/10] - 1/2)^2
   ]
   ,
   α ≥ 0 ∧ β ≥ 0
 }
 ,
 {α, β}
 ,
 Method -> "NelderMead" (* not needed but to make this explicit *)
]

{1.03161*10^-18, {α -> 0.4548, β -> 2.0491}}