EstimatdDistribution returns evaluated expression, not function name/

After a bit of spelunking, you could try the following to define a new distribution:

betaPrimeDistributionEval[a_,b_] := ProbabilityDistribution[x^(a-1) (1+x)^(-a-b)/Beta[a,b],{x,0,\[Infinity]},Assumptions->a>0&&b>0]
betaPrimeDistribution /: DistributionDomain[_betaPrimeDistribution] = Interval[{0, Infinity}];
betaPrimeDistribution /: DistributionParameterQ[_betaPrimeDistribution] = True;
betaPrimeDistribution /: DistributionParameterAssumptions[betaPrimeDistribution[a_,b_]] := a>0 && b>0
betaPrimeDistribution /: PDF[betaPrimeDistribution[a_, b_], c___] := PDF[betaPrimeDistributionEval[a, b], c]
betaPrimeDistribution /: RandomVariate[betaPrimeDistribution[a_, b_], c___] := RandomVariate[betaPrimeDistributionEval[a, b], c]

Then I think the following does what you want:

data = RandomVariate[betaPrimeDistribution[1, 2], 100];
EstimatedDistribution[data, betaPrimeDistribution[a, b]]

betaPrimeDistribution[0.899712, 1.704]

What do you mean? The beta prime distribution is right here. Even if it wasn't, you could easily define it with TransformedDistribution as

betaPrimeDistribution[α_, β_] := 
 TransformedDistribution[
   \[FormalX]/(1 - \[FormalX]),
   \[FormalX] \[Distributed] BetaDistribution[α, β]
]