How to factor all coefficients of a multivariable into prime factors

You could create a factoredForm wrapper that formats integers as desired:

MakeBoxes[factoredForm[e_], StandardForm] := Block[{$Factored=True},
    MakeBoxes[e]
]

Unprotect[Integer];

MakeBoxes[i_Integer, form_] /; $Factored := Block[{$Factored},
    TemplateBox[
        {RowBox[tosuperscript /@ FactorInteger[i]], MakeBoxes[i]},
        "FactoredInteger",
        DisplayFunction->(#1&),
        InterpretationFunction->(#2&)
    ]
]

tosuperscript[{-1, 1}] := "-"
tosuperscript[{a_, 1}] := MakeBoxes[a]
tosuperscript[{a_, b_}] := MakeBoxes[a^b]

Unprotect[Power];

MakeBoxes[Power[a_, i_Integer], StandardForm] /; $Factored := With[
    {exp = Block[{$Factored}, RawBoxes @ MakeBoxes[i]]},
    MakeBoxes[Power[a, exp]]
]

Protect[Integer, Power];

Some examples:

-10 x - 10240 y^3 - 1520 x y^4 //factoredForm

 (21 x^10 + 35 x^4 y^12)(-10 x - 10240 y^3 - 1520 x y^4) //factoredForm

25 y (20 + 160 x^4 y^4 + 5 x^6 y^5) //factoredForm

Another simple solution is to use pattern matching to find all the integer coefficients, FactorInteger them, and then reassemble the number but using an Inactive version of Power which doesn't evaluate.

InactiveFactorization[i_] := Inactive[Times] @@ (Inactive[Power] @@@ FactorInteger[i])
factorpolycoeffs = {
  i_Integer?Positive (e : x_Symbol^y_Integer | x_Symbol) :> InactiveFactorization[i] e,
  i_Integer?Negative (e : x_Symbol^y_Integer | x_Symbol) :> -InactiveFactorization[-i] e
}
cleanupprimepowers = {Inactive[Power][x_Integer, 1] -> x}

Here we explicitly address the positive and negative case to avoid an uglier factorization of negative coefficients and zeros by FactorInteger. Now we can use these rules to factor our

poly = -10 x - 10240 y^3 - 1520 x y^4;

(poly /. factorpolycoeffs)
% /. cleanupprimepowers

-x 2^1 5^1-y^3 2^11 5^1-x y^4 2^4 5^1 19^1

-x (2*5)-y^3 (2^11*5)-x y^4 (2^4*5*19)

We can Activate the Power and Times terms in the expression to let the kernel resolve the expression to the original poly

Activate[%]

-10 x-10240 y^3-1520 x y^4

Edit I added a second rule to simplify prime powers of one to make the result more readable which also required to Inactiveate Times, too.

The following functions using Rule[] and Interpretation[]:

fi[n_] := Times @@ (FactorInteger[n] /. {
  List[-1, 1] -> -1, 
  List[p_Integer, 1] :> Interpretation[ToString[p], p], 
  List[p_Integer, q_Integer] :> 
    Interpretation[ToString[p]^q, p^q]});
do[e_] := e /. {x_. n_Integer :> x fi[n],
             x_^y_. n_Integer :> x^y fi[n]};

used in an example code

 do[ (21 x^10 +35 x^4 y^12)(-10 x -10240 y^3 -1520 x y^4) ]

returns something very close to what you want.

The downside is that the numerical factored coefficients come after the powered variables and not before.There may be a way to fix that but I don't see it now. The advantage of using Interpretation[] is that you can copy/paste the resulting output and it evaluates the same as the original polynomial.

NEW:

I developed an alternate approach which does a much better job.

fi[n_Integer] := Sign[n] With[{fl = FactorInteger@Abs@n}, fl /.
   {List[1, 1] -> 1, List[p_Integer, 1] :> ToString[p], 
   List[p_Integer, q_Integer] :> (ToString[p])^q} /.
   {List[x_] :> x, List[x__] :> Inactive@Times@x}];
do[ex_] := Activate //@ (x_^n_Integer :> x^n, 
   n_Integer :> fi[n], n_Integer* x_. :> fi[n] x});

For example:

do[(21 x^10 +35 x^4 y^12) (-10 x -10240 y^3 -1520 x y^4)] // TeXForm

returns the result

\left(-19 2^4 5 x y^4+2^{11} (-5) y^3-2 5 x\right) \left(3 7 x^{10}+5 7 x^4 y^{12}\right)

which exposes a "misfeature" of TeXForm[]. For exmaple: TeXForm[-1-x^2 z] returns "x^2 (-z)-1".

The new code only requires Activate[], Inactive[] and a tweaked fi[]. It avoids the downside in the original method. It avoids the need to add rules to system objects Power and Integer which has global scope.