CompiledFunction showing its contents

This has to do, AFAIR, with whether the expression is marked internally as "evaluated" or perhaps "valid object." If it is so marked, it will be typeset by the Front End in the summary-box way; if it is not evaluated (or not valid), you will get the input form. This happens for other functions besides CompiledFunction, too. I don't believe there is a way at the user level to access the flag(s) that indicate the "evaluated" status of an expression. @WReach's traceView* functions seem to infer it from the evaluation sequence and marks them "inert", if you wish to explore the difference between Hold[Evaluate[comp]] and Hold[Evaluate[comp]] // Replace[#1, {}, Infinity] &.

What seems to happen is that when Replace scans the expression, it must do something that marks the expression as needing reevaluation, even if it does nothing:

Hold[Evaluate[comp]] // Replace[#1, {}, Infinity] &

In fact, in the case in question, any level specification of 5 or higher triggers the behavior:

Hold[Evaluate[comp]] // Replace[#1, {}, {5}] &

To get the summary box back, the CompiledFunction needs to be reevaluated. One way is with the following replacement:

Hold[Evaluate[comp]] // Replace[#1, {}, {5}] &
% /. cf_CompiledFunction :> Evaluate@cf /; True

Here's the same thing applied to InterpolatingFunction:

ifn = Interpolation[N@Range[4]^2];
Hold[Evaluate[ifn]] // Replace[#1, {}, Infinity] &
% /. f_InterpolatingFunction :> Evaluate@f /; True

Here are the outputs of traceView2[] on the two examples. The important difference is in the second-to-last line, which shows whether the CompiledFunction is marked "inert" or not:

This has everything to do with a special bit called ValidQ that is used all over the place in Mathematica. Replace by dint of its scanning and likely unpacking of some array of bytes forces a copy of the expression and thus removes this bit.

Just to check that this is the issue:

CompiledFunction // FormatValues // Keys

{HoldPattern[
  MakeBoxes[ElisionsDump`cf : CompiledFunction[BoxForm`args__], 
    BoxForm`fmt_] /; 
   BoxForm`UseIcons && 
    System`Private`ValidQ[Unevaluated[ElisionsDump`cf]]]}

We can see it is indeed checking for ValidQ.

Here's another example:

Extract[
 Replace[Hold[Evaluate[comp]], {Hold[x_] :> x}, Infinity], 
 1, 
 System`Private`HoldValidQ
 ]

False

We can also use this to make invalid input behave like a CompiledFunction:

System`Private`HoldSetValid[
   CompiledFunction["asdasdasd", "asdasdasd", "asdasdasd", 
    "asdasdasd", "asdasdasd", "asdasdasd", "asdasdasd", "asdasdasd"]]@
  1;

CompiledFunction::cfct: Number of arguments 1 does not match the length 8314036833820636001 of the argument template.

It actually thinks that's a CompiledFunction because of this bit. It's powerful.

This is also one of those sneaky places where the object-oriented nature of Mathematica's implementation sneaks through, too:

Replace[Hold[Evaluate[comp]], {}, Infinity] === Hold[Evaluate@comp]

True

But one isn't ValidQ and the other is.

Replace[x, ..., All] is documented. See section "Details and Options" in the documentation of Replace.

There we read that Infinity is equivalent to level spec {1, Infinity} so that the head of the expression is ignored. So in your example, just no replacement is happening.

However, All is equivalent to level spec {0, Infinity}, taking care also if the head of the expression.

But tell me: Why do you not use ReplaceAll with the very nice postfix syntax /. in the first place?

Hold[Evaluate[comp]] /. Hold[x_] :> x