Is there a way to prevent N from being applied to all parts of an expression

You can protect arguments inside brackets from N with the family of attributes, NHoldAll, NHoldFirst, and NHoldRest.

For an indexed variable such as x[1], x[2], etc., either

SetAttributes[x, NHoldAll]    (* what I would normally use *)

or

SetAttributes[x, NHoldFirst]  (* protects only the first argument *)

would keep the index from being numericized by N:

N[x[1]]
(*  x[1]  *)

Attributes of x can be cleared with ClearAttributes[x, <attribute>] or ClearAll[x], but NOT with Clear[x].

Built-in functions that require some or all arguments to be an integer (e.g. Take, Part), an exact number (e.g., AlgebraicNumber), or simply protected from being changed by N (e.g. Subscript) have one of these attributes.