What does it mean to apply an operator to a state?

It seems that OP's question arises because he assumes that a state $|\psi\rangle$ is normalized $\langle\psi |\psi\rangle=1$ at all stages of developing the quantum mechanical language.

Let $H$ be a Hilbert space. Note that the set $$\{|\psi\rangle\in H \mid \langle\psi |\psi\rangle=1\}$$ of normalized states is not a vector space, and therefore not a Hilbert space.

It is better to only assume that a state $|\psi\rangle$ is just normalizable

$$\langle\psi|\psi\rangle~<~\infty,$$

with the implicit assumption that when one wants a probabilistic interpretation, then one should normalize $|\psi\rangle$ via the standard procedure:

$$ |\psi\rangle ~\longrightarrow ~ |\psi^{\prime}\rangle~:=~\frac{|\psi\rangle}{\sqrt{\langle\psi|\psi\rangle}} , $$

so that $$\langle\psi^{\prime} \mid\psi^{\prime}\rangle~=~1.$$

So to answer OP's question, in the elementary version$^1$ of quantum mechanics, a state is a (ket) element $|\psi\rangle$ of the Hilbert space $H$. In particular, it is a normalizable element. An Observable is a linear Hermitian operators $\hat{A}:H\to H$ that takes states to states. The expectation value $\langle\hat{A}\rangle$ of the observable $\hat{A}$ in the state $|\psi\rangle$ is then

$$\langle\hat{A}\rangle ~=~ \frac{\langle\psi|\hat{A}|\psi\rangle}{\langle\psi|\psi\rangle}. $$

Concerning the subquestion about measurements in quantum mechanics and the collapse of the wave-function, I suggest to first check out Wikipedia, and if needed, then ask a more specific question.

--

$^1$The kind of version that ignores un-normalizable states and unbounded linear operators.

Your attempt to summarize a state vector by attaching to it a daily life meaning is probably a reason why you can't understand the more abstract situation prevailing in QM.

$A|\psi\rangle$ is simply the image of the vector $|\psi\rangle$ under the operator $A$. There is no need that either $|\psi\rangle$ or its image is a state in the sense of being normalized. They are just elements of the Hilbert space (or, sometimes, unnormalizable weak limits of such states.)

There is also no necessary relation to measurement. Interpreting realistic measurement is a quite complex matter, and the textbook recipe (Born's rule) is applicable only to the simplest or very idealized situations.

I think you may be misguided by the concept that we associate $\textbf{observables}$ to self-adjoint operators. They do operate on the Hilbert space, but to see them as entities that transform states or prepare them is a little bit tricky. I will describe here self-adjoint operators and preparation of states.

1) The true (physical) power of self-adjoint operators for describing observables lies in the spectral theorem, and not in its $\psi \mapsto A\psi$ action. Physically, what it means? There is a set called spectrum of an observable, and it is the set of possible outcomes on its measure for given states. For example, a spin observable $S$ on a 1/2-spin system has spectrum $\sigma(S) = \{-1/2,+1/2\}$, and decomposes as a sum of its spectral projections, $S = +1/2 P_{+} -1/2 P_{-}$. In general, there is a spectral resolution $E$, that is, a bunch of projection related to the spectrum, such that the operator can be written as $A = \int_{\sigma(A)}\lambda dE(\lambda)$.

And what are the spectral projections? Those are again (self-adjoint) operators, but the whole collection of spectral projections will give you a probability measure when coupled with a state. In the spin system example, if you take a state $\psi$, then $\langle\psi, P_+\psi\rangle$ would give you the probability of measuring a +1/2 spin, and likewise for -1/2.

Now suppose you had a 1/2 spin system with prepared state $\psi$, and you measure the spin, and get +1/2. After the measurement, your state collapses to a $|+1/2\rangle$ state.

In a more detailed formalism, suppose you have prepared a state $\psi$ and you are going to make a measurement of an observable expressed as $A = \int_{\sigma(A)} \lambda dE(\lambda)$ (where the $E$ is the spectral resolution of your operator, just think of the 1/2-spin example intuitively). Then suppose your measurement is on a subset $\Lambda \subset \sigma(A)$ (you may think of the set $\{+1/2\} \subset \{-1/2,+1/2\}$. Your state $\psi$ then collapses to the following state $\phi$:

$\psi \rightarrow \phi = \frac{E(\Lambda)\psi}{\|E(\Lambda)\psi\|}. $

(notice that $\phi$ is normalized and well defined, since $E(\Lambda)\psi=0$ then the probability of the outcome being in $\Lambda$ would be zero to start).

Summing up, you do not simply apply a self-adjoint operator on a state, since, as you have seen, it doesn't have much meaning. This is a point most introductory QM books do not stress as much as I would want. What happens with measurements and collapses and whatsoever uses, as I tried to point out, the spectral projections more than the operator itself. So, as you said about your Hamiltonian operator, it does not act like your syrup machine, which we will try to cover up next.

2) Now what you describe as "tools", in your example, the putting syrup, is not a measurement per se, it is a preparation of states, which would grab a state without syrup and put syrup in it. The modeling of such procedure is usually ignored, at least to my knowledge.

One choice would be just saying "my state now is syrup", end of discussion.

Other option is using unitary operators ($U$ such that $UU^* = U^*U = 1$). Those transform state vectors in state vectors.

If you would like more sophisticated examples, it starts to get tricky, and I will shut up before I say something very wrong about it. But rest assured this is not easy at all, and your question is really nice. Hope to see some other inspiring aswers.