What boundary conditions in a wave simulation would avoid reflections?

This is, unfortunately, not a simple task in general. My experience on non-reflecting boundary conditions is for the Navier-Stokes equations, but you should be able to do a similar approach for your system. As you noted, a fixed boundary ($u=0$) will lead to one type of wave while a free boundary ($\partial u/\partial x = 0$) leads to another wave. What you need to do is prescribe a $\partial u /\partial x$ so that it negates the wave. In effect, you are applying a damper to the system to absorb the wave.

For the Navier-Stokes equations, this is done by taking the 1D form of the governing equation (normal to the boundary). The system of variables is then recast into the symmeterizing variables (ie. variables are chosen such that the transformed system is symmetric). This gives a decoupled set of equations for the characteristic waves in the system. From this set of equations for the waves, you compute the variable you must impose at the boundary such that you maintain the target value. It is basically using the method of characteristics to decouple the equations.

For example, let's say that I want $u = 10$ on my boundary. A wave is coming towards the boundary and the value of $u$ next to the boundary is $u = 20$. Based on my boundary condition formulation, I compute the $u$ that I must set in my ghost cell (the "fake" cell outside my boundary that is used to enforce boundary conditions).

Okay, so what does this have to do with your problem? Well, you need to take your system of equations (sounds like right now, it's 1D so there is only a x-displacement but later it may be 2D or 3D and form a system) and recast it using a symmeterizing set of variables so that you now have your equations in characteristic space. This will give you equations for the characteristic waves and you use those to compute the target value you must impose to let the wave leave the domain without any reflections.

It looks and sounds scary but it is relatively straight forward math. It is, however, very tedious. You need to know the variable transformation you must use (which is known for the Navier-Stokes, but your system may be different -- could be easier, could be harder). You then do the transformation into characteristic space, find your Jacobians (which should now be diagonal), compute the eigenvalues (wave speeds) and eigenvectors and you're off running.

An alternative approach could be to use a sponge layer type approach. It's really only good for "outflows". But what you do is apply a damper to your solution so it kills the wave. This is what I did for my spring-lattice code I wrote for my masters thesis. A few springs away from the boundaries, add dampers in parallel to your springs. You'll have to hand-tune the damping coefficients (or non-dimensionalize it so you can just use the non-dimensional critical damping coefficient). Depending on the physics of the problem, this might be okay, but you'll wan to apply these conditions away from anywhere that you want something interesting to happen so it doesn't pollute the solution.

I asked a similar question here and with help of Alex Trounev I got the following answer which I leave here for easy access.

To achieve a non-reflective boundary, we must set the wave propagating from the right wall towards the left to zero. To do so, we can define a new variable that represents the leftward traveling wave (everywhere, not just at the boundary). The usual wave equation is $u_{tt} = u_{xx}$ and this new variable can be defined as such:

$$ v \equiv u_t + u_x $$ Then the wave equation becomes: $$ v_t = v_x $$

Hence, we must set $ v = 0 $ at the right boundary. Replacing $v$ with the usual variables, we get:

$$ u_t = u_x \;\;\; \text{at the right boundary}$$