Is Feynman's explanation of how the moon stays in orbit wrong?
What's actually happening is something more like this:
Here, $x_0$ and $v_0$ are the initial position and velocity of the moon, $a_0$ is the acceleration experienced by the moon due to gravity at $x_0$, and $\Delta t$ is a small time step.
In the absence of gravity, the moon would travel at the constant velocity $v_0$, and would thus move a distance of $v_0 \Delta t$ during the first time step, as shown by the arrow from the green circle to the red one. However, as it moves, the moon is also falling under gravity. Thus, the actual distance it travels, assuming the gravitational acceleration stays approximately constant, is $v_0 \Delta t + \frac12 a_0 \Delta t^2$ plus some higher-order terms caused by the change in the acceleration over time, which I'll neglect.
However, moon's velocity is also changing due to gravity. Assuming that the change in the gravitational acceleration is approximately linear, the new velocity of the moon, when it's at the blue circle marking its new position $x_1$ after the first time step, is $v_1 = v_0 + \frac12(a_0 + a_1)\Delta t$. Thus, after the first time step, the moon is no longer moving horizontally towards the gray circle, but again along the circle's tangent towards the spot marked with the second red circle.
Over the second time step, the moon again starts off moving towards the next red circle, but falls down to the blue circle due to gravity. In the process, its velocity also changes, so that it's now moving towards the third red circle, and so on.
The key thing to note is that, as the moon moves along its circular path, the acceleration due to gravity is always orthogonal to the moon's velocity. Thus, while the moon's velocity vector changes, its magnitude does not.
Ps. Of course, the picture I drew and described above, with its discrete time steps, is just an approximation of the true physics, where the position, velocity and acceleration of the moon all change continuously over time. While it is indeed a valid approximation, in the sense that we recover the correct differential equations of motion from it if we take the limit as $\Delta t$ tends towards zero, it's in that sense no more or less valid than any other such approximation, of which there are infinitely many.
However, I didn't just pull the particular approximation I showed above out of a hat. I chose it because it actually corresponds to a very nice method of numerically solving such equations of motion, known as the velocity Verlet method. The neat thing about the Verlet method is that it's a symplectic integrator, meaning that it conserves a quantity approximating the total energy of the system. In particular, this means that, if we use the velocity Verlet approximation to simulate the motion of the moon, it actually will stay in a stable orbit even if the time step is rather large, as it is in the picture above.
Assume the distance between the green and red balls is very short. The Moon's horizontal speed happens to be such that as it travels horizontally from the green to the red, it falls to the blue (this is what we're assuming in the first place, after all).
Now the Moon has traveled horizontally and fallen, so that it's at the same distance from the Earth as when it started.
We've just gone from step 0 to step 1.
Now redraw the diagram by rotating it slightly. The Moon is now at the position of the new, redrawn green ball, with the same velocity it had at step 0. It's going to do exactly the same thing going from step 1 to step 2, and 2 to 3, and so forth. This process repeats itself infinitesimally and continuously.
If the Moon's velocity were different, if it were moving faster or slower, or in a direction other than perpendicular to the line connecting the Moon and the Earth, then it wouldn't behave this way. Its velocity happens to be very close, in both direction and magnitude (speed) to keep it at the same distance from the Earth with a constant speed. And if it maintains a constant distance and speed over a short period of time, it will continue to do so indefinitely.
Note that this description doesn't make use of the fact that gravity falls off as the square of the distance; since the distance remains constant, all we need to assume is that it's constant for a given distance, and radially symmetric. In reality, the Moon's orbit is distinctly elliptical, something that's possible only with an inverse square force. If the gravitational force fell off as, say, the cube of the distance, then an elliptical orbit would not be stable -- but a perfectly circular orbit still would be, at least in the absence of any perturbations.
Picture this. Take your desk and put in top of the tallest building on earth. Put a penny on the desk and flick it with your finger. (The point of this is that we are only giving the penny a horizontal initial velocity and letting gravity do the rest for us, that is we are letting the 'falling' do all the work.) The penny falls and hits the ground. Now flick it a bit harder and it will go a bit further before it hits the ground. Now hit much harder. It will go much further, and if you hit it hard enough, you will get a bit of extra time in the air since the earth's curvature will start to come into play. Note that the penny is still falling in all cases. Now just keep flicking the penny until eventually you put it in orbit. That is, orbiting is just a special case of 'falling'.