Is there a maximum distance from a planet that a moon can orbit?

The concept you're looking for is that of a planet's Hill sphere. If a planet of mass $m$ is in a roughly circular orbit of radius $a$ about a star of mass $M$, then the radius of this "sphere" is given by $$ r_H = a \sqrt[3]{\frac{m}{3M}}. $$ For the Sun-Earth system, this yields $r_H \approx 0.01 \text{ AU}$, or about 1.5 million kilometers.

The calculation given in the Wikipedia article shows how to derive this in terms of rotating reference frames. But for a qualitative explanation of why your reasoning didn't work, you have to remember that the moon and the planet are not stationary; both of them are accelerating towards the star. This means that it's not the entire weight of the moon that matters, but rather the tidal force on the moon as measured in the planet's frame. This effect, along with the fact that the centripetal force needed for the star to "steal" the planet is a bit less when the moon is between the star and the planet, leads to the expression given above.

As pointed out by @uhoh in the comments, the L1 and L2 Earth-Sun Lagrange points are precisely this distance from the Earth. These are precisely the points where the gravitational forces of the Earth and the Sun combine in such a way that an object can orbit the Sun with the same period as the Earth, but at a different radius. In a rotating reference frame, this means that the influences of the Earth, the Sun, and the centrifugal force are precisely canceling out; any closer to Earth than that, and the Earth's forces dominate. Thus, the L1 and L2 Lagrange points are on the boundary of the Hill sphere.


What you want to look for is the Hill Sphere. The resultant distance is about 1.5 million km.


There are several different ways of characterizing whether something is "really" in orbit around the sun or Earth. One is asking which gravitational force is larger. As you found, by that standard, the moon primarily orbits around the sun, with its orbit being next to Earth's. Another criteria would be looking at which body's gravitational potential energy dominates: if you wanted to fly a rocket from the moon to outside the solar system, which body's gravity would contribute more to the energy you would need? For that question, the moon is even more clearly in orbit around the sun rather than Earth. But you can also ask at what point would Earth's gravity be so weak that perturbations from the sun would take the moon out of a stable orbit, which gives you the Hill sphere, of which the moon is well within. There are other pairs of bodies, such as Pluto and Charon, for which these different definitions also give different answers.