Did Newton estimate the gravitational constant $G$?
Without knowing the mass of the Earth, calculating the gravitational constant is impossible from $g$ and the acceleration of the Moon. The best you can do is calculate the product of the gravitational constant and the Earth's mass (GM). This is why Cavendish's experiments with the gravity of lead weights was important, since the mass of the body providing the gravitational force was known. Once $G$ was calculated from this experiment, the Earth could then be weighed from using either $g$ or the Moon's acceleration (both hopefully yielding the same answer).
The suggestion in the previous paragraph that Cavendish's experiment resulting in a value for $G$ is still not quite right. While a value for $G$ could have been determined from the experiment, Cavendish only reported the specific gravity (the ratio of a density to water's density) of Earth. According to Wikipedia, the first reference in the scientific literature to the gravitational constant is in 1873--75 years after Cavendish's experiment and 186 years after Newton's Principia was first published:
Cornu, A.; Baille, J. B. (1873). "Détermination nouvelle de la constante de l'attraction et de la densité moyenne de la Terre" [New Determination of the Constant of Attraction and the Average Density of Earth]. C. R. Acad. Sci. (in French). Paris. 76: 954–958.
Click on the link if you read French or can find a translator. Also, the symbol $f$ is used instead of $G$.
Newton's Principia can be downloaded here: https://archive.org/stream/newtonspmathema00newtrich#page/n0/mode/2up
Follow up questions copied from the comments (in case the comment-deletion strike force shows up):
So how exactly did Newton express his universal gravitational law. Was it like this "$F_g$ is equal to $GMm/r^2,$ but I must avow that I doth not know neither $G$ nor big $M$". Or did he just assign some number "$X$" to the gravitational effect due to the Earth, which ended up being $GM$?
Philip Wood: I'm pretty sure that Newton never wrote his law of gravitation in algebraic form, nor thought in terms of a gravitational constant. In fact the Principia looks more like geometry than algebra. Algebra was not the trusted universal tool that it is today. Even as late as the 1790s, Cavendish's lead balls experiment was described as 'weighing [finding the mass of] the Earth', rather than as determining the gravitational constant. Interestingly, Newton estimated the mean density of the Earth pretty accurately (how, I don't know) so he could have given a value for G if he'd thought algebraically
Mark H: Philip Wood is correct. Newton wrote Principia in sentences, not equations. The laws of gravity were described in two parts (quoting from a translation): "Tn two spheres mutually gravitating each towards the other, ... the weight of either sphere towards the other will be reciprocally as the square of the distance between their centres." And, "That there is a power of gravity tending to all bodies, proportional to the several quantities of matter which they contain." This is the full statement of the behavior of gravity. No equations or constants used.
Who first measured the standard gravitational acceleration 9.80 m/s/s? I assume that was well known by the time of Newton?
After a quick search, I can't find who first measured $g=9.8m/s^2$. It's not a difficult measurement, but would require accurate clocks with subsecond accuracy. This is an interesting article: https://en.wikipedia.org/wiki/Standard_gravity
Actually, on page 520, Newton lists the acceleration due to gravity at Earth's surface like so: "the same body, ... falling by the impulse of the same centripetal force as before [Earth's gravity], would, in one second of time, describe 15 1/12 Paris feet." So, the value was first measured sometime between Galileo's experiments and Newton's Principia.
Was Newton (and therefore all of us!) just a tiny bit luck y that the ratios worked out so nicely. I'm not putting down Sir Isaac (perhaps the smartest bloke who's ever drawn breath in tights), but even I might notice that $\frac{g(Earth)}{a_c(Moon)}=3600=\left(\frac{r(Earth−to−Moon)}{r(Earth)}\right)^2$. If the ratio had been a little messier, say one to 47½, it might have been a little harder to spot the connection.
Newton knew that the moon was not exactly 60 earth-radii distant. He quotes a number of measurements in Principia: "The mean distance of the moon from the centre of the earth, is, in semi-diameters of the earth, according to Ptolemy, Kepler in his Ephemerides, Bidliuldus, Hevelius, and Ricciolns, 59; according to Flamsted, 59 1/3; according to Tycho, 56 1/2; to Vendelin, 60; to Copernicus, 60 1/3; to Kircher, 62 1/2 (p . 391, 392, 393)." He used 60 as an average, which results in an easily calculable square, but squaring isn't a difficult calculation anyway.
The inverse square law was already being talked about by many scientists at the time, including Robert Hooke. Newton used the Moon as a confirmation of the inverse square law, not to discover it. He already knew what the answer should be if the inverse square law was true. In fact, it was the orbital laws discovered by Johannes Kepler--especially the constant ratio of the cube of the average distance from the central body and the square of the orbital period--that provided the best evidence for the inverse square law.
In "The System of the World" part of Newton's Principia, he uses astronomical data to show that gravity is a universal phenomena: the planets around the Sun, the moons around Jupiter, the moons around Saturn, and the Moon around Earth. For the last, in order to establish the ratio of forces and accelerations, you need at least two bodies. Since Earth only has one moon, he made the comparison with terrestrial acceleration.
I would love to read a proof (requiring less mathematical nous than Sir Isaac had at his disposal) for the connection from Kepler's 3rd law to Newton's inverse square. Do you know of one?
A simple version of Kepler's Third Law to the inverse square law can be shown for circular orbits pretty easily. Define $r$ as the constant radius of the orbit, $T$ as the time period of the orbit, $v$ as the planet's velocity, $m$ as the mass of the orbiting planet, $F$ as the gravitational force, and $k$ as some constant. \begin{align} \frac{r^3}{T^2} = k &\iff r^3 = kT^2 \\ &\iff r^3 = k\left(\frac{2\pi r}{v}\right)^2 \\ &\iff r = \frac{4\pi^2k}{v^2} \\ &\iff \frac{v^2}{r} = \frac{4\pi^2k}{r^2} \\ &\iff \frac{mv^2}{r} = \frac{4\pi^2km}{r^2} \\ &\iff F = \frac{4\pi^2km}{r^2} \end{align}
The quantity $v^2/r$ is the centripetal acceleration necessary for constant speed circular motion.
Starting on pp 287 of Andrew Motte's translation of Principia (Book II [1]), experiments are detailed in which objects are dropped by John Desaguilier [2] from various altitudes, from parapets, scaffolds and towers on St Paul's Cathedral [3]. I think Christopher Wren--also an aficionado of gravitational physics--was the architect [4]. In Experiment #14, John Desaguilier dropped this $2$-lb lead ball [5] from an altitude of $272$ feet [6]. Newton and several lackeys measured the descent with interesting contraptions involving quarter- and half-second pendulums, you'll have to read about them in The Principia. In Newton's words: "Now the leaden globe fell in about four seconds and 1/4 of time." Newton was not actually interested in finding $g$, the acceleration due to gravity. He was interested in objects falling in a gravitational field while descending through resisting media, like air. He probably wanted to test out his brand spanking new differential equations relevant to this kind of thing.
Anyway, if we plug $272$ feet and $4.25$ seconds into $S = \frac{1}{2}g t^2$ (law of falling bodies) we get a value for $g$ of $30$ ft/s$^2$. Not bad considering the technology employed. Nowadays kids use photo gates, and fancy graphing software. Also inches and feet had not yet been standardized and I'm not sure what kind of ruler he was using. He sometimes mentions Paris feet. I guess that was some kind of a standard back then but I don't know. You have to be careful with your scholarship when reading original source material from hundreds of years ago--I researched inches and feet once upon a time, long ago. They were different all over Europe. Even within a country there was quite a bit of variation.
Incidentally you can search through Galileo until you are blue in the face but you won't find a correct value for the acceleration due to gravity. He never actually gave a value and performing calculations on experiments to which he gives results don't give a correct value. He either made an error when recording the results of experiments, used really unusual measuring tools, or bungled the experiments. I don't mean to deride Galileo. He was brilliant. He just didn't care about details which are a big concern for us, like the value of physical constants or the length of a foot, Back then everything was expressed as proportionalities and ratios and there was no need for constants. You don't need constants and you don't need standard measurements when you express everything as proportionalities and ratios. Galileo and Newton were more interested in physical relations, not so much in constants, or "just how big is a foot, really?"
Ok, this is supposed to be about the gravitational constant and not the acceleration due to gravity. So, here we go: I'll simply say that you can't find $G$ from $g$ unless you know $M$, the mass of the earth, and $r$ the earth's radius, which was known approximately by the ancient Greeks (Eratosthenes figured that one out). Newton made an educated guess at $M$. In The Motte translation of The Principia on pp 336 we read: "Since therefore the common matter of our Earth on the surface thereof is about twice as heavy as water and a little lower in mines is found about three or four or even five times more heavy it is probable that the quantity of the whole matter of the Earth may be five or six times greater than if it consisted all of water.” He had other reasons for thinking that the Earths density was about $5.5 \times$ that of water. See The Principia for further justifications. The currently accepted value for the specific gravity of the whole earth is $5.513 \times$ that of water. Newton's guesstimate was pretty good. Given a correct value for $R$ it would have given a value for $G$ that squared nicely with the modern value but he doesn’t give the calculation in the Principia, nor anywhere else that I know of, because, as I mentioned previously, he expressed everything as proportionalities and ratios and didn’t need a constant. To find $G$ we use the density of the earth, $5.5$ g/cm$^3$, to calculate the mass of the Earth and plug $M$ an $g$ and $R$ into: $g=GM/R^2$ And voila, we have $G$.
It wasn't until Henry Cavendish perfected the torsion balance that we for sure knew the mass of the earth and the value of $G$. Henry Cavendish is the first guy who actually weighed the Earth. He got really good results. He was quite the perfectionist. It took about a century for people to make improvements to his experiment. He lived 50 or 75 years after Newton. I did the Cavendish experiment in college. It was very frustrating. It took me and my lab partner about a week and endless tweaking to get crappy results. Hats off to Cavendish.
[1] The book everyone loves to skip, and which Newton himself recommended skipping, but which actually has some pretty interesting stuff in it.
[2] A quite brilliant 17th/18th century engineer and freemason and disciple/graduate student of Newton's. He was also a refugee from Papist French persecution of protestants. That's one heck of a story too. He was smuggled out of France in a barrel at two years of age while he and his protestant parents were being chased by Catholic terrorists.
[3] Which was new at the time and maybe still under construction.
[4] And also a freemason.
[5] They also used glass balls filled with mercury-SPLAT-can you imagine?!! Call out the hazmat team!!!
[6] I had no idea St. Paul's was that tall. That's 27 stories!!!