Can the solar system really fit in a thimble?
Neither of those statements are true. It's an easy approximation to make: a neutron star has all of that 'space' removed from between nucleons --- so we just need to know how big a neutron star of mass equal to the solar system would be. Well, the only significant mass is the sun (jupiter is about 1% the mass of the sun---negligible). If the sun were compressed into a neutron star, it would have a radius of about 10km (up to 50% or so accuracy). See this nice talk about neutron star radii.
Solar System:
So if you removed all of the 'space' between all of the atoms in the solar systems, it would form an object about the size of a large town, or small city.
Universe:
Obviously collecting all of this mass would yield a black-hole. But conceptually, using some very order of magnitude estimates for the universe as a whole, if we assume there are roughly $10^{20}$ - $10^{22}$ stars (I think this estimate is quite high), then the radius would be something like a 1-100 Mpc or roughly 10 million to 1 billion light-years.
Edit (To address the question itself):
The concept of 'size' for atoms and nuclei has some grey area, but you can define the size of a hydrogen atom, or the size of a proton/neutron to an order of magnitude. A statement like 'remove all of the empty space' is much more nebulous, and ends up being largely a question of semantics. A more accurate way of phrasing the underlying concept being addressed might be something like:
'Roughly how much volume do the dominant mass-constituents of matter take up?'
The idea is that nucleons (protons and/or neutrons) are 2000 times more massive than electrons, and thus the important component of mass. At the same time, the electrons are the dominant volume-fillers (by a factor of about $10^{15}$).
If you could compress the mass into that small a space, it would collapse into a black hole, at which point the notion of "size" becomes harder to define, with space-time being so warped. The "event horizon" radius would be about 3 km, if I get the formula correctly.
The idea of "there's a lot of space in atoms" comes from computations which state that the "radius" of the nucleus is about 10-5 that of the atom -- so we could theoretically compress the Sun into a ball with a 10 km radius or so. But the notion of "radius" is not very clear-cut when we talk about sub-atomic particles.
The "removing the space" and "atoms are mostly empty" memes for atomic nuclei are interesting, but I do grit my teeth every time I hear this.
A description that fits better with me might be "remove the electromagnetic force". Concepts of size and space of particles are based on how they interact using forces. There is no evidence that fundamental particles have any measurable extent except that based on forces - even the nuclei only take up the space that they do due to the strong force.
Various theories of everything do propose that there is a smallest unit of size, and fundamental particles might possess a "size" at this level. It's a lot lot smaller than the size of an atomic nucleus.
So, yes in fact if you removed the electromagnetic, strong and weak forces you probably could fit the solar system's particles inside a thimble. Better remove gravity too though, otherwise it would be a black hole!