Does there exist a free good molecule / atom simulation software?
It most certainly exist outside secret labs :) Like Gerben wrote, the fields are called molecular dynamics (MD) and quantum chemistry which, as computers grow faster, will be essential tools of nanotechnology and medicine.
Molecular Dynamics is currently implemented by making certain approximations in that electron motion is not explicitely modelled. In practice, empirical forcefields are matched to experimental data and molecules are essentially modelled by summing the forces on each atom and using $F=ma$, then integrating the acceleration over time.
It's far from perfect - you usually can't match all measurable physical properties of a medium like water at the same time with the same forcefields. You can't normally model shifts of covalent bonds either since that involves changing the molecular composition which breaks the forcefield definitions. On the other hand, the methods are relatively quick and you can easily simulate systems of hundreds of thousands of atoms over a timescale of at least nanoseconds (microseconds if you have access to a supercomputer :). To interpret results you need some understanding of the deficiencies of the algorithms though.
Some interesting large simulations to date was of the protein factory, the ribosome, probably the evolutionary oldest complex part of life, and of a complete virus, the Satellite Tobacco Mosaic Virus. Below is a snapshot of a simulation I did of ion-channels in a cell membrane, you see a drug molecule bound in white in the center:
I would think that the most used (free) simulators are GROMACS and NAMD, with somewhat differing strengths but both are probably OK to start with. Usually paired with decent graphical tools to visualize what happens like PyMOL or VMD.
It's not particularly difficult to learn about this by writing your own simple simulator as well, if you're so inclined. In either case, simulating a 2x2x2 nm box of argon atoms or water-molecules for example illustrates many of the concepts.
Quantum Chemistry involves getting one step lower into the motion and shifts of the electrons. Thus you can simulate the breaking and bonding of covalent bonds and you don't have to rely on empirical forcefields for every molecule/atom you need in your simulation. However it's a lot slower computationally obviously (a factor of 100 at least? I'm not sure). You might use this to model in detail the mechanism of an active site in an enzyme for example.
The reason these are important fields for nanotech and medicine is that the simulations are the equivalents of the act of compiling and testing a program in computer software design, when it comes to designing new drugs and nano-scale machinery or materials. Essentially you want methods to test your ideas without manufacturing them first, and computer simulations take advantage of Moore's law which says that computing power doubles every 18 months...
Simulating the time-evolution of thousands of molecules interacting is generally the domain of molecular dynamics. MD codes usually dramatically simplify calculation by modelling atoms classically, generally with predefined connectivity, heavy parameterisation and bonds modeled by harmonic potentials. Whilst such approaches can give decent results for things like protein conformations they completely neglect quantum mechanics and as such fail to describe chemical reactivity. If this is what you mean by 'not really scientific', well, unfortunately QM calculations are quite computationally expensive**.
Even MM calculations get computationally expensive, depending on the time step size, the specific level of theory used and the size, number and degrees of freedom of the molecules you're modelling, but it is certainly possible to do useful MM calculations on desktop computers (this is the whole basis of Folding@Home, which aggregates small jobs done by idle desktop computers around the world).
Two very well known FOSS MM codes are GROMACS and TINKER. GROMACS is now optionally GPU accelerated using CUDA, making it very, very fast even on desktop computers (if you have a decent GPU).
** There exist hybrid QM/MM methodologies that combine an embedded QM region in a MM calculation - these are usually for looking at things like explicit solvent effects on molecules of interest. There are also ab initio MD codes such as Car-Parinello molecular dynamics (CPMD) which go some way to bridging the QM/MM scale divide.
LAMMPS is the software you are talking about. It effectively implements first principle MD of "old good cold war" type.