How does Windows determine/handle the DOS short name of any given file?

If I were you, I would never rely on any version of any file system driver (be it Microsoft's, be it another OS's) to be consistent about the algorithm it uses to generate short file names. The exact behavior of the Microsoft Fastfat and NTFS drivers is not "officially" documented (except as somewhat high level overviews) thus are not part of the API contract. What works today might not work tomorrow if you update the driver.

In addition, there is absolutely no requirement that short names contain tilde characters - see for example this post by Raymond Chen.

There's a treasure trove of info to be found about this topic in the MSDN blogs - for example:

• Registry key to force Windows to use short filenames
• NTFS curiosities (Part I): Short file names

Also, do not rely on the sole presence of alphanumerical characters. Look at the Linux VFAT driver which says, for example, that any combination of uppercase letters, digits, and the following characters is valid: $ % ' ` - @ { } ~ ! # ( ) & _ ^. NTFS will operate in compatibility mode with that...

The short filename is created with the file. The algorithm works like this (usually, but see moocha's reply):

counter = 1
stripped_filename = strip_dots(strip_non_ascii_characters(filename))
shortfn = first_6_characters(stripped_filename)
while (file_exists(shortfn + "~" + counter + "." + extension)) {
    increment counter by 1
    if more digits are added to counter, shorten shortfn by 1 
    /* e.g. if counter comes to 9 and shortf~9.txt is taken. try short~10.txt next */
}

This means that once the file is created, it will keep its short name until it's deleted.

As soon as the file is deleted, the short name may be used again.

If you move the file somewhere else, it may get a new short name (e.g. you're moving c:\somefilewithlongname.txt ("c:\somefi~1.txt") to d:\stuff\somefilewithlongname.txt, if there's d:\stuff\somefileelse.txt ("d:\stuff\somefi~1.txt"), the short name of the moved file will be somefi~2.txt). It seems that the short name is only persistent within a given directory on a given machine.

So: the short filenames will be generated by the filesystem, usually by the method outlined above. It is better to assume that short filenames are not persistent, as c:\longfi~1.txt on one machine might be "c:\longfilename.txt", whereas on another it might be "c:\longfish_story.txt"; also, when a file is deleted, the short name is immediately available again.