How do AV scanners in VirusTotal check if a file is malicious or not and how trustworthy is its report?
ad 1: It does upload your file, but only if the hash is not known. As the very first thing, a piece of Javascript will calculate a cryptographic hash (SHA-256 if I recall correctly, but might be wrong) and sends that. The engine then, rather than scanning, looks up the hash in a already-did-it database. Only if not present, or if you insist, it will upload the actual file.
How secure is this? Pretty much 100%, or as close as you can get to it. The odds that your file has the same hash as an already scanned file but isn't the same file are diminuitive. Not quite zero, admittedly, but as good as.
ad 2: It doesn't make much of a statement at all by itself other than "no problems were found" if there were zero hits reported. What it does is, it runsaround 60-70 different scanners, some of which are well-known, and some I've never heard of, and displays their output. Which, sometimes, contains false positives, and which may very well contain false negatives. The actual usefulness of virus scanners is being disputed, but alas it's as good as you can get. At least, virus scanners detect well-known threats relatively well.
Plus, there's the community thingie where users can give ratings, but it all comes down to trusting some unknown guy on the internet, so... bleh.
ad 3: Hardly. The best alternative is to never execute programs of unknown or even dubious origin. Anything else is reading tea leaves. Sure, some read the tea leaves better than others, but they are still only reading tea leaves. Dynamic analysis exists, pretty much every desktop AV software has that nowadays (running most system calls through a proxy lib), but whether it actually does something useful other than burn massive CPU is questionable. Virus Total may be somewhat better insofar as it runs a massive amount of scanners. Whether that actually increases security is uncertain. If you think about it: Assuming the scanners that are run are actually worth their salt, then a single one of them should do. On the other hand, if they're not worth their salt, what value is added by running more of them? Quoting an old wisdom: If you add nothing to nothing, the sum remains rather small. Or, as stated in Fidelio: "Nothing, if you add to nothing air".
If you got a file from a presumably trustworthy site, and you have zero hits on VT, then it's probably, usually, rather safe. If you keep the file around for two weeks and re-scan it again two weeks later, even better (assuming a new threat may become known in the mean time). That's what I'm doing, and I've done that years before VirusTotal even existed -- so far it works very well (or, it seems to work, I might have malware that I don't know about). In the end, you never know for sure.
About dynamic analysis
That's not happening. You can deduce what kind of analysis VirusTotal is doing from its display. It is almost certainly not just a forwarding of hashes to some other service (as suggested in a comment) because of the time it takes to scan, and because definition files are not always in sync (indeed most of the time they're not) with the manufacturers' tools. You can also unpack and re-pack a ZIP file (which almost certainly results in a non-identical file), and it will be scanned just fine. How would that work if only hashes are being handed around? It wouldn't... but it does. That, and the fact that sometimes one or another scanner fails to open an archive, indicates that actual scanning happens.
You can deduce that they do static signature matching and heuristic scanning because signature matching is what every AV does and has been doing for 30+ years by default, and heuristic analysis is what most (if not all) scanners do by default for at least 20 years, too.
On the other hand side, you can be pretty sure that no dynamic analysis is done because that is forbidding and impractical for a web service. In order to do dynamic analysis, you must run the binary in a secured environment (emulator, virtual machine or similar) which provides a complete operating system for the binary to run, and which is able to accomodate the memory and CPU requirements of a haphazard amount of a-priori unknown programs. That's hardly realistic. Plus, there is a very non-neglegible risk in running unknown software which could do basically everything including running malware services inside the VM, or breaking out of the VM. You know, for example a web browser or a mail client needs internet access to function properly. How do you provide that without also granting any malware that you run internet access? Google (the owner of VirusTotal) very certainly wouldn't like being accused of running malware services.
Lastly, you must have 100% coverage to be sure (else, the analysis is pretty pointless, malware does not necessarily do its work upon program launch), so you need either a human or the most advanced fuzzing robot in the world to provide adequate input to the program so it takes all possible paths. Do that for a web service which maybe ten million or so people use every day, and nobody likes to wait longer than a minute or two for the results, good luck.