How to approach replacing md5 for transporting Unity game data to a remote server

This approach is fundamentally flawed. Anything on the client side can and will be tampered with by players. It is the same problem which makes DRM untenable - the user owns the machine and all the data on it, including your executables, data in memory, etc. Keeping algorithms secret doesn't work (see Kerckhoffs's principle) because it only takes a small amount of reverse engineering work to work out what your code is doing.

For example, let's say you've got a routine in your game client which posts the level score up to the server, using some cryptography of whatever form to ensure that it isn't tampered with over the network. There are a whole bunch of ways to get around this:

• Use a memory editing tool such as Cheat Engine to scan for the current score (pause, search the score, unpause, wait for score to change, search again, repeat until you find the memory address which contains the value) and edit it. When the level completes the score value will be happily treated as legitimate by your code, and uploaded to the server.
• Modify the game executable on disk so that your "level complete" code ignores the real score value and picks a different one.
• Modify the game executable on disk so that simple things (e.g. killing one monster) increases your score by 1000x more than it should do.
• Modify the game executable on disk so that you can never die, or have infinite powerups, or one-hit kills, or any other number of helping things, so that you can easily attain a very high score.
• Perform any of those modifications in-memory after the game loads so that the original executable stays intact (useful for cases where there are annoying integrity checks)
• Simply expose the "we finished a level, now upload the score" code externally from the process so that it can be called by anyone's program. In native code you can inject a small stub and add an entry to the export table, or just directly copy the code into your own executable. In .NET it's trivial to just modify the class and method's visibility flags and import it into a new program. Now you can submit whatever score values you like without ever even running the game.
• Reverse engineer the game and get hold of the "secret" key and write your own app to send the score value to the server.

This is just the tip of the iceberg. For more complex games there are all sorts of workarounds to anti-cheat and other problems, but regardless of the defense tricks used there will always be a way to mess with client-side values.

The critical feature of a secure approach is that nothing on the player's computer should be trusted. When your server code receives a packet from a player, assume that your game might not even be running - it could be a totally homebrew piece of code that lies about everything.

Securing multiplayer games (or any kind of game where verifying game state is a requirement) isn't easy. The problem isn't really even a security one, it's a usability and performance one. The simplest way to secure a multiplayer game is to keep the entire game state on the server side, have all the game logic executed and maintained there, and have the client do nothing but send player input over to the server ("user clicked the mouse, user is holding W key") and present the audio and video back to the player. The problem with doing this is that it doesn't make for a very fun game due to network latency, and it's quite hard to scale on the server side. Instead, you have to find a balance between keeping things client-side and server-side. For example, the logic for "does the player have a key to open this door?" must be checked server side, but the logic of when to show the context icon for "open door" stays on the client side. Again, things like enemy health and position must be kept on the server side, and the player's ability to deal damage to that enemy must also be verified (is it near enough?) which means that AI can't be kept client-side, but things like the choice of which idle animation the enemy is displaying will probably be a client-side thing.

We have a few questions on here about game security, so I suggest reading through those:

• Preventing artificial latency or "Lag Hacking" in multiplayer games

• Secure Software: How to ensure caller is authentic?

• Securely sending packets without them being spoofed

• Checking a locally stored string for tamper

I also recommend these questions over at GameDev SE:

• How can a web game store points online without giving the user the possibility to do the same call but with more points? (almost literally your question)

• What are some ways to prevent or reduce cheating in online multiplayer games?

• How should multiplayer games handle authentication?

There's also a great paper on multiplayer security from BlackHat EU 2013, and an article on non-authoritative P2P networking, which may both be of use.

There are three problems here:

• What you're trying to do is fundamentally misguided in several ways, as described in Polynomial's answer.

• If you insist on doing it anyway, then you're not even using the right primitive! Authenticating messages requires a MAC, not a hash. It's possible to build a MAC out of a hash -- this is what HMAC does -- but it's not as trivial as it looks. What the code you pasted is doing is trying to naively build a MAC out of a hash, and these kinds of naive constructions are usually broken.

• Also yeah MD5 is a bad hash function. But this is really the least of your worries... MD5 is not the weak point of this design. You're asking for advice on how to replace the bars over your windows with 2-inch thick steel plating -- but your front door is wide open. Justifying this as being "better security" makes no sense.

Edit: To clarify, I can't speak to the security of your approach, only the hash portion. There are some strong comments here discouraging your approach. (i.e. a good hash can be part of a secure approach or an insecure approach, just like a good lock can be used on either a good door/frame, or a flimsy door/frame)

In response to your TL;DR I can say quite assuredly that SHA-2 is the recommended drop-in replacement hash function.

If length is an issue, it is better to use a truncated SHA-2 than MD5.

SHA-2 comes in 4 sizes: 224, 256, 384, or 512; with 256 and 512 being the most common.

Keep in mind, this is a General Purpose (Fast) hash, and is not suitable for password storage, however it is suitable to ensure data integrity as your TL;DR suggests.

BCrypt is a Slow Password Hash and is used if you need a sort of one way encryption. (i.e. the password is never stored, only its hash, and is difficult to brute force)