how to make software reverse engineering difficult

Here's an example in x86 assembly:

MOV AX, 4Ch
MOV [someaddr], AX
MOV AH, 30h
INT 21
CMP AL, 5

This will take advantage of the instruction pipelining of the processor and the failure of debuggers (as they single step through the code) and can as well be an example of an anti-debug measure. This is so because int 21(DOS interrupt) along with an 30h in AH will return the dos version of the platform. However the first two lines have been so crafted that [someaddr] will become 4Ch (someaddr has been so chosen such that it contains the address where 30h has been stored). Thus, 4Ch in AH followed by int 21 will terminate the program (interrupt thingy!). Without the influence of a debugger this won't be a problem as the third line will be prefetched and the first two lines will have no effect. But debuggers will be broken as they single step the code and AH will now contain 4Ch which will of course terminate the program. You asked for the higher level version of it. For that you will want to know how gcc (or any other compiler toolchain) converts C to asm code and then decompile the above code. http://www.programmingforums.org/thread4484.html

But some really cool guys came up with a solution and actually wrote a countermeasure (debugging a loop which has been software-pipelined): read Non-transparent debugging for software-pipelined loops by Hugo Venturini, Frederic Riss, Jean-Claude Fernandez and Miguel Santana.

Another solution to it would be to make the debugger aware of the pipelining.(drop a mail to Oleh!)

In modern computing regardless of a high level language or not the result is the same because it all compiles down to a set of known instructions that the processor can execute.

Since the current generation of processors can’t process encrypted instructions options are limited, and even if this was possible there are several layers in-between that need to be factored in such as operating systems, driver etc.. which perform processor scheduling and memory management to some degree and need to understand what the request is.

Reverse engineering, could be used for many purposes including:

• Malicious code injection
• Digital rights, copy protection
• Extraction of source code, intellectual property

Fortunately we can make reverse engineering harder by:

• Obfuscation of strings, resources, entry points and memory

Coding signing will also reduce the risk that someone will run modified/malicious code if it is checked by the system running the code.

But when you get down to the point of code running on a processor it doesn’t lie.

What you are talking about is in the realm of quantum computing which may offer a solution to your problem. I can’t help you with source code on that one though.