Malware sandbox detection
Start with this guide -- https://github.com/hfiref0x/VBoxHardenedLoader -- it's incredibly up-to date in terms of making a VirtualBox guest VM more-difficult to detect, including techniques valid in 2017. This will definitely make your automated malware analysis initiatives and goals easier to achieve.
Consider tracing and debugging outside of the guest VM using VirtualKD, as seen in the three books 1) Windows Malware Analysis Essentials, 2) Gray Hat Hacking 4th Edition, and 3) Practical Reverse Engineering. You can sync views between Windbg and IDA Pro using qb-sync.
Prefer VirtualKD to other, userland-debugging techniques like Scylla, HideToolz, HideCon, et al. These are not as good because defeating VM detection is difficult enough. Defeating anti-debugging and associated bad behavior just takes too-much work in userland.
Pafish, like sems tool, is also a bit dated. I would suggest that you use -- https://github.com/LordNoteworthy/al-khaser -- to stress test your automated malware analysis with sandboxes such as guest VMs.
There is additional information about VM detection on these blogs:
- https://byte-atlas.blogspot.com/2017/02/hardening-vbox-win7x64.html
- https://bdavis-cybersecurity.blogspot.com
Please direct binary analysis and reverse engineering questions to https://reverseengineering.stackexchange.com/ especially if you are just starting out, since there is already a wealth of relevant knowledge that has been shared there and is frequented by and contributed to by professional reverse engineers and malware analysts.
Possibly relevant existing Q&A:
First and foremost: How can I analyze a potentially harmful binary safely?
If you want to dynamically analyze PE binaries that employ anti-analysis techniques, ScyllaHide, Immunity Debugger and Cheat Engine may be useful. See
- Getting past a whole lot of anti-debug measures for a windows exe
- Are there any OllyDbg anti-debug/anti-anti-debug plugins what work with Windows 7 / NT 6.x?
for more info.
According to the article A Look at Malware with Virtual Machine Detection, depending on the methods employed to detect whether or not the program is running in a VM it may be straightforward to patch the binary such that those methods do not execute.
A 2006 paper by Ed Skoudis mentioned in the above article called On the Cutting Edge: Thwarting Virtual Machine Detection may also give you some ideas.
The VM detection techniques employed by the binary may also depend on when it was written: according to the 2014 Symantec article Does malware still detect virtual machines?,
Most of the samples use a runtime packer with built in VM detection. Generally, this means the packer or crypter will perform the detections, not the sample itself. Malware authors have realized that it is suspicious when an application detects that it is running on a VM, so they have stopped using those features in recent years.
If this is the case with some of the binaries you would like to analyze, then these may contain useful information:
- Unpacking binaries in a generic way
- Unpacking binary statically
In addition to traditional methods, there exist binary instrumentation frameworks such as angr and valgrind that accomplish dynamic analysis without ever executing the original object code of the binary.
If none of this is helpful to you, then ask a new question on https://reverseengineering.stackexchange.com/.