What does a common workstation setup look like for pentesting and vulnerability research?
It sounds like you are confusing your work environment with your target environment.
Which OS should you work from? There are far more useful tools in Linux than in Windows. Use Linux (or even a penetration testing distro like Backtrack) as a work environment. If you say that you could roll your own security tools (and why would you when there is a community of developers pumping out quality tools?) then it should not take long to get up to speed in a Linux environment.
Which OS should you become proficient in testing/penetrating/audit/fuzz? That is up to you. The technical side is as doyler says: you can always run virtual machines of either OS. In fact, with only a little virtual machine networking configuring, you could create your own test network with a mix of OS'es.
Which OS should you target from a learning/gaining expertise perspective? It depends on where you want to go with it. And, you might need to provide more details on your goals for us to weigh in on that.
You have the pen-test laptop and the fuzzing server.
Pen-test laptop is typically an HP dm1z with 8GB of RAM running Win7 with VMware Workstation 8.0.1 with BackTrack 5R1 and potentially other Linux/BSD VMs. You can even run Mac OS X Snow Leopard and Lion in separate VMs using iBoot/Multibeast/Unibeast. Use of 2 ALFA USB WiFi AWUS036Hs via HakShop would be an excellent addition to this laptop, as would a nice wireless headset, and internal/external SED-SSDs (or just RAID-0 SEDs). The Rapid7 and ErrataSec guys probably all roll with something similar. However, when you go to Defcon CTFs, almost everything these days has BackTrack 5R1 running directly on the latest and greatest MacBook Air.
The fuzzing server is typically running ESX or ESXi (potentially Xen, KVM, or Hyper-V instead) with or without vSphere or other managed VM environment. Sometimes this is cloud based. Most fuzzing servers are setup for file fuzzing, not protocol -- so they host many VMs with many OSes with many Office and PDF programs that handle various types of files. VNCRobot, AutoHotKeys, and many other tools are used to automate what a user would typically do manually.
I suggest a different environment based on my own research and needs:
- A jailbroken iPad running 4.3.3 is a must, as is a new iPod Touch running 5.01
- A MacBook of some kind (probably Air) running Mac OS X Lion guest VM via BackTrack 5R1 host VM via VMware Workstation 8.01
- Full size PCIe Radeon HD cards in some sort of desktop for password cracking and general Linux/BSD/Windows use
Some of the most important applications to add to Windows would be Visual Studio 2010, IDA Pro, and the Elcomsoft packages. Most important on Mac OS X would be Eclipse (with ADT and PhoneGap) and IntelliJ IDEA Ultimate. I would reserve a Linux (or AMI) instance for Metasploit Express or Pro, perhaps even with NeXpose (or just use the community editions like I do). Some people swear by CANVAS and CORE IMPACT. I'd install CANVAS with SILICA on some sort of WiFi Pineapple sized device that had nearly unlimited (i.e. weeks on standby) battery life (or perhaps a working AC power bar like this GSM Bug).
I've always wanted to get AR working with pen-testing. It would be fun to develop pen-test or vulnerability research apps for the Microsoft Surface SUR40. Any R&D shops want to donate one to me?
You should run a Windows box with a Linux VM or vice-versa
Not to mention a lot of PoC code depends more on the language than the OS (not 100% of the time obviously)
But other than that it really depends on what particularly you were trying to perform or test against, but I've always been a fan of Linux with the Windows VM on a laptop since I've had more problems configuring the wireless in the Linux VM to do what I want vs the Windows one. (On a desktop I've never had strong feelings in either direction, but I prefer the Linux tools more often than not so it is generally my base OS)
Though really the better base OS is going to be the one you're more comfortable with in the beginning (if you have little to no experience with one)