In-place upgrade from Windows Server Standard to Enterprise or Datacenter
Solution 1:
I know this question was originally regarding Server 2008, but I thought I'd post an easier answer that applies to Server 2008 R2 (and Win7 for that matter). I got the original info from TechNet Server Core blog post.
Starting with Windows 7 and 2008 R2, it's possible to upgrade editions in-place with the dism utility. You can only go upwards in edition (i.e. Standard to Enterprise) and you can't reverse the change. You'll also need the associated KMS Client Setup Key for the version you're going to. Here's a link to the list of KMS Client Setup Keys
To determine the installed edition, run:
DISM /online /Get-CurrentEdition
To check the possible target editions, run:
DISM /online /Get-TargetEditions
Finally, to initiate an upgrade, run:
DISM /online /Set-Edition:<edition ID> /ProductKey:XXXXX-XXXXX-XXXXX-XXXXX-XXXXX
For example, to upgrade to Enterprise from a downlevel version, run:
DISM /online /Set-Edition:ServerEnterprise /ProductKey:489J6-VHDMP-X63PK-3K798-CPX3Y
Solution 2:
I fired up a VM with Windows 2008 Standard on it, and then upgraded it to Enterprise. At first I couldn't because I didn't have enough disk space available on the C drive. It appears that 15-20 Gigs of free space is required. Once I made the C drive larger it upgraded without issue.
When there wasn't enough space it wouldn't upgrade and would require that I install a new OS much like you said you saw.
After I clicked upgrade, the installer confirmed that I wanted to upgrade then it started the upgrade. It's still running at the moment, but I'm sure it'll finish eventually.
Solution 3:
Microsoft seems to say so: http://technet.microsoft.com/en-us/library/cc755199(WS.10).aspx
This discussion seems to indicate that it's a matter of starting SETUP.EXE from inside the existing OS, as you did: http://social.technet.microsoft.com/Forums/en-US/winserversetup/thread/a9fb29a1-9cad-4d01-9e65-ce9b2b88232b
I'd say spin up a scratch box and give it a try... (or start a community IT support site and wait for some poor addicted fool to do it and report back on whether or not it works... >smile<)