Detect via Apex/Visualforce if org has MyDomain enabled

I would just check:

Boolean isMyDomainEnabled = URL.getSalesforceBaseUrl().getHost().endsWith('my.salesforce.com');

Or maybe:

Boolean isMyDomainEnabled = URL.getSalesforceBaseUrl().getHost().contains('--');

That should return true for all orgs with MyDomain enabled, as the url format there would be domain--sandbox.cs42.my.salesforce.com. In orgs without, it would just be cs42.salesforce.com.

Try URL.getSalesforceBaseUrl().getHost();. In some cases you might also get useful info out of a query such as [SELECT fullPhotoUrl FROM User WHERE userType = 'Standard' LIMIT 1].

Credit to Cristophe Coenrats for both. His repo here https://github.com/ccoenraets/lightning-component-apex-continuation contains code that allows Apex continuations in Lightning, and as a result his Visualforce and Lightning components need to successfully guess each other's base URLs in order to communicate successfully.