Detect if Visualforce Page is in Salesforce1

If you can make due with JavaScript checking you can use the method outlined in this documentation that looks for the sforce JavaScript object, and just adapt it to do your work.

if( (typeof sforce != 'undefined') && (sforce != null) ) {
    // do your SF1 mobile button stuff
}
else {
    // do your desktop stuff
}

There isn't an official way to do a server side check. See this twitter conversation for more, where the following gist by TehNrd (Jason Venable) is referenced which checks for the presence of certain parameters.

//Determine if this is SF1 app
public static Boolean isSF1(){
    if(String.isNotBlank(ApexPages.currentPage().getParameters().get('sfdcIFrameHost')) ||
        String.isNotBlank(ApexPages.currentPage().getParameters().get('sfdcIFrameOrigin')) ||
        ApexPages.currentPage().getParameters().get('isdtp') == 'p1' ||
        (ApexPages.currentPage().getParameters().get('retURL') != null && ApexPages.currentPage().getParameters().get('retURL').contains('projectone') )
    ){
        return true;
    }else{
        return false;
    }
}

There is also an Idea,"Provide a supported url param to determine if page is in SF1", on the Idea Exchange.

In the Salesforce Spring 16 release, there will be global variables and apex methods you can call to detect the user's UI theme context. This may make it easier for developers to detect the user's experience without relying on the existence of certain javascript objects or URL parameters.

http://docs.releasenotes.salesforce.com/en-us/spring16/release-notes/rn_vf_uitheme.htm#rn_vf_uitheme

The User.UITheme and User.UIThemeDisplayed global variables and the UserInfo.getUiTheme() and UserInfo.getUiThemeDisplayed() Apex utility methods are improved to support the Salesforce1 mobile app and Lightning Experience.

These existing variables and system calls have expanded the range of return values to support the new user experience contexts. Possible return values include the following.

• Theme1—Obsolete Salesforce theme

• Theme2—Salesforce Classic 2005 user interface theme

• Theme3—Salesforce Classic 2010 user interface theme

• Theme4d—Modern “Lightning Experience” Salesforce theme

• Theme4t—Salesforce1 mobile Salesforce theme

• PortalDefault—Salesforce Customer Portal theme

• Webstore—Salesforce AppExchange theme

These global variables and system calls can replace other, more fragile methods of detecting the user experience context, such as testing for the presence of the sforce.one JavaScript global. Use them to write Visualforce pages and apps that adapt to the user experience context in which they’re running.

In addition to the officially sanctioned methods above, I wanted to add a way to check if the user is on a mobile device using user agent detection in apex:

public static boolean isMobile(String userAgentString) {
    //Using RegEx, figure out if the user is on a mobile device based on the user-agent string
    if (userAgentString == null)  return false;
    Pattern p = Pattern.compile('Mobile|iP(hone|od|ad)|Android|BlackBerry|IEMobile|Kindle|NetFront|Silk-Accelerated|(hpw|web)OS|Fennec|Minimo|Opera M(obi|ini)|Blazer|Dolfin|Dolphin|Skyfire|Zune');
    Matcher pm = p.matcher( userAgentString );

    return pm.find();

}