How to call a visualforce page from a custom button?

Use a javascript for that:

window.location = 'apex/YourVisualforcePage';

This is my example of the button:

And the visualforce page:

<apex:page standardController="Account" recordSetVar="accounts">
    <apex:pageBlock>
        <apex:pageBlockTable value="{!accounts}" var="a">
            <apex:column value="{!a.name}"/>
            <apex:column value="{!a.CustomerPriority__c}"/>
        </apex:pageBlockTable>
    </apex:pageBlock>
</apex:page>

Here is the result page:

And here is an screenshot of the account page and the result of the click on the button:

Now add the button to the object page layout. Go to some of the object records page, find a lind called Edit Layout and click on it:

Now drag your button to the page and drop it in to the Custom buttons area, then save the layout: