Apple - Is it possible to only see contacts with phone numbers?

The native Contacts and Phone apps do not support this in iOS. As a workaround, you could use a third party contacts manager which supports smart groups, creating “Has a phone number” and “Has an email address” groups and working from these. Note that the contact lookup sheets shown by other apps (like, say, Mail) are provided by the native app and thus will not show your new groups.

A free example of such an app for the iPhone would be Tacts – searching for “smart groups” on the App Store should get you more.

The iPhone Contacts app will not let you do this. You have a few options though. One, just add the people you call a lot to your Favorites list in Contacts. The other is to rely more on searching for a contact (by scrolling to the very top of the screen) rather than swiping. I do this almost exclusively versus scrolling through the list, as I've found it much faster.