Is there a Youtube API that gives only audio?

YES YOU CAN

Yes, it is certainly possible, here is an API for that:

https://market.mashape.com/ehmo/savedeo

While it is against YouTube Terms of Service, it is technically possible to hack your way to the audio-only download link, and that's how the API's above work.

The is how I imagine the SaveDeo API does its magic: Download ONLY audio from a youtube video

This is what the guy at SaveDeo told me:

Youtube has audio only files. Savedeo doesn't do anything beyond providing you with the link. How we're doing this is the reason why are you using savedeo, isn't it.

Regarding the Terms of Service of the YouTube API

https://developers.google.com/youtube/terms/developer-policies

YOU CAN'T :

separate, isolate, or modify the audio or video components of any YouTube audiovisual content made available through the YouTube API;

promote separately the audio or video components of any YouTube audiovisual content made available through the YouTube API;