Smallest audio file: MP3, Ogg, or Wav?

2019: mp3 is your friend.

Type  | Compression  | Browser Support | Quality                | Size
---------------------------------------------------------------------------
MP3   | Compressed   | Great           | Depends on compression | Small
OGG   | Compressed   | Not so good     | Depends on compression | Smallest
WAV   | Uncompressed | Good            | Best                   | Large

Browser support: mp3, ogg, wav

Of those three, Ogg would usually be smaller than MP3. Both would be much smaller than the uncompressed WAV. Of course, there may be other factors that come into play for your site such as quality (not too much of a noticeable difference for most purposes) and browser support for each type.

The file size will only affect the time it takes to download the file to the user's machine. It won't necessarily determine Javascript execution speed. There may be other things in your code causing the performance drops (unless you've narrowed it down to the file size of the audio files).