Comparing the "Tone" of musical instruments in MATLAB

I'm not an expert in the subject, but I'm aware of a couple of audio features that can help in such problems: Linear Predictive Coding (LPC) and Mel-Frequency Cepstral Coefficients (MFCCs).

A quick search will reveal plenty of information. As an example I found this one and this one (didn't read them, but they looked relevant).

That should get you started. Depending on your interest, you can go really deep in this topic. For example, one thing is to compare the steady state of the notes played by different instruments, but my understanding is that the transient (attack) is extremely relevant perceptually.

Good luck!

With respect to your specific question, the quantity you've computed is essentially the maximum value of the spectral coherence function. The problem is that the spectral coherence is only a good measure of the correlation between two signals if the signals are statistically stationary. That is, if the probability distribution of frequencies in the signals do not vary with time.

Unfortunately, musical instrument note signals are not likely to be stationary, because the very features most important in classifying the difference between how the same note "sounds" to the human ear on different instruments are due to harmonics and modulations that are more than likely time varying over the duration of the note.

So rather than using the spectral coherence, you need a frequency domain or time-frequency domain metric that better captures the similarity between the non-stationary parts of the note spectra.

At this point, it's less of a problem of which MATLAB functions to select (although a look at this example from the Signal Processing Toolbox documentation may help you get started, if you have that toolbox). It is more a question of researching signal processing and feature classification techniques. Here you really have to go to the literature on musical acoustics. Here is just one abstract link - I don't have access to the ACM but you may have access through your university if you are a student.

Good luck with what sounds like an interesting problem !