What are the key differences between Mac OS and Linux that prevent application compatibility?
Mac OS is based on a BSD code base, while Linux is an independent development of a unix-like system. This means that these systems are similar, but not binary compatible.
Furthermore, Mac OS has lots of applications that are not open source and are build on libraries that are not open source. Because of this reason, it is not possible to port those applications to run on Linux without being the copyright owner of those applications and libraries.
From using all 3 major Operating Systems (considering Ubuntu as the third, as the representative of Linux,) I can tell you that Ubuntu is very similar to Mac OS X in simplicity, also I have noticed window dialogs tend to be remarkably similar to that of which we see in Ubuntu. While Ubuntu maintains certain factors that are similar to Windows, such as the default taskbar (that is typically replaced anyhow.)
From a usability respect, both Operating Systems are nearly equal.
As twxwikinger said, the major difference between Mac OS X and Ubuntu would have to be that Mac OS X is closed source, as he explained. Essentially, Ubuntu is free due to it's Open Source licensing, Mac OS X; due to being closed source, isn't.
Beyond that, Mac OS X and Ubuntu are cousins, Mac OS X being based off of FreeBSD/BSD, and Ubuntu being Linux based, which are two separate branches off of UNIX.
Mac OS X is just basically just a special kernel (because Apple can lock down what hardware gets used so effectively, they can optimize without having to cater to myriad hardware) and a GUI pasted on top of UNIX. It's also incredibly close to completely compatible with Linux technically except for the steps Apple has taken to keep open source away. It's quite possible to drop a linux packaging system like apt-get into an OSX installation and have it work fine.
VLC actually sued them to remove the VLC Player from the App Store because they didn't want to see them benefit financially by their "1-way" street of taking open code, changing it, and then locking it down.
Really I guess I'm trying to say that OSX and Linux are more alike than either community would probably lead you to believe. And like Grayson was saying, both of them are incredibly close to their grandfather UNIX - you can effectively learn 90% of both OSX and Linux systems by just studying UNIX.