Using mathematical typesetting packages with `XeLaTeX` and `LuaLaTeX`

unicode-math and and amsmath are not really related, the first is primarily about font setup and the latter is about defining macros for math layout structures and is more or less independent of the fonts used.

If typesetting mathematics using OpenType math fonts, it is recommended that you load both (load amsmath first). In fact the development version (v0.8p) loads amsmath automatically so just loading unicode-math will be enough in future.

For both lualatex and xelatex the default setup is to use OpenType fonts for text but classic 8bit tfm font metrics for type 1 fonts for math. For that setup it is recommended just to load amsmath.

As David Carlisle said, amsmath works with all font packages. It’s so ubiquitous, I’m not aware of any that are incompatible with it. Many newer packages want to modify how amsmath works, and amsmath does not deign to change itself for them, so it is very rare for any package to need to be loaded before amsmath. (Except document classes, which expect to come first.) I often load mathtools, which loads amsmath and adds some fixes and extensions, rather than amsmath.

If you want to use modern (Unicode) fonts for both text and math, load unicode-math. If you also use amsmath or mathtools, load that first. You’ll get a message about how unicode-math knows about the other package and is overriding some of its definitions. Although explicitly loading amsmath may in the future become unnecessary, I’ve often found that I need to support older installations for years after the latest and greatest version makes something the default.

Legacy symbol packages, such as stmaryrd, will still work with unicode-math. Unicode has added the most common symbols from traditional LaTeX packages, and unicode-math uses the same names for them when possible. Whenever two packages define the same command, load whichever package you want to have priority last, so it can override the definitions of whatever commands both have in common. If you want to have both versions available as different commands, but the packages use the same identifier, you can load a package, save a symbol under a new name with \let, and then load the other package.

If you want to use modern fonts for text and legacy (8-bit) fonts for math, load fontspec. Loading fontspec will change your fonts for \mathrm and a few other things, and if you load fontspec after amsmath, it will also change the fonts for a few more like \text and log-like operators. To turn this behavior off, give fontspec the option no-math. The unicode-math package loads fontspec, and can pass options to it. It is possible to load fontspec explicitly, and then unicode-math, if you need to resolve a clash between packages that load fontspec with different options.

You can load mathastext after setting up your text font (which can be 8-bit or modern) and your legacy math fonts. This will use the main font for letters in math mode, too. You can also set up unicode-math to do that, but it takes more effort.

Finally, you can use mathspec in XeTeX to load modern text fonts as math alphabets, along with legacy math fonts. This supports a few more use cases than fontspec and mathastext, such as using a modern font other than the main text font for math letters, or setting a sans-serif math alphabet.