Is there a way to get rid of the default file manager for Xubuntu without having to get rid of the desktop?

Removing xubuntu-desktop will actually do little.

xubuntu-desktop is only a metapackage.

This means that it has no files or code whatsoever, except for a control file that says it depends on thunar and other stuff, and possibly a bit of filler code. ^{[Citation needed]}. You can safely remove this package without losing anything, since all its installation did was pull in a ton of other packages, which actually make the Xubuntu interface work, that aren;t on the list of removals.

The reason it gets removed, is it "depends" on Thunar in order to get Thunar installed in a Xubuntu installation, so when Thunar is removed, xubuntu-desktop is als oremoved(without ill effect).

With that said, updates that add Xubuntu packages(uncommon) will go unnoticed, but updates on existing Xubuntu packages(common) will be performed.