Change inactive selection color in Firefox

No, you can't

Not on Firefox at least.

Reason I'm answering with a no, is to save both of your time and others who might try to find some solutions / hacks.

Since you already know about the css specification. I might want to add that,

Remember Firefox has it's own version of ::selection, ::-moz-selection. It also has it's own version of :window-inactive, :-moz-window-inactive. Unfortunately using these things together doesn't work.

Source: CSS Tricks

/* Does work */
::-moz-selection {
  background: rgba(255,0,0,0.9);
  color: white;
}
/* Doesn't work */
::-moz-selection:-moz-window-inactive {
  background: rgba(255,0,0,0.3);
}
/* Nor this */
:-moz-window-inactive::-moz-selection {
  background: rgba(255,0,0,0.3);
}

Also, Bugzilla has years old bugs requesting this feature and talking about it's inability to handle inactive selections but no responses on those. Here is a list. Some of them are even 11 years old. I am planning to talk to someone about this and report a new bug myself with some more details, might add their response or the bug number here so that you can get updates.

So, for now I think you shouldn't be looking for some hacks, it'll only waste your time.

Thanks

Update: here is the bug to keep an eye on bugzilla, lets see what the dev team has to say.