Can the absence of information provide which-way knowledge?
The OP's confusion seems to stem from the incorrect assumption that
if my detector isn't triggered I cannot see how one could argue it interacted [with the electron]
Just because the detector sometimes does not click, does not mean that there is no interaction at all.
A good way to think about this is in terms of continuous measurement. This and this are good (albeit quite involved) references for further reading on this topic.
You know that, uprange of the detector, the electron probability amplitude (or if you insist, the Dirac field) is delocalised in space. In particular, there is some amplitude for the electron to be found at the position of the detector. So in fact, the detector is always interacting with the electron (continuously measuring it). However, this interaction is weak because the detector doesn't cover all of space. Therefore the electron-detector interaction is not strong enough to cause the detector to "click" (i.e. trigger it) with 100% probability on a single run of the experiment.
More precisely, at the end of the experiment the detector and the electron (or if you insist, the Dirac field) are in the entangled state (roughly speaking) $$ \lvert \psi \rangle = \lvert A\rangle_e \lvert \mathrm{click}\rangle_d + \lvert B\rangle_e \lvert \mathrm{no~click}\rangle_d,$$ where $e,d$ label the states of the $e$lectron (or if you insist, the Dirac field) and $d$etector. You can see already that there is an interaction, because the presence of the electron changes the state of the detector (which was initialised in the pure state $\lvert \mathrm{no~click}\rangle$). You run into conceptual difficulty only if you believe that the state of the detector and the electron can be described independently of each other: in QM probability amplitudes refer to the state of the system as a whole. If you do not observe the detector to click on a given run of the experiment, the state of the electron is correctly described by $\lvert B\rangle_e$. However, in order to see interference, the electron (or if you insist, the Dirac field) must instead be in the state $\lvert A\rangle_e + \lvert B\rangle_e$. Therefore there is no interference.
The problem is that you are treating quantum objects as both classical waves and classical particles simultaneously. More specifically, you talk about them passing through one slit or the other and sensing which slit an electron goes through. But in order for the interference pattern to emerge, the electrons have to pass through both slits at a time. We can expect one of two outcomes in your hypothetical scenario:
The electrons pass through one slit at a time. Perhaps you can unintrusively detect them at one slit, but even without a detector you end up with two overlapping single-slit diffraction patterns, since we're only using one slit at a time.
The electrons pass through both slits and we get an interference pattern, but consequently your sensor detects an electron at its slit every single time.
In neither case can you have both which-way information and an interference pattern, because either the electron takes both paths, or it doesn't self-interfere.