How can you trust that there is no backdoor in your hardware?
The short answer is, you can't. The longer answer: there are a few things that can be done to increase your trust in hardware, though they also just shift the root of trust elsewhere.
A first interesting question you pose is the software/hardware distinction. To not go into the discussion about the possibly blurred boundary between the two here, I'll understand "hardware" to be non-reconfigurable logic implemented in some physical device (i.e. I'll exclude firmware such as the intel ME or microcode).
Backdoors can be inserted into a physical device in a number of stages: from conceptual architecture, through logic design, up to fabrication. To ensure no backdoors are inserted, you would need to validate the whole process from the end product to the beginning.
The good news is that the initial stages are very similiar to software - in fact, logic is usually designed using hardware description languages (HDL). These designs can be audited in the same way software can be audited. The step from here to fabrication involves multiple conversions e.g. to lithography masks using synthesis software, in a similar way to how software is compiled using a compiler. These too can be audited just like a compiler. (As a tangent - the bootstrap problem is a really interesting problem where you consider the possibility of the compiler compiling your compiler being untrustworthy)
So this leaves the last step: fabrication. Validation at this stage is usually done both by inspecting the fabrication process, and by randomly sampling devices from the same production batch (produced using the same lithography masks). For instance, the masks used can be compared to a validated trustworthy copy to ensure that no backdoors are inserted at this stage. Similarly, randomly sampled devices can be delayered and inspected under an electron microscope.
However, as a consumer, these steps are usually not available to you. For most chip producers, this whole process involves a lot of closely kept trade secrets, and aren't publicly available. This is why there is a movement towards creating open-source hardware toolchains and HDL implementations of common logic modules and systems - though there are a number of problems here too.
Finally, as @knallfrosch correctly points out in the comments, backdoors may also be inserted after production, either at a distributor, while the product is being shipped to a customer, or in-place (c.f. evil maid attack). An example of such practices by the NSA has come to light through the Snowden affair. Tampering at this stage may range from hardware implants added to the device to editing the circuit on a silicon die e.g. using a Focused Ion Beam (FIB). Mitigations at this stage usually rely on these kinds of tampering leaving externally visible traces, which may be additionally enforced using e.g. tamper-evident packaging (something every-day users can do is the glitter and nailpolish technique). Furthermore, minute device-specific imperfections that are a side product of fabrication may be used to create so-called Physically Unclonable Functions (PUFs) which may be designed in a way that tampering will most certainly alter or destroy the PUF and therefore be detectable.
plonk's answer already outlines the technical options for increasing trust in your hardware, such as device inspection or open-source hardware.
However, at the end of the day, the thing is:
You need to trust someone.
Fundamentally, this is the same for any type of hardware (or service) use use: How do you know that the manufacturer or the installer of your door lock did not retain a key? How do you know that your physician does not secretly share your confidential health information with a shady company to get kickbacks?
The answer is: You cannot know for sure - but there are strong social and legal incentives for them to play by the rules.
So in the end, you can only try to choose trusworthy vendors. That's more of a social than a technical problem; some things you can look for are:
- What are the vendor's incentives? Do they seem to want to build a sustainable business?
- What laws is the vendor bound by? What to they have to lose when they break them?
- What type of scrutiny are they subjected to? How likely is it for any backdoors to be found?
The last point specifically is where things like open-source hardware help:
Even if you cannot verify the hardware, someone else (or multiple someones) may, and the risk of being found out will help to keep a vendor honest.
It doesn't matter that much
Hardware backdoors are expensive. Very expensive. You need to influence complex supply chains, and the people responsible for them. Each time a backdoor is used it risks being discovered and published, becoming useless over time, and bringing down costly reputations of people and companies. For each of these times you need to account the benefit brought to the attacker vs the cost of the attack.
Hardware is usually the most secure part of systems. The security track record of software you typically have on your computer is frankly terrible. Even if you are an above-average user, there's good chances you'll have at least one critical vulnerability in your computer software within a year.
Also, there is no way to have absolute trust in anything. As the old saying goes: the only secure computer is the one that is turned off! And disconnected from any power sources.
But that doesn't really answer your question - How can you trust it (somewhat)? There's actually a number of practical things that you can do as an interested user or developer.
Detecting hardware backdoors
In most realistic attack scenarios, a hardware backdoor will have to escalate to a intrusion on the OS level or communicate over a distance (network, radio, ...). You can put systems in place to detect these.
Intrusion detection systems can help you detect backdoors. This can be done both within the potentially compromised system and at the network level. Obviously you can only trust the former so far (read on for mitigations to that problem).
Anti-virus and rootkit detection might offer some very baseline of protection if they're heuristic-based, but most likely would be evaded by an attacker that has a hardware backdoor if they're well known.
A SDR receiver let's you monitor the airwaves for anomalies if you have reason to suspect them. Easier said than done, but within the realm of possibility in a controlled environment.
The RTL SDR costs around $25
It's worth noting that a sophisticated attacker will likely keep the detectable time window very small, so detection alone might not help you a lot.
Mitigating attacks
Using open source hardware is the easiest thing you can do. There's a number of vendors selling completely open systems, including BIOS and peripherals firmware, or disabling Intel's Management Engine, for example.
A well configured hardware firewall can prevent undesirable communication with the outside world.
Of course, the firewall itself can be compromised. Stacking several firewalls/IDS with different hardware, software and OS makes things more expensive for your attacker (and you!)
If your attacker is very sophisticated they might decide to build a mesh network to bypass your firewall entirely, so it might be a good idea to consider what other things your computer can communicate with on the same network.
Unplugging the network might not be a bad idea, actually. While you are at it, do the same to any plugged USB cables that might have embedded radio transmitters, including your keyboard. Or maybe just put your computer into a large safe, that could be safer and almost as convenient.
If you really have enough time money and willpower to do the whole computer-inside-safe thing, you're a real pain. Your attacker will avoid the network altogether. They will transmit data from your computer using sounds you can't hear, heating up your CPU or transforming your motherboard wires into a GSM antenna
Perhaps you should consider using a typewriter instead? :)