As a company, how can we prevent penetration testers from compromising our system?
You're looking for a technical solution to a legal problem. This won't work.
What you worry about is primarily a legal problem. Penetration testers operate under a Non-Disclosure Agreement, which is the legal equivalent of "keep your mouth shut about anything you see here". Non-Disclosure Agreements, or NDAs for short, are what prevents a penetration tester from talking about the cool vulnerabilities they found when they tested ACME Corp. last week.
But why would a penetration tester honour a NDA? Because not doing so would basically destroy their career. If the company learns that a penetration tester disclosed internal information, they will sue the pen-tester for damages, which can be upwards of millions.
Furthermore, it'll completely destroy the reputation of the pen-tester, ensuring that nobody would ever hire them ever again. To a penetration tester, this means that the knowledge they have spent years or decades to accumulate is essentially worthless. Even if the idea seems sweet to a morally corrupt pen-tester, the punishment is magnitudes worse.
Furthermore, most pen-testers just have no interest in compromising a client. Why would they? It's in their best interest to ensure that the client is satisfied, so that they hire them again and again.
As for why you would not put in technical restrictions, there are several reasons. First, as a pentester, you feel like you are treated like a criminal. A lot of pentesters are proud of the work they do, and treating them like criminals just leaves a sour taste in their mouth. A pentester understands that a company has certain policies, but if a company goes above and beyond and escorting them with an armed guard to the toilet, just to make sure they don't look for post-it notes with passwords on them on their way back, they will feel mistrusted. This can and most likely will lower morale, and may cause a pentester to not give their absolute best.
Furthermore, absurd technical constraints can also just make things difficult for a pen-tester. For example, if their company-provided domain account gets blocked as soon as they start Wireshark or nmap, it takes time for that account to get reactivated. It prevents a pentester from launching all their tools to find vulnerabilities as effectively as possible, and wastes a lot of their time.
This is bad for both the pentester and the customer, and will likely result in a worse overall experience for both of them.
I'm asking if companies can use some tools, some techniques to avoid that penetration testers behaving maliciously can exfiltrate some data or compromise the system permanently.
You could record all the traffic behind your firewall or stay awake all night long watching Wireshark output but without technical skills it's going to be hard to make sense of the bits flying in front of you. A data loss prevention system is probably what you have in mind, but it is going to interfere with the pentest, unless this is precisely the equipment that you want to test.
The answer is due diligence. Before hiring a company check out their credentials. Ask questions, ask for sample reports too. Some outfits will do little more than run an automated scan and tick boxes on a template sheet. This is not what we want. What we want is talented pentesters who think outside the box and devise original, manual attacks based on their reconnaissance efforts (which for the most part are automated). A good pentest should be a tailor-made operation and not a cookie cutter exercise.
I wouldn't do business with a company that won't provide sample reports (curated reports of course).
My biggest worry is not dishonesty but rather lack of competence which means you pay for a useless deliverable.
So this is my first filter. A box-ticking company is in my opinion less ethical because it knowingly provides a service of questionable value. Probably better than nothing but you want value for money.
I cannot remember one single instance of a pentest company being used for criminal actions. However a few have been sued for what would amount to 'malpractice'. Example: Affinity Gaming vs Trustwave.
The contract should be clear as to what is allowed and what isn't. Make sure there are no misunderstandings and that the person hiring you has full authority. What could possibly go wrong: Iowa vs Coalfire
Surely, 'rogue penetration testers' (oxymoron) who want to break into your systems won't ask for your permission to test and then go beyond the scope of the assignment. They will just invite themselves.
Don't know if you are one of those, but some companies/government agencies require a security clearance. That raises the bar a little bit: felons are unlikely to have a clearance. There are exceptions like Mr Snowden, 0% risk doesn't exist.
If your company is involved in objectionable activity, damaging the environment, selling weapons to tyrants then you may legitimately be worried about whistle blowers. This is a conundrum - you have sensitive information and want to keep it secret, but to protect it you must allow an outsider to have access to it. You should select a provider that has experience working with companies in your field of activity and is comfortable with what you do. Perhaps your trade body or business partners can provide recommendations. Word of mouth.
If you think your company could sustain financial damage/fines in case of data exposure (accidental or otherwise), talk to your insurance company. By the way, the pentest company should have liability insurance too. This is one question to ask.
Breaches are a fact of life. Pretty much any company has been hacked at least once, or will be hacked in the future. This is something to consider. You should have a disaster recovery plan ready, regardless of whether you decide to proceed with the pentest. Which I think is a good idea: if things go wrong at least you can demonstrate that you undertook reasonable efforts to prevent a breach. A company that is found to be negligent can expect stronger sanctions in terms of: regulatory fines, lawsuits, consumer backlash, negative media exposure, shareholder revolt etc.
If you're developing products, you should have an SDLC pipeline with different environments like DTAP, where you should have penetration testers testing on the acceptance environment. It's a security best-practice to keep the environments completely seperated from production. So your acceptance environment should be a functional copy of your production environment, but it should not contain production credentials, data from your users, connections to production environments, etc.
Creating an acceptance environment for a company is often a challenge on the network and server level. If this is the case, you can make a claus saying they should stop the moment they hit a production system. If penetration testers do manage to find production credentials somewhere, or hack a production server by accident, you just act like a real breach happened - except for calling the police - and the change login credentials / monitor the server / etc.