Why is external access to a server via SSH considered insecure?

SSH is not typically considered insecure in and of itself but it is an administrative protocol and some organizations require two or more layers of control to get access to an administrative console. For example connecting via a VPN first then opening an SSH session which connects through that VPN. This simply provides multiple layers of defense in depth and prevents your systems from being directly affected by the latest SSH vulnerabilities.

Note: This is NOT a weakness in SSH itself and many organizations will still expose SSH on high TCP ports for use as SFTP servers then have a script move data to and from that system (not allowing the external SSH/SFTP server to connect to the rest of their network).

All protocols eventually have vulnerabilities so if you can require the use of two different ones (i.e. IPSEC VPN and SSH) and stay disciplined about remediation efforts when vulnerabilities are discovered then the window of time where known vulnerabilities exist in both protocols at the same time on your network should be very small. Statistically, the requirement for using two protocols should reduce the amount of time where you would have a single protocol exposed with a known vulnerability (assuming you actually patch/remediate vulnerabilities).

As a poor text graphic look the following:

Time ---> 

IPSEC ------------------------     ----------------

SSH   ---------       -----------------------------

Versus having just SSH, or a VPN, by itself:

SSH   ---------       -----------------------------

In the first example when an SSH vulnerability came out there wasn't one for IPSEC and vice versa so there was never a time, in this crude example, where your systems had vulnerabilities at both layers. This defense in depth is what is protecting the system behind these protocols which occasionally may have vulnerabilities.

In the second example, SSH by itself, the moment there is a vulnerability, or a password breach, or any number of other issues, an attacker can directly access your vulnerable system during the window of exposure.

I would ask if any other administrative protocols are being exposed and if so then you can question the technology bias but most likely you may be in an organization that doesn't allow any administrative protocols to be accessed directly from the Internet.

I asked for an explanation and unfortunately did not understand it - the contractor stated SSH was insecure but did not know why.

Why is SSH insecure? There seems to be something I missed during our conversation that I desperately want to understand.

Treating security as a binary "secure" or "not secure" reflects a poor understanding of security; security is always a continuum, and nothing is perfectly secure.

The contractor in question seems to have a large lack of information about SSH if he/she can't even state why SSH is insecure, but adamantly believes that it is. Ignorance breeds fear, and my guess is the contractor is reacting to fear rather than knowledge.

SSH is no less secure than VPNs from major vendors. VPNs have security vulnerabilities too, and aren't "made of magic". We know this from the various leaks from the NSA, and there's no reason to believe these vulnerabilities are in the hands of only the NSA.

The real question comes down to need for access vs need for security, not one protocol over another. Remote access will likely decrease your security somewhat. Two remote access methods will decrease your security method further. Whether you enable one or more methods of remote access should be judged by the tradeoffs involved, not by maintaining a false sense of "security" as a binary concept.

What is insecure, exactly?

To state it shortly, "Why is external access to SSH considered insecure?": it is not "SSH" which is insecure, it the "external access" part of your question which is.

SSH is just one technical mean amongst others to open your internal network to the outside world (which highly risky). It may be used, either standalone or associated with other technologies, in order to implement your remote access policy.

The remote access policy is the formal definition stating who can access what, when and from where, it defines all the rules which will then be implemented using various technical controls which will, in turns, provide proper authentication, authorization and audit services. All of this, of course, need to be properly documented and maintained.

Of course, you could just go on without all these administrative and maintenance burdens, but here is the point: taking such shortcut is precisely what would be insecure in your question.

So, why is external access to SSH considered insecure? Because it would cost too much to implement it properly in regards to the return on investment expected by the company.

Cost question

Cost here is not about buying software, it will merely be about paying people time first to design and setup this service, and then to maintain it over time (while these people are occupied with this there are other tasks they won't be doing). The actual cost will therefore be very dependent on the salary, current competencies and the complexity of your current infrastructure.

From a technical point-of-view, key based authentication and fail2ban is a good and well documented solution. Running this on high ports will also get it out of the majority of bots sight. But this will not prevent for instance a genuine employee from connecting to the internal network from his family computer overflowing with worms, viruses and backdoors of all sorts, thus unwittingly comprising the company's network. This is one of risks you may have to address.

From a management perspective chances are that the "boss" may be more interested in (monetary) weighting the various risks against the expected return on investments: what this will cost to setup and maintain? How much will this allow the company to earn or economize? What could it cost in case a disaster happens?

Risk is always there, with everything. If you manage to show that from a business perspective opening the internal network (or maybe some well-defined parts of it, which would be an effective way to reduce the risk) will be a profitable move for the company, you will have done half if not most of the journey. How to do it will then be just a matter of technical choices depending on what you planned to do. But you certainly must solve the question from a business and functional point-view before going down into technical details.