Is the hierarchyid CLR open-source?
A lot of the .NET source code is available, and much of it is Open Source, though under various licenses. Still other code is available, but only for reference purposes and cannot be used in your code.
Given that the stated goal is:
I would like to increase that limit for a complex existing application that is bound to hit this limit. I don't wish to change the interface of the type. I am not confident that I could pull off changing the interface of the type without introducing subtle bugs into all the code that would have to be changed as a result.
if the intention is to update the current Hierarchy type, then whether or not the code is Open Source is irrelevant because even if it is, you would never be able to introduce any changes into the overall system: even if you could replace the Assembly in SQL Server, you wouldn't be able to sign it with the same Certificate / Private Key and hence it wouldn't be used by the system.
However, you could build your own SQLCLR User-Defined Type (UDT) to provide the desired functionality.
Still, as it relates to gaining a better understanding of the .NET Framework:
You can view the source code of most stuff here: https://github.com/Microsoft/referencesource
and even more stuff here: https://github.com/dotnet/corefx
Both of the repositories noted above are Open Source and governed mostly under the MIT license, though you need to review the licensing info in each location, and sometimes in various folders, to be certain.
A lot of the remaining code can be found here: http://referencesource.microsoft.com/
The code found at ReferenceSource.Microsoft.com (the link directly above) that is not also found at the first two links (which are Open Source) is not Open Source and is instead governed by the MICROSOFT REFERENCE SOURCE LICENSE (MS-RSL), the relevant portion being (lightly edited to put the definition of "reference use" just following the usage of that phrase):
the Licensor grants you a non-transferable, non-exclusive, worldwide, royalty-free copyright license to reproduce the software for reference use: use of the software within your company as a reference, in read only form, for the sole purposes of debugging your products, maintaining your products, or enhancing the interoperability of your products with the software, and specifically excludes the right to distribute the software outside of your company.
However, it appears that the Microsoft.SqlServer.* namespace is not available in any of those locations. Which means that the simple answer to the question of: "is HierarchyID Open Source?" is: No.
If the desire is to see the source code, then you would need to open the DLL -- Microsoft.SqlServer.Types.dll -- with a decompiler. BUT, there are potential legal and/or ethical issues to deal with. This is proprietary code so there are two main points to consider:
as stated in the SQL Server End-User License Agreement (EULA) (and as quoted in @TomV's answer), you can only "reverse engineer, decompile or disassemble" non-provided code "only to the extent that applicable law expressly permits". Meaning that for some folks, it is perfectly legal to do this, but for other folks, not so much. So you might want to consult a local intellectual property lawyer.
if it is legal for you to decompile, the intent of the legal language, plus the fact that some source code is provided (as linked at the top) but this particular code is not, certainly implies that the desire is for the general public to not see or have this code. So being legal to decompile shouldn't be taken as it also being legal to duplicate, but more so that it can be used as a reference (i.e. for debugging, etc).
As srutzky pointed you to some locations in his answer where some of Microsoft's source code is available and the function you requested isn't listed, I think we can conclude the source code isn't available under any open source license, and it looks like your question was about it being open source, not about being able to see the source code.
Given that the EULA states the following:
In doing so, you must comply with any technical limitations in the software that only allow you to use it in certain ways. You may not:
...
reverse engineer, decompile or disassemble the software, except and only to the extent that applicable law expressly permits, despite this limitation;
So I'm afraid you're left to implement the logic yourself.