Why is the Java 11 runtime ignoring my jar containing sun.misc classes?

In past versions, Sun went to great lengths to ensure that there was no way to tamper with the runtime the way you are trying now (had it been as easy as this, everyone could have created their own private variant of the runtime - and surely you'll understand that wouldn't have been a good thing). I don't know the details but they probably boiled down to "if package name is this or that or such or so, then loading will only happen from rt.jar" - hardcoded in the classloaders stuff. It seems very likely those enforcement measures (or similar) are still in effect.

Scrapped the rest of my answer because I think you understand very well what your options are and that wasn't the question.

This answer is written using

> java --version
openjdk 11.0.3 2019-04-16
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.3+7)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.3+7, mixed mode)

First, you create the project that will have the overriden BASE64Encoder. Let's call it sun-misc-override. Under src\main\java create the sun.misc package with BASE64Encoder class.

package sun.misc;

public class BASE64Encoder {
    public String encode(byte[] aBuffer) {
        return "Fake it until you make it!";
    }
}

If you try to compile it, you'll get a sun\misc\BASE64Encoder.java:1: error: package exists in another module: jdk.unsupported error.

That gives us a hint that we need to patch module jdk.unsupported. This is because classes from the original sun.misc package have been moved to the jdk.unsupported module when the module system was rolled out in Java 9 and then over time removed (see JEP-260).

With Maven you can configure your compiler plugin like this:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>3.8.1</version>
    <configuration>
        <release>11</release>
        <compilerArgs>
            <arg>--patch-module</arg>
            <arg>jdk.unsupported=${project.basedir}/src/main/java</arg>
        </compilerArgs>
    </configuration>
</plugin>

After com.example:sun-misc-override:1.0.0-SNAPSHOT is built, place the resulting JAR in your "main" project - like you did eg. in a lib directory. I haven't found a way to make it work with a regular Maven dependency.

Now, configure the compiler plugin in your "main" project:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>3.8.1</version>
    <configuration>
        <release>11</release>
        <compilerArgs>
            <arg>--patch-module=jdk.unsupported=${project.basedir}/lib/sun-misc-override-1.0.0-SNAPSHOT.jar</arg>
        </compilerArgs>
    </configuration>
</plugin>

(Probably due to MCOMPILER-311 I was getting a NPE when I tried to use

<compilerArgs>
    <arg>--patch-module</arg>
    <arg>jdk.unsupported=${project.basedir}/lib/sun-misc-override-1.0.0-SNAPSHOT.jar</arg>
</compilerArgs>

even though the bug was supposed to be fixed with maven-compiler-plugin 3.8.0 and it worked fine in the POM of sun-misc-override.)

Now my "main" project is called j11 and has one class:

package com.example;

import sun.misc.BASE64Encoder;

public class BASE64EncoderTest {
    public static void main(String[] args) {
        System.out.println("Testing - " + new BASE64Encoder().encode(new byte[0]));
    }
}

To run it you need to specify --patch-module again:

> java --patch-module=jdk.unsupported=lib\sun-misc-override-1.0.0-SNAPSHOT.jar -cp target\j11-1.0.0-SNAPSHOT.jar com.example.BASE64EncoderTest
Testing - Fake it until you make it!