Why isn't optional used for instance variables?

Not all Java developers even agree with the approach you described. Please check this post by the creator of Lombok.

I guess that the reason for a different approach of using Optional in Java is that Java's community lived without it up till Java 8, so most people were used to null. In one hand a lot of new APIs (like findAny from Stream) return Optional, but still a lot of standard library methods just return null, so you always have to remember either to wrap your function calls with Optional.ofNullable or check if the value is not null.

Optional was added to Java 8 but it is discouraged to use it as class fields because Optional is not implementing Serializable (and Java's serialization is used as the default serialization engine by many frameworks or systems like Akka, Spark, Kafka, etc.).

On the other hand Option is very tightly bound with the Scala standard library. As far as I know, no Scala's standard library APIs return null, but Option, and it is discouraged to use null at all in your Scala code. You can even configure your project that it will fail to compile if null is used.

Option is also Serializable and its normal practice to use is as class fields for values that can be empty.

If you want to use a similar approach in your Java code please check Option from Vavr. It's serializable, so it can be safely used as fields and it also has two subclasses None and Some (similarly as Scala's Option), so it can be used in Vavr's pattern matching:

Match(option).of(
    Case($Some($()), "defined"),
    Case($None(), "empty")
);

In Scala, Option is tightly integrated into the language's API.

Represents optional values. Instances of Option are either an instance of scala.Some or the object None. The most idiomatic way to use an scala.Option instance is to treat it as a collection or monad and use map,flatMap, filter, or foreach.

As you can see from the quote above, there is no null explanation there, because it should be used as a mondad or as a collection.

In Java, Optional is used to save us from NullPointerException cases, by using it as:

A container object which may or may not contain a non-null value. If a value is present, isPresent() will return true and get() will return the value.

A programming language that shows the user very clearly if the variable can be null or not is Kotlin, by using ? safe call, and shows you compilation errors:

var a: String = "abc"
a = null // compilation error

var b: String? = "abc"
b = null // ok
print(b)

There is the position, that Optional, like Stream, is an iterating «event» class, not deserving to be used as field in a "real" object.

Then there is the short-coming of not being Serializable (which one may work around).

However my opinion is that some variables may be optional, of cardinality 0 and 1. Just as a list is valid too. With List fields the trend is (IMHO) that it is not let to be null initially, but to always have an (empty) list.

In the same coding style, an Optional ensures that only safe access is used, and especially one may map in a chaining style:

Optional<TextField> t = ...

String s = t.map(t::getText).orElse("N/A");
Optional<String> u = t.map(t::getText);
t.ifPresent(s -> { ... });

Especially the ifPresent ensures not working with null unintendedly.

Optional is a valuable specification.