by José Paumard
New, elegant patterns that provide ways of handling corner cases in data processing pipelines
Many things have been introduced in Java SE 8 that bring deep changes to the way we write applications and APIs. This is the case with lambda expressions, of course, and also for the Stream and Collectors API. Lambda expressions induced changes to the way interfaces work in Java, bringing default and static methods. With these new tools, the Collections Framework has seen major additions in nearly all its interfaces.
Another element has been introduced: the final class Optional. This element changes the way we can write data processing pipelines built on streams for better and more-fluent code. The goal of this article is to give details about the concept of an "optional," and to show the new and elegant patterns optionals bring to our toolbox. Optionals can work very efficiently with streams, as we are going to see.
What Is an Optional?
The concept of an optional is not new and has been already implemented in several other languages. This concept proves very helpful when modeling the fact that a method call can return an unknown value or a value that does not exist.
Most of the time, methods return a default value. This is the design choice made most often. For instance, a call to map.get(key) returns null when the key is not present in the map. This is how the maps in the Collections Framework work, even if this behavior sounds doubtful. In fact, this method cannot be used to tell whether a given key is present in a map or not. Indeed, if this key has been associated with a null value (this is certainly not something you should do!), the call to map.get(key) will also return null.
The right way to determine whether a key is present in a map is to call the following code:
map.containsKey(key);
The Optional class could have been returned by the map.get() method, even if it might have been cumbersome to constantly check the returned object for its content at each call. That being said, it would be a mistake to think that the handling of corner cases is the only role of Optional. In fact, we have been handling corner cases without optionals in Java for 20 years.
The way this class has been designed brings new and very elegant patterns that, along with the patterns brought by the Stream API, provide new ways of handling corner cases in data processing pipelines.
How Can We Build Optionals?
From a technical point of view, Optional is a final class with a private constructor. In order to build an optional, we need to use one of the two factory methods provided.
The first one is Optional.of(). As an argument, it takes an object that should not be null. If we pass a null object to this method, we will get a NullPointerException. This method can be used to wrap any non‐null object in an optional.
The second factory method is Optional.ofNullable(). This method also takes an object as an argument, the difference being that this object can be null. But do not expect to get a null wrapped in an optional: if you pass a null object to this method, you will get an empty optional.
So there is no way to create an optional wrapping a null object. An optional is a wrapper that holds a non‐null object or that is empty.
Why Do We Need Optionals?
This concept of "a result that does not exist" is used in many cases. Let's take a very simple example based on the Stream API. Let's compute a max using a stream:
IntStream stream = Stream.range(‐100, 100);
OptionalInt opt = stream.max(Comparator.naturalOrder());
This max() method returns a result wrapped in an OptionalInt. Let's explore what would have been wrong if it had been an int.
A problem arises if the stream on which we compute this max turns out to be empty. Remember that a real‐life stream can be the result of a complex computation, involving mappings, filterings, selections, and the like. It can also be the result of the splitting of a main stream into substreams distributed among the several cores of a CPU for parallel computation. So, an empty stream is something that can definitely happen, even when we do not expect it.
Let's write the code that is going to be executed in the case of splitting. Let's suppose that the max() method returns an int instead of an optional.
// This part of the code is run on CPU‐1
IntStream stream1 = Stream.range(‐100, 0);
int max1 = stream1.max(Comparator.naturalOrder()); // max1 = ‐1
// And this part on CPU‐2
IntStream stream2 = Stream.range(0, 100);
int max2 = stream2.max(Comparator.naturalOrder()); // max2 = 99
int max = Integer.max(max1, max2);
Everything runs fine here, because our streams are not empty.
Let's now have a look at a faulty case, where one of the streams turns out to be empty.
IntStream stream1 = Stream.range(‐100, 0);
int max1 = stream1.max(Comparator.naturalOrder()); // max1 = ‐1
IntStream stream2 = Stream.empty();
int max2 = stream2.max(Comparator.naturalOrder()); // Suppose max2 = 0
int max = Integer.max(max1, max2); // result is 0
We can see that choosing int as the return type of the max() method leads to a faulty result for empty streams. The reason is the following: the returned value of an operation conducted on an empty stream should be the identity element of that operation. Since the max() operation does not have an identity element, this returned value does not exist. Choosing to return 0 leads to faulty results.
You might ask: what about returning Integer.MIN_VALUE? This is indeed a possibility, but we need to be absolutely sure that this int will not be converted into a long. This trick has been used in the IntSummaryStatistics class from the JDK. But it can certainly not be used in general cases.
Optional to the Rescue
The max of an empty set is not defined, and choosing a general default value is dangerous. It could lead to corrupt results in our data processing pipelines. The max is not the only operation that has no identity element; this is also the case for the min or the average.
The optional type has been introduced to handle those cases properly. It models the fact that a value might not exist. A value that might not exist is different from a value that is null, or equals 0, or equals whatever other default value you can think of.
Optionals: First Patterns
There are, in fact, two types of patterns exposed by the Optional class. The first one sees an optional object as a wrapping object, just like Integer, Long, and the like. The difference is that there might be no value in this wrapper.
There are a couple of methods to handle an optional as a wrapper: isPresent() and get().
Optional<Person> opt = ...;
if (opt.isPresent()) {
int value = opt.get(); // there is a value
} else {
// decide what to do
}
This first pattern is straightforward and easy to understand. If we see that there is no value in the wrapper, we can decide the default value or follow another pattern than the main one. It is when the get() method is called with no value in the optional that a NoSuchElementException is thrown.
Then we have variant for this pattern. We can use the orElse() with a default value.
Optional<Person> opt = ...;
Person result = opt.orElse(Person.DEFAULT_PERSON);
In this case, if the optional is empty, the default person will be returned.
This pattern is nice as long as the object Person.DEFAULT_PERSON has already been built, or it is not too expensive to build. If it has not been built, or if performance is critical in our application, then we probably do not want to use this pattern.
We can use this second variant:
Optional<Person> opt = ...;
Person result = opt.orElseGet(() ‐> Person.DEFAULT_PERSON);
In this last case, instead of passing a fully built object, we pass a supplier that will be called if this object needs to be built. A supplier is a functional interface that models a function that does not take any argument and returns a value.
If we want to throw an exception, we can use this fourth and last variant:
Optional<Person> opt = ...;
Person result = opt.orElseThrow(() ‐> new MyCustomException());
In this last case, an exception will be built if needed, and thrown by the orElseThrow() method.
This first family of patterns is quite classical: we check whether there is something in our optional. If we discover that this optional is empty, then we decide either to return a default value or to throw an exception. As a bonus, thanks to the introduction of lambdas, we can provide a constructor for this default value or this exception, in the form of a supplier.
Optionals: Second Patterns
But we can do much better than that!
By checking the methods exposed by the Optional class, we can see that there is a family of methods that is the same as what we have in the Stream interface—map(), filter(), and flatMap()—and ifPresent(), which looks like the forEach() method. Because there cannot be more than one element in an optional, it would not make much sense to call this method forEach().
Let's see how we can leverage these methods. And for that, let's use an example from the Java SE 8 for the Really Impatient book by Cay Horstmann (Addison-Wesley, 2014).
We all know that it is not possible to compute the inverse of 0 or the square root of a negative number. What was done to handle those undefined math expressions was to introduce a special number called NaN (not a number).
What about using optionals instead of this trick? Let's build an OptionalMath class that will return optionals for its operations. If the given operation can be computed on the passed parameter, the result will be put in the optional. If not, the returned optional will be empty. This OptionalMath class is as follows:
public class OptionalMath {
public static Optional<Double> sqrt(Double d) {
return d >= 0 ? Optional.of(Math.sqrt(d)):
Optional.empty();
}
public static Optional<Double> inv(Double d) {
return d != 0 ? Optional.of(1/d):
Optional.empty();
}
}
The idea here is really simple. We always return the same object type, with no exception being thrown.
Suppose we have a stream of doubles to process.
Calculating the Inverse of a Square Root, First Version
A first (and not so good) pattern we may write is the following:
doubles[] doubles = ...; // the doubles we need to process
List<Double> result = new ArraysList<>(); // the result of our processing
Arrays.stream(doubles).boxed()
.forEach(
d ‐> OptionalMath.sqrt(d)
.flatMap(OptionalMath::inv)
.ifPresent(result::add)
);
First, we can notice that, thanks to the Optional.flatMap() method, we can chain the calls to the inverse and square root operations.
Second, the ifPresent() method allows us to chain the adding of the result to the result list, regardless of whether there is a result. The final pattern is very clean and very slick, with no cluttered if‐then‐else and no exception handling. The values that cannot be processed are just naturally and silently removed from the stream.
The only problem here is that our lambda expression is mutating an external list: result. This is not so bad and will work as we expect. Still, it is a performance hit: accessing the enclosing context should be avoided for a lambda.
There is another hidden performance hit. Because we are mutating an external ArrayList, going parallel for this computation is not possible. We are missing a nice optimization opportunity here.
Calculating the Inverse of a Square Root, Second Version
Would it be possible to write real stream processing code using this OptionalMath class using a pattern that would not mutate an external list, which is something that should be avoided? In fact, the answer is yes, but we need to reconsider the way we process our data.
The natural way of processing this stream is to write it in this way:
Stream<Optional<Double>> intermediateResult =
Arrays.stream(doubles).boxed()
.map(d ‐> OptionalMath.inv(d).flatMap(OptionalMath::sqrt);
The problem is that a list of optionals is not a great idea; we would prefer to collect a list of the results directly. We need a way to map this stream of optionals to a stream of values. If an optional from that stream is empty, then it will, silently, not put any value in the stream.
So we need a function that takes a double and returns a stream instead of an optional. That stream would be empty if this optional is empty, and it would hold the value wrapped in this optional if there is one.
This can be done with the following function:
Function<Double, Stream<Double>> invSqrt =
d ‐> OptionalMath.inv(d).flatMap(OptionalMath::sqrt)
.map(result ‐> Stream.of(result))
.orElseGet(() ‐> Stream.empty());
This function can be written in a much more elegant way using method reference:
Function<Double, Stream<Double>> invSqrt =
d ‐> OptionalMath.inv(d).flatMap(OptionalMath::sqrt)
.map(Stream::of)
.orElseGet(Stream::empty);
How does this function work? Well, first it does the computation we need: it calculates the inverse of the square root and wraps the result in an optional.
Then, if there is a result (the map method is from the Optional class), it maps the result in a stream. This call to map() returns an Optional<Stream<Double>>. If the previous step returned an empty optional, this map does not do anything and returns an empty optional.
The last step opens this optional. If there is a value in it, wrapped in a stream, the orElse() call just returns that stream, wrapping the result value. If there is no value, it returns the provided empty stream. So an empty optional is converted to an empty stream in this step.
This function does exactly what we need: it takes a double and returns a stream with the inverse of the square root of this double. If this operation cannot be done because this double is negative or null, the function returns an empty stream.
So we can now rewrite our data processing pipeline in a truly "streamish" way!
doubles[] doubles = ...; // the doubles we need to process
List<Double> result = Arrays.stream(doubles).boxed()
.flatMap(invSqrt)
.collect(Collectors.toList());
There is no more mutation of an external list; this stream can be computed in parallel with nice performance gains!
Conclusion
The second way of using optionals is much more interesting than the first one. An optional can be seen as a stream, either empty or a singleton. This leads to very natural and fluent patterns that can be computed in parallel. All the faulty values are naturally removed from the stream, with no if‐then‐else, no exception to handle, and no NaN in the case of the processing of doubles. This pattern relies on a special function that might look tedious to write: one to convert an optional to a stream. We will have a new function in Java SE 9, Optional.stream() that will do this conversion directly. Our invSqrt function will, thus, be written this way in Java SE 9:
Function<Double, Stream<Double>> invSqrt =
d ‐> OptionalMath.inv(d).flatMap(OptionalMath::sqrt).stream();
See Also
About the Author
José is an assistant professor at the Institut Galilée (Université Paris 13), and has a PhD in applied mathematics from the ENS de Cachan. He has been teaching about Java technologies at the university since 1998. José has also worked as an independent consultant for twenty years and is a well-known expert Java/Java EE/software craftsman and trainer. He gives talks at conferences, including JavaOne and Devoxx. He also writes technical articles for various media including Java Magazine and Oracle Technology Network. Passionate about education, he publishes massive open online courses (MOOC) for several companies, for example, for Oracle Virtual Technology Summit, PluralSight, Microsoft Virtual Academy, and Voxxed. He also is a member of the Java community in Paris, has been one of the lead members of the Paris JUG for six years, and is cofounder of Devoxx France. Follow him @JosePaumard.
Join the Conversation
Join the Java community conversation on Facebook, Twitter, and the Oracle Java Blog!