Swift Generics 2: Existentials Boogaloo

It’s been just over two years since we first saw “Improving the UI of generics,” the discussion post about potential changes to make generic types easier to work with in Swift, and five years since the first version of the Swift Generics Manifesto. (Time flies when you’re building a language!) Last week, generics landed back in the spotlight, as Anthony Latsis, Filip Sakel, and Suyash Srijan proposed SE-0309, with a major change that addresses one of the most infamous errors in Swift:

Protocol can only be used as a generic constraint because it has ‘Self’ or associated type requirements.

While this change is still in review (through May 1), it looks very promising — but, as often happens when dealing with the corners of the type system, the details can be a bit opaque. Much like last time, this post will walk through the nitty-gritty of the proposed changes in SE-0309, and offer some practical scenarios where they might improve day-to-day Swift usage.

Last Time, on Swift Evolution…

First, let’s start with a quick review of where the Swift langauge is right now. There are a couple significant concepts it’s important to have a grasp on before we dive into the proposed changes.

Most of us will be familiar with protocols and how to conform to them. We most often use protocols to define some common behavior, and conform when specific, concrete things have that behavior:

protocol Shape {}

struct Rectangle: Shape {}
struct Circle: Shape {}

We start to veer off the beaten path almost immediately, though, with the term “existential.” An existential is closely related to a protocol — but while a protocol is a pretty abstract type, the word “existential” is used in relation to concrete values. More specifically, we say “existential” when we mean one of two things:

  • A protocol when it’s being used as a concrete type for a value
  • A specific value, with “only” a protocol for a type

Clear as mud? Let’s walk our Shape protocol through a couple of different spots in the language, and try to identify the differences.

// This is a protocol type - the initial declaration of the Shape protocol.
protocol Shape {}

// Conformances also refer to Shape as a protocol.
struct Rectangle: Shape {}
struct Circle: Shape {}

// Here, Shape is used as an existential type — it is for a particular value.
//
// Even though you and I can read the body of the method and see the "real"
// type is either Circle or Rectangle, the only thing that callers know is
// that it's a Shape.
func myFavoriteShape() -> Shape {
    // On weekends I like circles, but during the work week it's all rectangles.
    if Calendar.current.isDateInWeekend(Date()) {
        return Circle()
    } else {
        return Rectangle()
    }
}

// Shape is also an existential type here, and we might refer to `favorite`
// as an existential value. We don't know the "true" inner type.
let favorite: Shape = myFavoriteShape()

SE-0309 refers to existentials as a kind of “box,” where any conforming type can go in the box. If we had said var favorite: Shape in the above example, we could later replace favorite with a different Rectangle() or Circle(), or some hypothetical Line(), or any number of different concrete types — so long as they all conform to the Shape protocol, we can swap out the existential’s underlying concrete type all day long.

Someone? Anyone? Bueller?

The last big generics proposal, in 2019, discussed adding a couple of keywords to the Swift language: some and any. These are also closely tied up in the new proposal, so let’s recap these two as well.

The first keyword, some, made it into the Swift language early in the 5.x series, right around the introduction of SwiftUI. It can be used in the type of a property or a function’s return value, as a sort of “reverse generics.” When we see some in code, it means that the value is one particular concrete type that conforms to a protocol.

This sounds a whole lot like an existential, right? The big difference is in how they behave. While an existential is a “box” that can hold different concrete types, as long as they all conform to the right protocol, a some type is a fixed (albeit hidden) single concrete type conforming to that protocol. That concrete type is chosen by the implementation of the property or function that returns it.

In practice, this means there are some differences in how we can use values that we get from one of these properties or functions. For example, we can reassign a local variable with an existential type, due to its “box-like” behavior. The existential box will then forward any Shape protocol methods to whatever “real” type is inside the box.

func myFavoriteShape() -> Shape {
    return Calendar.current.isDateInWeekend(Date()) ? Circle() : Rectangle()
}
var favorite = myFavoriteShape()
favorite = Circle()  // Totally okay! Changing the contents of the box.

But we can’t do the same with a variable of a some type, because the concrete type has been picked for us. There’s no abstracting box here — just one specific object, whose type is held by the compiler. This shows up in two different ways: first, we can’t reassign a result variable after the fact, and second, we can’t even have varying types inside the method itself! some relies on the type being known at compile time, even if it’s “hidden” from the callers themselves.

func myFavoriteShape() -> some Shape {
    // `some` methods can only have a single, consistent internal return type
    return Rectangle()
}
var favorite = myFavoriteShape()
favorite = Circle()  // error: cannot assign value of type 'Circle' to type 'some Shape'

This is a pretty subtle distinction, and the various authors involved in Swift proposals know it. In fact, the general confusion between protocols and existentials has been a thorn in the side of Swift developers for years. When we see folks talk about the “spelling” of protocols and existentials, this is what they’re referring to — developers type exactly the same characters to refer to Shape, the protocol, and Shape, the existential type. That blurs the line between protocols and existentials, making it hard to discuss and reason about.

One proposal to clear up the difference is to introduce the any keyword for existentials. Sharp readers may recall this was a topic the last time around, in the “Clarifying existential types” section of the generics UI post, but (unfortunately?) any didn’t make it into the language back then. The keyword is back with a vengeance, though, both in the discussion of SE-0309 and in the Future Directions section of the proposal.

In the proposed future, we’d use any to mark the use of a protocol as an existential type — or put another way, whenever we’re making a specific value with a protocol type. Let’s reconsider that first example snippet with any involved in the syntax.

// Protocol types are spelled the same as before, with no extra keyword
protocol Shape {}
struct Circle: Shape {}
struct Rectangle: Shape {}

// Existential types would get an explicit `any` in their spelling
func myFavoriteShape() -> any Shape {
    return Calendar.current.isDateInWeekend(Date()) ? Circle() : Rectangle()
}
let favorite: any Shape = myFavoriteShape()

While any isn’t an integral part of SE-0309, I’m still hopeful it can come to the language soon. But we’ve spent enough time on recap — let’s get to the actual proposal before us!

Return to Your Regularly Scheduled Syntax Changes

The big hook for this proposal is that it would remove that well-known error:

Protocol can only be used as a generic constraint because it has ‘Self’ or associated type requirements.

This error has haunted developers pretty much since the origins of Swift, and related type concerns have some great discussion in one of Brent Simmons’ blog posts from all the way back in 2015. It’s maybe one of the most confusing things to run into today, so I’m personally thrilled about this proposal.

SE-0309 goes about alleviating this error in two different ways, because it turns out the error crops up in two different scenarios. (You might have guessed this due to the “or” in the error text.) Since the error has two different origins, the proposal talks about getting rid of it in two different ways. Let’s tackle each one individually.

Episode I: Associated Types

First, the error shows up when the protocol has an associatedtype in its definition. Remember, an associated type is another generic type that’s tied to the protocol. For example, we might decide that our Shape needs to know how it’s going to be drawn onscreen, so we could add an associated type for the object that is going to render the shape.

protocol Shape {
    associatedtype Renderer
    func render(with renderer: Renderer)
}

In current Swift, this would immediately prevent our Shape from being used as an existential type, just because of the presence of an associated type. This is a bummer for developers writing their own protocols, but lots of Swift standard library protocols have these associated types, and are restricted by this rule, too.

For example, every Sequence — including common types like Array, Set, and Dictionary — has an associated type named Element, which says what the type the objects in that sequence are. Likewise, everything that’s RawRepresentable has an associated RawValue, which lets us know what type we’ll get when we call rawValue.

Thankfully, this first restriction is actually dismissed relatively quickly in the proposal text. SE-0309 says:

The first condition is a relic of an incomplete implementation of protocol witness tables that didn’t allow associated type metadata to be recovered dynamically.

That’s a very concise, compiler-oriented description of the problem, and we don’t need to delve into all the details right now. At its heart, this is saying that older versions of Swift couldn’t always figure out everything they needed to about an associated type. Newer versions of the language improved this implementation, so having an associated type doesn’t need to automatically disqualify the protocol from use as an existential type anymore.

Episode II: Self Requirements

Second, the error appears if the protocol refers to Self. This capital-S keyword means “the type I actually am right now,” and shows up in very core protocols like Equatable — the two arguments to == are of type Self, so that any two identically-typed Equatable values can be, well, equated.

This condition is where things really heat up. It turns out the underlying goal of the current restriction has to do a lot with existentials: if a protocol has a reference to Self, it can sometimes be impossible to use its members (required properties or functions), depending on where exactly that reference to Self is.

But why is that? Well, if we’re going to access a property or function on an existential (or anything in Swift, really), we have to know what type it is. And with existentials, the most specific we can get about the type is that it’s “something conforming to the protocol.” In current Swift, this means that any reference to Self gets type-erased — it’s translated to the closest “known” type, which is the protocol type itself.

This sounds kind of circular, so let’s consider some examples. If we were to use our Shape protocol in a more complex graphics program, we might want to make copies of a shape, or compare two to see if they would look the same onscreen. Let’s add that to our protocol.

protocol Shape {
    /// Make another Shape, of the same type and with the same properties
    func duplicate() -> Self

    /// Returns whether the Shape looks the same as another Shape of the same type
    func matches(_ other: Self) -> Bool
}

Now, if we were to have concrete implementations for these on our shapes — Circle and Rectangle — how might we go about using them? If we know a particular variable is a Rectangle, it’s easy.

struct Rectangle: Shape { /* let's take this for granted */ }

let r1 = Rectangle()     // This infers type Rectangle
let r2 = r1.duplicate()  // …so this is also type Rectangle
assert(r1.matches(r2))   // This works just fine!

But when existentials get involved, like with myFavoriteShape() earlier, things get a lot messier. The following snippet won’t compile in current Swift, but let’s reason through it as if it did.

let r1: Shape = Rectangle()  // This would have existential type Shape
let r2 = r1.duplicate()      // What type is this?
assert(r1.matches(r2))       // Can we call this safely?

In practice, we immediately get the “Self or associated type requirements” error on the first line here. But since SE-0309 is all about alleviating that error, let’s think through it like a compiler. Assume for the moment that we can make r1 just fine, and give it the existential type Shape — it’s a box to hold some concrete shape.

On the second line, then, all we can really know about r2 is that it’s a duplicate of r1. The return type of duplicate() is Self, so we — in our compiler-brains — would think that r2 is also a Shape existential. That’s all we can really know about it! As humans, we can say “hey, r1 is really a Rectangle“… but if r1 came from another function, or had some distant origin, we wouldn’t know that. So we’re stuck only knowing that both r1 and r2 are Shapes.

Even so, that’s okay! We declare values with existential types all the time, and it shouldn’t make a difference if we have one more. The real problem surfaces on the last line, with the call to matches(_:). Remember that the original requirement said that the argument to matches(_:) had to be of type Self, or — put another way — the concrete types of r1 and r2, the callee and the argument, had to be the same.

But all we know about r1 and r2 are that they’re Shapes. There isn’t a guarantee that they are the same concrete Shape, inside the existential box. This means the concrete implementation of matches(_:) that will get called on r1 might be passed a value for other that is an entirely unexpected type, and then it’s almost guaranteed to crash trying to compare properties and return a result.

This issue is the origin of the “Self requirements” restriction. Even with a smarter compiler and reduced restrictions on existential use, it is sometimes possible to back ourselves into a corner where some protocol uses would be entirely illegal because of a concrete type mismatch. This was the original problem that the “Self requirements” error was meant to prevent.

At the time of early Swift, a blanket implementation across the entire protocol was an effective block for dealing with this issue. After all, if a protocol could never become an existential type with this kind of Self-referential method, then calling that method could never become a problem! But now, a few years later, hindsight lets us see that hammer was too big: there are valuable use cases being blocked by this protocol-wide rule, and a narrower implementation can help us do more with Swift.

Interviewing Members of the Cast

So how does SE-0309 propose to work around this problem? Well, as we saw, we only run into illegal-type issues with certain protocol requirements. In some positions in the protocol, it would be completely legal to use Self and then access the member on an existential. We saw this just now with duplicate(). In other positions, though, it remains unsafe to use Self; this is what happened with matches(_:).

The big change that SE-0309 would bring is to eliminate the blanket “Self requirements” error, and instead evaluate each individual protocol member separately from the rest to determine whether it’s safe to call. The heart of the proposal is in a couple sentences near the bottom of the Proposed Solution section:

…a protocol or protocol extension member (method/property/subscript/initializer) may be used on an existential value unless the type of the invoked member … contains references to Self or Self-rooted associated types in non-covariant position.

This would shift the error away from covering all uses of a protocol with an associated type or Self reference, and more towards compiler errors on specific uses that violate these rules. There’s one new term here that distinguishes “legal” from “illegal” uses, and that’s “covariant.”

Let’s take a brief detour for a definition. Imagine a class hierarchy of things conforming to Shape. We’ve been using structs up until now, but if we needed to start inheriting more behavior, we might write something like this:

protocol Shape {}
class 2DShape: Shape {}
class Circle: 2DShape {}

A covariant position is one where the parameter type (our Self) changes along with the declaring type, or “in the same direction.” Return values are a great example. The return type of Shape.duplicate() is covariant, since it would return a general 2DShape on the 2DShape class — but when we get more specific, and subclass to Circle, the duplicate() method now also returns a more specific Circle. The changes to the parameter follow the changes to the enclosing type.

The opposite kind of change is known as contravariant — a parameter that changes “in the other direction” from the enclosing type. Arguments to methods are generally contravariant; consider our earlier Shape.matches(_:) as an example. 2DShape could maybe implement matches(_:), if it were the parent class of everything that conformed to Shape, but Circle is in trouble: its matches(_:) might have to accept a value of the more generic type 2DShape. As our enclosing type got more specific, the potential argument to matches(_:) got more general. (For more about covariance and contravariance, see this great 2015 writeup from Mike Ash.)

Getting back to our proposal: SE-0309 would allow the use any protocol members that don’t reference Self at all, or that only reference Self in covariant positions. This opens up several really cool use cases, since it means that we can suddenly start using entire protocols that were off-limits up until now — we just have to avoid the specific members that have non-covariant Self requirements, and the rest of the protocol is open to us.

We’ve already seen one such example above, where we could use duplicate() on a Shape existential to get another shape back. But there are other neat alternatives available to us too, and SE-0309 provides some solid examples. Let’s adapt a few of them into our example graphics program.

I Don’t Want You Associating with Those Types

First off, we know that some graphics can get pretty involved, with tons of different shapes all on the same canvas. It’s probably a good idea to be able to uniquely identify specific Shape instances as they flow through our logic. A first pass at this might add a required property to Shape, and let’s say (for no particular reason) that our identifiers are going to use the standard library’s UUID type.

protocol Shape {
    var id: UUID { get }
}

This is all well and good, but what if we wanted tighter integration with other Swift standard library constructs? Well, it turns out Swift has a protocol for identifying things: Identifiable, which has an associated type ID. Before SE-0309, it might have been a little restrictive to add conformance to this protocol, if Shape didn’t already have a Self requirement — we’d lose a lot of flexibility with the protocol. In the proposed world, though, we could do this freely:

protocol Shape: Identifiable where ID == UUID {}

This is totally safe, because by picking a specific type for ID right up front, we are completely separating that conformance from any potential Self requirements. The compiler can know, beyond a shadow of a doubt, that any use of the id property on something conforming to Shape is going to return a UUID.

This means we can go one step further, and replace our error-prone matches(_:) method from before with something that compares identifiers, instead:

protocol Shape: Identifiable where ID == UUID {}
extension Shape {
    func matchesIdentifier(_ other: Self.ID) -> Bool {
        return id == other
    }
}

This might give us pause at first — after all, it says Self, right there in a contravariant position! However, this has become okay, because in the original declaration for Shape, we guaranteed that the Self.ID associated type was always going to be UUID. The actual identifier type is no longer variant at all with specific conformances to Shape — any concrete shape is going to use a UUID as its identifier, and so we can always call matchesIdentifier(_:) on any shape. (If it helps, think of the compiler as “replacing” Self.ID with UUID in the method signature.)

Guaranteed VIP Member Access

Another huge win in SE-0309 is that, even for protocols that still have Self (or “Self-rooted”) requirements, we would still be able to use the protocol as an existential type. We’d only be barred from accessing members of the protocol that would violate the requirements described above.

One immediate win here is that more protocols could start declaring inheritance from Equatable, which famously has a Self requirement. We danced around this discussion with matches(_:), above, but a more idiomatic way to write that might simply be:

protocol Shape: Equatable {}

With this declaration, any concrete shape would have to provide an implementation of ==, which has Self in a contravariant position (twice!). In current Swift, this would disqualify Shape from use as an existential altogether — but with SE-0309, you would still be able to use it as an existential, and just not compare two shapes without casting them first.

struct Rectangle: Shape { /* take == for granted */ }

let s1: Shape = Rectangle()
let s2: Shape = Rectangle()

assert(s1 == s2)  // Still illegal, even with SE-0309

if let r1 = s1 as? Rectangle, let r2 = s2 as? Rectangle {
    assert(r1 == r2)  // This is okay, because r1 and r2 are concrete Rectangles
}

Quiet on the Set (or Array)

So far, we’ve looked at gains that SE-0309 will bring to the use of individual variables. But a very popular use case — and one that might be the first place many new Swift developers encounter the “Self requirements” error — is trying to declare a collection whose elements are an existential type.

In our graphics app, it’s easy to come up with lots of examples. Maybe there’s a point where we need to keep track of all the shapes in a layer, or we want to perform some operation on multiple shapes. We could even conceive of a “group” or “union” graphic, that represents the combination of multiple other shapes. How would each of these look?

struct Layer {
    // All the Shapes in the layer, unordered
    var shapes: Set<Shape>
}

// Runs the given operation over all the Shapes in the collection
func apply(_ operation: (Shape) -> (), to: [Shape]) { … }

// Represents a group graphic that contains multiple simpler shapes
struct Union: Shape {
    private(set) var contents: [Shape]
}

In current Swift, this is both easy and tempting to write. We learn very quickly about arrays, and it’s natural to think of a “list of shapes,” even if Shape itself is a protocol. And this code will even work… right up until Shape gets an associated type or Self requirement, and the dreaded error rears its head. (This will happen faster with Set than with Array, because of the implicit Equatable requirement for the elements of a set, but it’ll hit all of the collections as soon as Shape violates the current rules.)

This, to me, is the beauty of the very first clause in SE-0309’s Proposed Solution section:

We suggest allowing any protocol to be used as a type…

This means that, even if some of the calling code still needs to be adjusted for other restrictions, all of these example declarations become legal! I suspect this will be a huge boon for a lot of practical, real-world code bases.

There are certainly some where it will only kick the can down the road. For example, it’s common to try to look up indexes in an array based on equality — I might want to know where in our Union a particular child Shape lives, for example.

let union = Union(…)
let index = union.contents.firstIndex(of: favorite)

This is probably still illegal, even after SE-0309, because using firstIndex(of:) relies on using == to find the element equal to favorite — and as we recall, Equatable has those contravariant Self types in its arguments.

But the good news is twofold. First, there are plenty of call sites that won’t immediately fall foul of the new rules, so this will still open lots of doors for developers. And second, the new diagnostics proposed should help folks understand which particular protocol member isn’t available and why. Lots of attention is paid in SE-0309 to good, educational compiler notes!

Sequels and Spinoffs

SE-0309 is already a big potential improvement, and I’m excited to see it finish out its review process and (hopefully) become part of the language. As with many changes of its size, there are lots of other improvements that we might see as followups, and the proposal takes time to list a few. (Our perennial favorite any keyword is among this list.)

One that gets the most attention from the authors is a very short sentence that might catch us off guard:

Deemphasize existential types.

Given how much time we’ve dedicated to existentials in this post, this suggestion might seem counterintuitive. The authors go on to explain, though, and they make a compelling case for using generic types with protocol requirements more often. A specific example in the proposal is the difference between these two:

func foo(s: Sequence)        // Sequence is an existential type here
func foo<S: Sequence>(s: S)  // S is a concrete type that conforms to Sequence

This might seem like a tiny concern, but the two are saying pretty different things. In fact, the former — where s is declared as the existential Sequence — isn’t even legal in current Swift, and while it might become okay under SE-0309, it’s far more likely that folks intend to use the generic type S: Sequence in day-to-day use. The authors encourage better communication about the difference in the Swift Language Guide.

Beyond this educational initiative, the Future Directions section also touches on simplifying the type-erasing Any* types, and otherwise improving the ergonomics of existentials. Each of these could honestly be an entire post of their own (and might be, if they make it to the Swift-Evolution proposal stage). We’ve gone on long enough for today, though!

As always, it’s thrilling to see Swift take these kinds of steps. It’s so enjoyable to be working in a language that’s not only continuing to evolve and improve the experience for developers, but one that does the evolution in public, with spirited debate from all corners and ideas welcome from everywhere. My thanks to the Swift team for five years of generics improvements, and here’s hoping for many more.

My gratitude to members of the Seattle Xcoders community, who motivated this post and offered valuable feedback on an early draft. Thank you!