XCTExpectFailure: Expected test failures explained with code examples

XCTExpectFailure was introduced in Xcode 12.5 and allows marking test failures as expected. The first time I read about this new API I was kind of confused: why wouldn’t we use methods like XCTAssertThrowsError instead?

I continued my journey and quickly realised this API is a welcome addition to the XCTest framework. In fact, it would have been a great fit in my Unit tests best practices post. Let’s dive in and explain to you why.

Why would you use XCTExpectFailure?

While developing apps or framework it’s common to refactor a piece of code. Although often temporary, it’s common to run into failing tests while the code is still being refactored. Up until we could use XCTExpectFailure I would disable those tests and only run them once I expected them to succeed.

In other words, there are scenarios in which you realise a failure is expected. Without letting Xcode know this is true, your tests would report a failure and your CI would no longer report green.

Another benefit is that Xcode will report a test as failed once an expected failure does not occur. You can see Xcode as a guard to make sure your tests match upon current expectations.

How to set expected failures

In the following example, we have an avatar color generator that we know returns the color blue for name “Antoine”. We’ve configured the test accordingly which succeeded up until our refactor:

 func testAvatarColorGeneration() throws {
     let color = AvatarColorGenerator.generate(for: "Antoine")
     XCTAssertEqual(color, .blue)
 } 

However, now that we’re refactoring the color generation code, we are temporarily getting back wrong colors. A failure of this test is expected so we want to configure the test as such:

 func testAvatarColorGeneration() throws {
     XCTExpectFailure("We're refactoring avatar colors generation, so colors don't match.")
     let color = AvatarColorGenerator.generate(for: "Antoine")
     XCTAssertEqual(color, .blue)
 } 

Running this test will still show feedback the the test failed but marks the test as succeeded:

The importance of order

It’s important to point out that the order of assertions and expectations matter. As copied from the documentation of XCTExpectFailure:

Declares that the test is expected to fail at some point beyond the call.

Only failures beyond the expected failure declarations are suppressed. If we would’ve configured our tests as follows, it would’ve been reported as failed:

Marking a specific assertion as expected to fail

In some cases we only want to mark a specific assertion as expected to fail while other assertions should still succeed.

Imagine our avatar color generator is returning a more detailed color configuration with both a background and border color. We expected the background color to change but the border should still be returned as white:

func testAvatarColorGeneration() throws {
     let avatarColors = AvatarColorGenerator.generate(for: "Antoine")
     XCTAssertEqual(avatarColors.background, .blue) // This line fails.
     XCTAssertEqual(avatarColors.border, .white)
 }

We can use a closure inside the XCTExpectFailure to only mark the background color assertion as expected to fail:

func testAvatarColorGeneration() throws {
     let avatarColors = AvatarColorGenerator.generate(for: "Antoine")
     XCTExpectFailure {
         XCTAssertEqual(avatarColors.background, .blue) // This line fails.
     }
     XCTAssertEqual(avatarColors.border, .white)
 }

This way we still benefit from our assertion to validate our border outcome. This becomes especially useful when refactoring pieces of code while having the benefit of existing unit tests during your process.

Reasons for using XCTExpectFailure over XCTSkip

I can hear you thinking:

Isn’t this a reason to use XCTSkip instead? or to just disable the test temporarily?

Well, there’s an important difference to consider when using XCTSkip instead of XCTExpectFailure. A skipped or disabled test will always be skipped, even if you’re done refactoring and your test would succeed again.

When using XCTExpectFailure, you give yourself a checkpoint that informs you to re-enable a test. If our avatar color correctly returns blue again but we still have our expected failure configured, Xcode will inform us with a failing test:

This is a great way to keep up the quality of your code and don’t end up with skipped or disabled tests while they could successfully run.

Strictness configuration

Each XCTExpectFailure can be configured with options to temporarily disable the expected failure or to set its strictness:

 func testAvatarColorGeneration() throws {
     let avatarColors = AvatarColorGenerator.generate(for: "Antoine")
     let options = XCTExpectedFailure.Options()
     
     // Set its strictness: Test succeeds, even if an expected failure doesn't occur.
     options.isStrict = false
     
     // Temporarily disable an expected failure.
     options.isEnabled = true
     
     // Pass in the options:
     XCTExpectFailure(options: options) {
         XCTAssertEqual(avatarColors.background, .blue) // This line fails.
     }
     XCTAssertEqual(avatarColors.border, .white)
 } 

Once you configured your expected failure as non-strict it will no longer mark your test as failed once an expected failure doesn’t occur.

I purposely moved this section to the bottom of this post as I’m not convinced you ever need this configuration. In my opinion, it’s always better to let your tests report the failure as a reminder to yourself to fix the failing tests. In the end: why would you mark a failure as expected while it doesn’t affect your test result?

Conclusion

XCTExpectFailure is a welcome addition to the XCTest framework and allows us to benefit from existing tests while refactoring code. Xcode will be a great companion in reporting expected and unexpected failures. It will be a challenge to forget about temporarily disabled assertions, unlike with XCTSkip or disabled unit tests.

If you like to improve your Swift knowledge, even more, check out the Swift category page. Feel free to contact me or tweet to me on Twitter if you have any additional tips or feedback.

Thanks!

<!– Disable cross link to Medium

Also published on Medium.

–>