testthat 3e

Deprecations

A number of outdated functions have been deprecated. Most of these functions have not been recommended for a number of years, but before the introduction of the edition idea, I didn’t have a good way of preventing people from using them without breaking a lot of code on CRAN.

• context() is formally deprecated. testthat has been moving away from context() in favour of file names for quite some time, and now you’ll be strongly encouraged remove these calls from your tests.

• expect_is() is deprecated in favour of the more specific expect_type(), expect_s3_class(), and expect_s4_class(). This ensures that you check the expected class along with the expected OO system.

• The very old expect_that() syntax is now deprecated. This was an overly clever API that I regretted even before the release of testthat 1.0.0.

• expect_equivalent() has been deprecated since it is now equivalent (HA HA) to expect_equal(ignore_attr = TRUE). The main difference is that it won’t ignore names; so you’ll need an explicit unname() if you deliberately want to ignore names.

• setup() and teardown() are deprecated in favour of test fixtures. See vignette("test-fixtures") for details.

• expect_known_output(), expect_known_value(), expect_known_hash(), and expect_equal_to_reference() are all deprecated in favour of expect_snapshot_output() and expect_snapshot_value().

• with_mock() and local_mock() are deprecated; please use with_mocked_bindings() or local_mocked_bindings() instead.

Fixing these deprecation warnings should be straightforward.

Warnings

In the second edition, expect_warning() swallows all warnings regardless of whether or not they match the regexp or class:

f <- function() {
  warning("First warning")
  warning("Second warning")
  warning("Third warning")
}

local_edition(2)
expect_warning(f(), "First")

In the third edition, expect_warning() captures at most one warning so the others will bubble up:

local_edition(3)
expect_warning(f(), "First")
#> Warning in f(): Second warning
#> Warning in f(): Third warning

You can either add additional expectations to catch these warnings, or silence them all with suppressWarnings():

f() %>% 
  expect_warning("First") %>% 
  expect_warning("Second") %>% 
  expect_warning("Third")

f() %>% 
  expect_warning("First") %>% 
  suppressWarnings()

Alternatively, you might want to capture them all in a snapshot test:

test_that("f() produces expected outputs/messages/warnings", {
  expect_snapshot(f())  
})
#> ── Snapshot ────────────────────────────────────────────────────────────────────
#> ℹ Can't save or compare to reference when testing interactively.
#> Code
#>   f()
#> Condition
#>   Warning in `f()`:
#>   First warning
#>   Warning in `f()`:
#>   Second warning
#>   Warning in `f()`:
#>   Third warning
#> ────────────────────────────────────────────────────────────────────────────────
#> ── Skip: f() produces expected outputs/messages/warnings ───────────────────────
#> Reason: empty test

The same principle also applies to expect_message(), but message handling has changed in a more radical way, as described next.

Messages

For reasons that I can no longer remember, testthat silently ignores all messages. This is inconsistent with other types of output, so as of the 3rd edition, they now bubble up to your test results. You’ll have to explicit ignore them with suppressMessages(), or if they’re important, test for their presence with expect_message().

waldo

Probably the biggest day-to-day difference (and the biggest reason to upgrade!) is the use of waldo::compare() inside of expect_equal() and expect_identical(). The goal of waldo is to find and concisely describe the difference between a pair of R objects, and it’s designed specifically to help you figure out what’s gone wrong in your unit tests.

f1 <- factor(letters[1:3])
f2 <- ordered(letters[1:3], levels = letters[1:4])

local_edition(2)
expect_equal(f1, f2)
#> Error: `f1` not equal to `f2`.
#> Attributes: < Component "class": Lengths (1, 2) differ (string compare on first 1) >
#> Attributes: < Component "class": 1 string mismatch >
#> Attributes: < Component "levels": Lengths (3, 4) differ (string compare on first 3) >

local_edition(3)
expect_equal(f1, f2)
#> Error: `f1` (`actual`) not equal to `f2` (`expected`).
#> 
#> `levels(actual)`:   "a" "b" "c"    
#> `levels(expected)`: "a" "b" "c" "d"

waldo looks even better in your console because it carefully uses colours to help highlight the differences.

The use of waldo also makes precise the difference between expect_equal() and expect_identical(): expect_equal() sets tolerance so that waldo will ignore small numerical differences arising from floating point computation. Otherwise the functions are identical (HA HA).

This change is likely to result in the most work during an upgrade, because waldo can give slightly different results to both identical() and all.equal() in moderately common situations. I believe on the whole the differences are meaningful and useful, so you’ll need to handle them by tweaking your tests. The following changes are most likely to affect you:

• expect_equal() previously ignored the environments of formulas and functions. This is most like to arise if you are testing models. It’s worth thinking about what the correct values should be, but if that is to annoying you can opt out of the comparison with ignore_function_env or ignore_formula_env.

• expect_equal() used a combination of all.equal() and a home-grown testthat::compare() which unfortunately used a slightly different definition of tolerance. Now expect_equal() always uses the same definition of tolerance everywhere, which may require tweaks to your exising tolerance values.

• expect_equal() previously ignored timezone differences when one object had the current timezone set implicitly (with "") and the other had it set explicitly:

  dt1 <- dt2 <- ISOdatetime(2020, 1, 2, 3, 4, 0)
  attr(dt1, "tzone") <- ""
  attr(dt2, "tzone") <- Sys.timezone()
  
  local_edition(2)
  expect_equal(dt1, dt2)
  
  local_edition(3)
  expect_equal(dt1, dt2)
  #> Error: `dt1` (`actual`) not equal to `dt2` (`expected`).
  #> 
  #> `attr(actual, 'tzone')`:   ""       
  #> `attr(expected, 'tzone')`: "Etc/UTC"

Reproducible output

In the third edition, test_that() automatically calls local_reproducible_output() which automatically sets a number of options and environment variables to ensure output is as reproducible across systems. This includes setting:

• options(crayon.enabled = FALSE) and options(cli.unicode = FALSE) so that the crayon and cli packages produce raw ASCII output.

• Sys.setLocale("LC_COLLATE" = "C") so that sorting a character vector returns the same order regardless of the system language.

• options(width = 80) so print methods always generate the same output regardless of your actual console width.

See the documentation for more details.