Deprecation Tracker

In order to control the deprecation warnings that occur during a test run, we create a shitlist of deprecation warnings that we expect to see for each spec file. If code run by a spec file changes how often or what deprecation warning occurs, an error is raised after the RSpec run is complete.

It looks something like this:

An error occurred in an `after(:suite)` hook.
Failure/Error: raise UnexpectedDeprecations, message

DeprecationTracker::UnexpectedDeprecations:
  ⚠️  Deprecation warnings have changed!

  Code called by the following spec files is now generating different deprecation warnings:

  ./spec/deprecation_spec.rb

  Here is a diff between what is expected and what was generated by this process:

  diff --git a/spec/support/deprecation_tracker.json b/var/folders/mv/x81dlrp92w5053_m9bgqbkmm0000gp/T/test-deprecations20180328-33449-xgor20
  index 76d2118f9f2..257be30d46c 100644
  --- a/spec/support/deprecation_tracker.json
  +++ b/var/folders/mv/x81dlrp92w5053_m9bgqbkmm0000gp/T/test-deprecations20180328-33449-xgor20
  @@ -1,7 +1,8 @@
   {
     "./spec/deprecation_spec.rb": [
       "DEPRECATION WARNING: `ActiveRecord::Base.symbolized_base_class` is deprecated and will be removed without replacement. (called from block (2 levels) in <top (required)> at /Users/Jordan/projects/themis3/spec/deprecation_spec.rb:5)",
  -    "DEPRECATION WARNING: `ActiveRecord::Base.symbolized_base_class` is deprecated and will be removed without replacement. (called from block (2 levels) in <top (required)> at /Users/Jordan/projects/themis3/spec/deprecation_spec.rb:6)"
  +    "DEPRECATION WARNING: `ActiveRecord::Base.symbolized_base_class` is deprecated and will be removed without replacement. (called from block (2 levels) in <top (required)> at /Users/Jordan/projects/themis3/spec/deprecation_spec.rb:6)",
  +    "DEPRECATION WARNING: `ActiveRecord::Base.symbolized_base_class` is deprecated and will be removed without replacement. (called from block (2 levels) in <top (required)> at /Users/Jordan/projects/themis3/spec/deprecation_spec.rb:7)"
     ],
     "./spec/models/refund_spec.rb": [

# ./spec/support/deprecation_tracker.rb:65:in `compare'
# ./spec/support/deprecation_tracker.rb:29:in `block in track_rspec'
# /Users/Jordan/.rbenv/versions/2.4.3/bin/rspec:23:in `load'
# /Users/Jordan/.rbenv/versions/2.4.3/bin/rspec:23:in `<top (required)>'
# /Users/Jordan/.rbenv/versions/2.4.3/bin/bundle:23:in `load'
# /Users/Jordan/.rbenv/versions/2.4.3/bin/bundle:23:in `<main>'

Finished in 1.83 seconds (files took 9.09 seconds to load)
1 example, 0 failures, 1 error occurred outside of examples

When the diff shows a line was removed, it means we expected to see that deprecation message but didn't. When the diff shows a line was added, it means didn't expect to see that deprecation message.

Protip: If you find the diff difficult to read in your terminal, copy/paste it into your text editor and set syntax highlighting to "diff" for a colorized view.

Keeping a list of all deprecation warnings has two primary benefits:

We can fail CI when new deprecation warnings are added (i.e., "stop the bleeding")
We can more easily find and eliminate deprecation warnings

Modes

The deprecation tracker has three mode: compare, save, and off.

DEPRECATION_TRACKER=compare rspec foo_spec.rb: in compare mode, changes to the deprecation warnings output by a test file will trigger an error.
DEPRECATION_TRACKER=save rspec foo_spec.rb: in save mode, changes to the deprecation warnings output by a test file will update the shitlist.
rspec foo_spec.rb: when turned off, changes to the deprecation warnings output by a test file won't trigger a warning or update the shitlist.

What does it track?

This tracks deprecations from Rails and Kernel#warn, the latter often used by gems that don't use ActiveSupport.

It only tracks deprecation warnings that occur during a test and not before or after. This means that some deprecations, for example the ones you might see while the app boots, won't be tracked.

It also doesn't track constant deprecations (warning: constant Foo is deprecated) because those happen in C code. (If you can think of a way to capture these, do it!)

How do I get my pull request green on CI?

There are added deprecation warnings

If the diff shows added deprecation warnings, you'll need to go back and change code until those messages don't happen. Under normal circumstances, we shouldn't increase the number of deprecation warnings.

You can check if your test file has changed deprecation warnings by running this command:

DEPRECATION_TRACKER=compare bundle exec rspec spec/foo_spec.rb

In this case, an error will be raised if spec/foo_spec.rb triggers deprecation warnings that are different than we expect.

An example of when it would be OK to increase the number of deprecation warnings is during a Rails upgrade. We're not introducing code that adds deprecation warnings but rather change libraries so what used to be safe to call is now deprecated.

There are removed deprecation warnings

If the diff shows removed deprecation warnings, congratulations! Pat yourself on the back and then run this command:

DEPRECATION_TRACKER=save bundle exec rspec <spec files mentioned in the error>

This will rerun the tests that reduces the number of deprecation warnings and save that to the log.

Be sure to commit your changes, then push and wait for CI to go ✅.

The deprecation warnings are different

If the diff shows added and removed lines, the deprecation messages we expected to see may have changed. Most deprecation warnings contain the file and line of app code that caused it. If the line number changes, the deprecation warning message will also change. In this case, you can follow the same steps as you would if you removed deprecation warnings to update the stale deprecation messages:

DEPRECATION_TRACKER=save bundle exec rspec <spec files mentioned in the error>

This will update the deprecation warning shitlist. Be sure to commit your changes, then push and wait for CI to go ✅.

Where is the shitlist stored?

spec/support/deprecation_warnings.shitlist.json

How does it work?

While running tests, we keep track of what test file is being run and record each deprecation warning as it happens. After the test run is complete, we compare the deprecation messages we saw with the deprecation messages we expected to see and raise an error if the two differ.

See spec/support/deprecation_tracker.rb for implementation details.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!