MAINT: Deterministic SVG and PDF tests #7748

jkseppan · 2017-01-05T18:24:00Z

There has been plenty of work (at least #4434, #5671, #5766, #6597) to make it possible to have deterministic svg and pdf output, or at least closer to deterministic than before. This change adds one related feature: callers can remove the bits of pdf metadata that contain the date and matplotlib version, so that new releases don't necessarily change the result of every test. Furthermore, this change enables all the optional pdf and svg determinism features in tests.

I am developing some improvements to test-result caching on another branch, and in my tests only the following test files still change from run to run after this patch (although in my environment I get KNOWNFAIL=27, SKIP=8 so some more are possible):

clip_path_clipping.svg
hatching.svg
pgf_bbox_inches.pdf
pgf_mixedmode.pdf
pgf_pdflatex.pdf
pgf_rcupdate1.pdf
pgf_rcupdate2.pdf
pgf_xelatex.pdf
bbox_inches_tight_clipping.svg
image_clip.svg
scaled_lines.svg
offsetbox_clipping.svg
skew_axes.pdf
skew_axes.svg

And document the standard keys in PdfPages. Also insert the matplotlib version in the default value of Producer as well as Creator, to retain a debugging clue in case the user overrides one of them.

Remove the metadata entries that depend on the date and the exact version.

codecov-io · 2017-01-05T20:25:37Z

Current coverage is 62.12% (diff: 100%)

Merging #7748 into master will increase coverage by <.01%

@@             master      #7748   diff @@
==========================================
  Files           174        174          
  Lines         56028      56028          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          34805      34806     +1   
+ Misses        21223      21222     -1   
  Partials          0          0

Powered by Codecov. Last update 1fa4dd7...eab297c

tacaswell · 2017-01-05T21:03:37Z

lib/matplotlib/testing/decorators.py

@@ -298,7 +298,12 @@ def compare(self, idx, baseline, extension):
            remove_ticks_and_titles(fig)

        actual_fname = os.path.join(self.result_dir, baseline) + '.' + extension
-        fig.savefig(actual_fname, **self.savefig_kwargs)
+        kwargs = self.savefig_kwargs.copy()
+        if extension == 'pdf':


Doesn't this break the test of SOURCE_DATE_EPOCH ?

No, that test starts a separate subprocess to write the output and doesn't use the image_comparison decorator at all.

tacaswell · 2017-01-05T21:05:37Z

the pgf tests roll their own image comparison code.

attn @astrofrog this may need to be ported to pytest-mpl?

QuLogic · 2017-01-05T22:26:08Z

lib/matplotlib/testing/__init__.py

@@ -136,6 +136,10 @@ def set_font_settings_for_testing():
    rcParams['text.hinting_factor'] = 8


+def set_reproducibility_for_testing():
+    rcParams['svg.hashsalt'] = 'matplotlib'


This will also break, in a way, the determinism test.

Can you explain in more detail? The _test_determinism_save function in test_backend_svg.py file also sets this rc parameter to a constant value.

There is a cache of png files keyed by the MD5 hashes of corresponding svg and pdf files, which helps reduce test suite running times for svg and pdf files that stay exactly the same from one run to the next. This patch enables caching of test results, not only expected results, which is only useful if the tests are mostly deterministic (see matplotlib#7748). It adds reporting of cache misses, which can be helpful in getting tests to stay deterministic, and expiration since the test results are going to change more often than the expected results.

QuLogic

Some minor corrections, but otherwise fine.

QuLogic · 2017-01-20T08:07:12Z

doc/users/whats_new/reproducible_ps_pdf.rst

 The reproducibility of the output from the PS and PDF backends has so
 far been tested using various plot elements but only default values of
 options such as ``{ps,pdf}.fonttype`` that can affect the output at a
 low level, and not with the mathtext or usetex features. When
 matplotlib calls external tools (such as PS distillers or LaTeX) their
 versions need to be kept constant for reproducibility, and they may
 add sources of nondeterminism outside the control of matplotlib.
+
+For SVG output, the ``svg.hashsalt`` rc parameter has been added in an
+earlier release. In can be used to change some random id values in the


QuLogic · 2017-01-20T08:07:36Z

doc/users/whats_new/reproducible_ps_pdf.rst

+For SVG output, the ``svg.hashsalt`` rc parameter has been added in an
+earlier release. In can be used to change some random id values in the
+output to be deterministic, at the cost that including multiple such
+svg files in one document can lead to collisions.


QuLogic · 2017-01-20T08:07:49Z

doc/users/whats_new/reproducible_ps_pdf.rst

+output to be deterministic, at the cost that including multiple such
+svg files in one document can lead to collisions.
+
+These features are now enabled in the tests for the pdf and svg


pdf -> PDF
svg -> SVG

QuLogic · 2017-01-20T08:09:14Z

doc/users/whats_new/reproducible_ps_pdf.rst


 The ``SOURCE_DATE_EPOCH`` environment variable can now be used to set
 the timestamp value in the PS and PDF outputs. See
 https://reproducible-builds.org/specs/source-date-epoch/

+Alternatively, calling ``savefig`` with ``metadata={creationDate=None}``


This dict literal is a bit broken; should be dict(creationDate=None) or {'creationDate': None}.

QuLogic · 2017-01-20T08:11:32Z

lib/matplotlib/backends/backend_pdf.py

@@ -484,11 +483,13 @@ def __init__(self, filename, metadata=None):

        self.infoDict = {
            'Creator': 'matplotlib %s, http://matplotlib.org' % __version__,
-            'Producer': 'matplotlib pdf backend%s' % revision,
+            'Producer': 'matplotlib pdf backend %s' % __version__,


Do we even need the version here? It's in the Creator tag.

I thought including the version in both would help with debugging when the user has overridden just one of these. If you use Matplotlib as a component of some larger application, you might want to override Creator and leave Producer pointing to the PDF backend.

Fair enough.

NelleV · 2017-01-20T19:08:44Z

doc/users/whats_new/reproducible_ps_pdf.rst

 The reproducibility of the output from the PS and PDF backends has so
 far been tested using various plot elements but only default values of
 options such as ``{ps,pdf}.fonttype`` that can affect the output at a
 low level, and not with the mathtext or usetex features. When
 matplotlib calls external tools (such as PS distillers or LaTeX) their
 versions need to be kept constant for reproducibility, and they may
 add sources of nondeterminism outside the control of matplotlib.
+
+For SVG output, the ``svg.hashsalt`` rc parameter has been added in an
+earlier release. It can be used to change some random id values in the


I don't understand the following sentence:
"It can be used to change some random id values in the output to be deterministic, at the cost that including multiple such SVG files in one document can lead to collisions."
Isn't there an grammar problem?

I'll try to rephrase.

NelleV · 2017-01-20T21:03:40Z

Thanks! This looks good to me.
👍

QuLogic · 2017-01-20T22:22:00Z

doc/users/whats_new/reproducible_ps_pdf.rst

-SVG files in one document can lead to collisions.
+earlier release. This parameter changes some random identifiers in the
+SVG file to be deterministic. The downside of this setting is that if
+more than one file is generated using with deterministic identifiers


using with -> using.

There is a cache of png files keyed by the MD5 hashes of corresponding svg and pdf files, which helps reduce test suite running times for svg and pdf files that stay exactly the same from one run to the next. This patch enables caching of test results, not only expected results, which is only useful if the tests are mostly deterministic (see matplotlib#7748). It adds reporting of cache misses, which can be helpful in getting tests to stay deterministic, and expiration since the test results are going to change more often than the expected results.

jkseppan added 4 commits January 5, 2017 19:27

Allow deleting pdf metadata keys

529afdc

And document the standard keys in PdfPages. Also insert the matplotlib version in the default value of Producer as well as Creator, to retain a debugging clue in case the user overrides one of them.

Make pdf test results (more) deterministic

8c7adbb

Remove the metadata entries that depend on the date and the exact version.

Reproducible svg

93b2eea

Document the changes in whats_new

eab297c

jkseppan added the topic: testing label Jan 5, 2017

tacaswell added this to the 2.1 (next point release) milestone Jan 5, 2017

tacaswell reviewed Jan 5, 2017

View reviewed changes

QuLogic reviewed Jan 5, 2017

View reviewed changes

jkseppan mentioned this pull request Jan 7, 2017

[MRG+2] MAINT: Refactor the converted-image cache #7764

Closed

QuLogic reviewed Jan 20, 2017

View reviewed changes

Fix documentation mistakes

ec941b4

QuLogic approved these changes Jan 20, 2017

View reviewed changes

QuLogic changed the title ~~MAINT: Deterministic SVG and PDF tests~~ [MRG+1] MAINT: Deterministic SVG and PDF tests Jan 20, 2017

NelleV reviewed Jan 20, 2017

View reviewed changes

Try to improve wording

749e81f

NelleV changed the title ~~[MRG+1] MAINT: Deterministic SVG and PDF tests~~ [MRG+2] MAINT: Deterministic SVG and PDF tests Jan 20, 2017

QuLogic reviewed Jan 20, 2017

View reviewed changes

Another wording fix

6736f7d

QuLogic changed the title ~~[MRG+2] MAINT: Deterministic SVG and PDF tests~~ MAINT: Deterministic SVG and PDF tests Jan 21, 2017

QuLogic merged commit b6f13f1 into matplotlib:master Jan 21, 2017

jkseppan deleted the deterministic-tests branch January 22, 2017 07:37

astrofrog mentioned this pull request Jan 26, 2017

Check whether any updates are needed following Matplotlib changes matplotlib/pytest-mpl#41

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MAINT: Deterministic SVG and PDF tests #7748

MAINT: Deterministic SVG and PDF tests #7748

MAINT: Deterministic SVG and PDF tests #7748

MAINT: Deterministic SVG and PDF tests #7748

Conversation

Current coverage is 62.12% (diff: 100%)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment