Description
Of course I can run coverage for tests locally, but I think it should be accessible on coveralls. You cannot blindly rely on a test report until you are sure that they are actually working, otherwise it could give you a false sense of security.
I have faced a problem where I cannot be sure that pytest runs all the tests. Moreover I have got a test failures with pytest on tests that nose does not perform (and I do not why then they are not deleted). Here is the list:
- test_sankey not deprecated, runs without issues
- test_skew not deprecated, has outdated baseline images or tolerance problems (
..FF.F: RMS 0.014, 0.264, 0.065) - test_ttconv not deprecated, runs without issues
- test_usetex not deprecated, has outdated baseline images or tolerance problems (
.F: RMS 0.261)
What benefits can we have by including test files coverage in the report? I collected dark spots in the test suite:
Important (test not running, uncovered cases):
- test_animation:test_no_length_frames
- test_axes:test_pcolorargs
- test_backend_pdf:test_use14corefonts does not run on travis (
TRAVISenv variable was removed?) - test_contour:test_contour_manual_labels
- test_image:test_minimized_rasterized
- test_mlab:Test_detrend.test_detrend_mean_2D_none
- test_mlab:Test_detrend.test_detrend_mean_2D_none_T
- test_mlab:Test_detrend.test_detrend_mean_2D_axis0
- test_mlab:Test_detrend.test_detrend_mean_2D_axis1
- test_mlab:Test_detrend.test_detrend_mean_2D_axism1
- test_mlab:Test_detrend.test_detrend_str_linear_2d_slope_off_axis0
- test_mlab:Test_detrend.test_detrend_detrend_linear_1d_slope_off_axis1
- test_mlab:test_griddata_nn
- test_subplots:test_shared
- test_patheffects:test_PathEffect_points_to_pixels
Less important (partial coverage):
- test_backend_svg _test_determinism
- test_cbook:test_results_whiskers_range
- test_cbook:test_results_whiskers_percentiles
- test_dates:test_date2num_dst
test_delaunay- test_mlab:Test_window
- test_mlab:Test_spectral_nosig_real_onesided
- test_units:test_numpy_facade
Not important (requires tests for the tests, and other cases):
- test_collections check_allprop unused function?
- test_collections check_segments
- test_transforms:test_non_affine_caching transform_path_non_affine&transform_non_affine
- test_triplot_return
Someone can say that we need tests for the tests to have full coverage -- I would say no, some special places can be marked with #pragma: no cover, but other can be fixed to have full coverage.
Currently this can be done with nose-cov plugin and after migrating to pytest we will have exactly same coverage reports with pytest-cov.