8000 Codecov failing on 3.11 · Issue #463 · python/bedevere · GitHub
[go: up one dir, main page]
Skip to content

Codecov failing on 3.11 #463

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ezio-melotti opened this issue May 30, 2022 · 9 comments
Closed

Codecov failing on 3.11 #463

ezio-melotti opened this issue May 30, 2022 · 9 comments
Assignees
@ezio-melotti
Labels
bug

Comments

@ezio-melotti
Copy link
Member

In https://github.com/python/bedevere/runs/6651826354?check_suite_focus=true the test fail because codecov doesn't reach 100%:

See full pytest output 
Run pytest --cov --cov-report=xml
  pytest --cov --cov-report=xml
  shell: /usr/bin/bash -e {0}
  env:
    FORCE_COLOR: 1
    pythonLocation: /opt/hostedtoolcache/Python/3.11.0-beta.1/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.0-beta.1/x64/lib
============================= test session starts ==============================
platform linux -- Python 3.11.0b1, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/runner/work/bedevere/bedevere, configfile: pytest.ini, testpaths: tests
plugins: cov-3.0.0, asyncio-0.18.3, aiohttp-1.0.4
asyncio: mode=Mode.AUTO
collected 172 items

tests/test___main__.py ...                                               [  1%]
tests/test_backport.py ..........................                        [ 16%]
tests/test_close_pr.py .......                                           [ 20%]
tests/test_filepaths.py ............                                     [ 27%]
tests/test_gh_issue.py ....................................              [ 48%]
tests/test_news.py .................................................     [ 77%]
tests/test_prtype.py .........                                           [ 82%]
tests/test_stage.py ....................                                 [ 94%]
tests/test_util.py ..........                                            [100%]

----------- coverage: platform linux, python 3.11.0-beta-1 -----------
Coverage XML written to file coverage.xml

FAIL Required test coverage of 100.0% not reached. Total coverage: 99.02%

============================= 172 passed in 6.27s ==============================
Error: Process completed with exit code 1.

This however only seems to affect 3.11.0-apha, and not the other Python versions. This should be fixed however -- apparently because of #462 -- the uploaded report doesn't include the failure:

See full codecov output 
Run codecov/codecov-action@v2
  with:
    file: ./coverage.xml
  env:
    FORCE_COLOR: 1
    pythonLocation: /opt/hostedtoolcache/Python/3.11.0-beta.1/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.0-beta.1/x64/lib
==> linux OS detected
https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
==> SHASUM file signed by key id 806bb28aed779869
==> Uploader SHASUM verified (648b599397548e4bb92429eec6391374c2cbb0edb835e3b3f03d4281c011f401  codecov)
==> Running version latest
==> Running version v0.2.3
/home/runner/work/_actions/codecov/codecov-action/v2/dist/codecov -n  -Q github-action-2.1.0 -f ./coverage.xml -C 571de6f456fc8379015f221d8b2d796f1bc88818
[2022-05-30T09:11:54.662Z] ['info'] 
     _____          _
    / ____|        | |
   | |     ___   __| | ___  ___ _____   __
   | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
   | |___| (_) | (_| |  __/ (_| (_) \ V /
    \_____\___/ \__,_|\___|\___\___/ \_/

  Codecov report uploader 0.2.3
[2022-05-30T09:11:54.669Z] ['info'] => Project root located at: /home/runner/work/bedevere/bedevere
[2022-05-30T09:11:54.672Z] ['info'] -> No token specified or token is empty
[2022-05-30T09:11:54.781Z] ['info'] Searching for coverage files...
[2022-05-30T09:11:54.839Z] ['info'] => Found 1 possible coverage files:
  ./coverage.xml
[2022-05-30T09:11:54.839Z] ['info'] Processing ./coverage.xml...
[2022-05-30T09:11:54.845Z] ['info'] Detected GitHub Actions as the CI provider.
[2022-05-30T09:11:54.847Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=github-action-2.1.0-uploader-0.2.3&token=*******&branch=dependabot%2Fpip%2Fcachetools-5.2.0&build=2407989696&build_url=https%3A%2F%2Fgithub.com%2Fpython%2Fbedevere%2Factions%2Fruns%2F2407989696&commit=571de6f456fc8379015f221d8b2d796f1bc88818&job=Tests&pr=460&service=github-actions&slug=python%2Fbedevere&name=&tag=&flags=&parent=
[2022-05-30T09:11:55.423Z] ['info'] https://codecov.io/github/python/bedevere/commit/571de6f456fc8379015f221d8b2d796f1bc88818
https://storage.googleapis.com/codecov/v4/raw/2022-05-30/D77050ECB3DC286B3CD9265D83DDA42F/571de6f456fc8379015f221d8b2d796f1bc88818/e482bb87-1a14-49a8-998d-c64096a45b86.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EKKHVKCKHW7KBCGM7IHR55T63V2PAVJWLVFNITJHDU5G6R5IRN3LMWJA%2F20220530%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20220530T091155Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=ba144246e7b0cc69b67e35caad1afce4ce9fbdc60902a813d9854590416d23e9
[2022-05-30T09:11:55.423Z] ['info'] Uploading...
[2022-05-30T09:11:55.624Z] ['info'] {"status":"success","resultURL":"https://codecov.io/github/python/bedevere/commit/571de6f456fc8379015f221d8b2d796f1bc88818"}
@ezio-melotti ezio-melotti mentioned this issue May 30, 2022
Closed
@hugovk
Copy link
Member
hugovk commented May 30, 2022 •
edited
Loading

Technically Codecov isn't failing, the pytest/coverage check is failing because there's less than 100% coverage for each individual build (fail_under = 100 in setup.cfg).

When all the coverage uploaded to Codecov and combined, the aggregate coverage is 100%.

If we add --cov-report=term to the pytest run (hugovk@186818e), we can 100% coverage for 3.10 but some missing coverage for 3.11:

Name                      Stmts   Miss Branch BrPart  Cover
-----------------------------------------------------------
bedevere/__init__.py          0      0      0      0   100%
bedevere/__main__.py         37      0      4      1    98%
bedevere/backport.py         63      0     36      0   100%
bedevere/close_pr.py         22      0     14      0   100%
bedevere/filepaths.py        15      0     12      0   100%
bedevere/gh_issue.py         72      0     34      1    99%
bedevere/news.py             51      0     28      0   100%
bedevere/prtype.py           33      0     19      0   100%
bedevere/stage.py           117      0     60      9    95%
bedevere/util.py             67      0     24      1    99%
tests/__init__.py             0      0      0      0   100%
tests/test___main__.py       25      0      0      0   100%
tests/test_backport.py      146      0     22      0   100%
tests/test_close_pr.py       95      0     16      0   100%
tests/test_filepaths.py     106      0      8      0   100%
tests/test_gh_issue.py      249      0     96      9    97%
tests/test_news.py          155      0     28      0   100%
tests/test_prtype.py        103      0      2      0   100%
tests/test_stage.py         291      0     10      0   100%
tests/test_util.py           73      0      6      0   100%
-----------------------------------------------------------
TOTAL                      1720      0    419     21    99%
Coverage XML written to file coverage.xml

FAIL Required test coverage of 100.0% not reached. Total coverage: 99.02%

(Aside: PR #461 would have prevented me from running the above tests without (a) creating a spammy PR, (b) temporarily editing the workflow to allow the feature branch to run or (c) making the changes in main.)

htmlcov.zip is the output of running locally with --cov-report html on Python 3.11. The missing lines are things like:

image

@ezio-melotti
Copy link
Member Author

Is there a reason why 3.10 and 3.11 report different coverage for the same code?

@hugovk
Copy link
Member
hugovk commented May 30, 2022

That's a very good question... I've not seen these partial coverage results in Coverage.py before.

It could be a new feature only in 3.11, or (more likely?) a bug in Coverage.py for 3.11, or possibly in Python 3.11 itself?

@ezio-melotti
Copy link
Member Author

It could be a new feature only in 3.11, or (more likely?) a bug in Coverage.py for 3.11, or possibly in Python 3.11 itself?

This is what I'm wondering 🤔

All the issues seem to be with async with, async for, and if await lines.

@AlexWaygood
Copy link
Member
AlexWaygood commented May 30, 2022 •
edited
Loading

It could be a new feature only in 3.11, or (more likely?) a bug in Coverage.py for 3.11, or possibly in Python 3.11 itself?

This is what I'm wondering 🤔

All the issues seem to be with async with, async for, and if await lines.

Maybe this bug is the cause? (I think it will be fixed in beta 2.)

@nightlark nightlark mentioned this issue May 30, 2022
Closed
@hugovk
Copy link
Member
hugovk commented May 30, 2022

Good find!

Using https://github.com/deadsnakes/action to test a nightly build of 3.11: hugovk@c045c8f

And 3.11 passes with 100% coverage :)

https://github.com/hugovk/bedevere/runs/6659010706?check_suite_focus=true#step:7:48

3.11 beta 2 is due today, so all should be good tomorrow or the day after when it reaches GitHub Actions.

@ezio-melotti
Copy link
Member Author

I'll leave this open for now and check back tomorrow to see if the issue is solved.

@ezio-melotti ezio-melotti self-assigned this May 30, 2022
@ezio-melotti
Copy link
Member Author

It took yet another beta, but this seems now fixed.

@AlexWaygood
Copy link
Member

It took yet another beta, but this seems now fixed.

I remember a time when you had to wait a month before your next beta. Now? A dozen a day. Smdh.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ezio-melotti ezio-melotti

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@hugovk @ezio-melotti @AlexWaygood
0