correct implementation of soft assertions #53
Description
Hi!
This is not a bug report, but more like a discussion kick-starter regarding soft assertions. And if we happen to agree on a different implementation, I'll be more than happy to create a PR.
What I suggest is soft assertions to be implemented as in other languages libraries. E.g soft assertions in AssertJ.
Soft assertions usually have a special value in higher levels of testing than unit, e.g. integration or system tests, when the result feedback is not as fast as with unit test.
So basically a test is more "expensive" to run in terms of resources like CPU, memory, and specially time. And we want to take the most value out of each execution.
Let's assume that I have a test that:
- Logs in
- Creates a new user in the system with default settings
- And verifies that the user has a
- default locale = X
- default timezone = Y
- default privileges = Z
- etc, etc
If any of these are missing or wrong, I want the test to fail. However, with regular assertions if the locale is missing the test will fail as expected but I won't have any information whether the system meets the other requirements.
As you know, that's when soft assertions come handy. The problem I see though, is that in your implementation, you silently pass, I mean.. you print a warning in stdout, but the test will pass. And that's a wrong approach IMO, as it requires human intervention (someone reading the screen), so those assertions won't have any effect if tests are run, for instance, in jenkins as part of CI/CD pipeline.
What I suggest is what AssertJ does. You run assertions in a group, so even if the "locale" assertion fails, you still run the "timezone" and the "privileges" assertions. After all the assertions have been executed an AssertionError is raised if at least one assertion in the group failed. The error will contain the details of all those assertions in the group that failed.
Does all this make sense to you? WDYT?
Regards!