binary.sh.tpl doesn't check that arguments are passed when used as a test #233

JasonLunn · 2025-05-09T20:07:20Z

It is possible to end up in a situation where an rb_test rule produces a shell script from binary.sh.tpl that never receives arguments. In this case, executing the rule results in a false negative - in other words, bazel test will pass even though the Ruby test has never been executed because the only thing that runs is bundle exec ruby which returns successfully.

I'm still trying to determine what I've done in my local setup to misconfigure the rb_test in this way, but having defense in depth could help others avoid losing the same time that I have tracking this down.

Something like:

if [[ -n "${TEST_BINARY:-}" && $# -le 0 ]]; then
  echo "Error! No arguments provided to test binary"
  exit -1
fi

This approach would rely on TEST_BINARY (set by test-setup.sh) to be present so that the check for missing arguments doesn't apply to vanilla rb_binary rules.

WDYT, @p0deje ?

The text was updated successfully, but these errors were encountered:

p0deje · 2025-05-09T21:05:06Z

@JasonLunn This makes sense, would you mind opening a PR for this? We'll also need to do the same in binary.cmd.tpl.

JasonLunn · 2025-05-09T22:31:10Z

Will do.

On the subject of how I got here, it appears that the reason I don't get any args to bundle exec ruby is that I'm not specifying any in my rb_test rule explicitly.

For argument's sake, let's say my test rules reads like this:

rb_test(
    name = "basic",
    srcs = ["basic.rb"],
    deps = [
        ":test_lib",
        "@my_bundle",
    ],
)

My casual reading of https://github.com/bazel-contrib/rules_ruby/blob/main/docs/rules.md#rb_test had left me with the impression that args would be args to the test itself, which I omitted because my basic test doesn't take any arguments, and that the test would execute the equivalent of bundle exec ruby basic.rb. It seems like I'm mistaken, and that when there's no main specified, the script isn't the first argument to Ruby automatically. Is this working as intended?

Maybe the examples could be extended to demonstrate how minitest users (and anyone that is used to executing their tests without an explicit Ruby binary) should construct their test rules.

p0deje · 2025-05-12T14:42:18Z

My casual reading of https://github.com/bazel-contrib/rules_ruby/blob/main/docs/rules.md#rb_test had left me with the impression that args would be args to the test itself, which I omitted because my basic test doesn't take any arguments, and that the test would execute the equivalent of bundle exec ruby basic.rb. It seems like I'm mistaken, and that when there's no main specified, the script isn't the first argument to Ruby automatically. Is this working as intended?

I haven't thought about this scenario and now I think the code should be changed so that if both main and args are omitted, it makes sense to populate args with srcs rather than silently succeed by running ruby.

Maybe the examples could be extended to demonstrate how minitest users (and anyone that is used to executing their tests without an explicit Ruby binary) should construct their test rules.

Maybe @noahkawasakigoogle was running into a similar issue in #223. His proposal for rb_minitest would work better for your case as well.

Additional details discussed in bazel-contrib/rules_ruby#233 and bazel-contrib/rules_ruby#223 Restoring tests revealed latent bug in JRuby's `RubyMessage#convert` that was exposed by the introduction `test_to_hash` while tests were not running; fix included. Closes #21733 COPYBARA_INTEGRATE_REVIEW=#21733 from protocolbuffers:fix_rb_test 3f8418f PiperOrigin-RevId: 758470283

) Additional details discussed in bazel-contrib/rules_ruby#233 and bazel-contrib/rules_ruby#223 Restoring tests revealed latent bug in JRuby's `RubyMessage#convert` that was exposed by the introduction `test_to_hash` while tests were not running; fix included. Closes protocolbuffers#21733 COPYBARA_INTEGRATE_REVIEW=protocolbuffers#21733 from protocolbuffers:fix_rb_test 3f8418f PiperOrigin-RevId: 758470283

JasonLunn mentioned this issue May 13, 2025

Fix silent failure of rb_test rules to run test protocolbuffers/protobuf#21733

Closed

JasonLunn mentioned this issue May 14, 2025

Fix silent failure of rb_test rules to run test (#21733) protocolbuffers/protobuf#21740

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

binary.sh.tpl doesn't check that arguments are passed when used as a test #233

binary.sh.tpl doesn't check that arguments are passed when used as a test #233

binary.sh.tpl doesn't check that arguments are passed when used as a test #233

binary.sh.tpl doesn't check that arguments are passed when used as a test #233

Comments