run_binary will put files in the sandbox but then can't find them for outs.

[dougthor42]

Summary

run_binary generates files in the sandbox directory but cannot find them when trying to set the outs values.

So it seems like we can't use run_binary as a source of files/data for other targets. Is this expected?

Steps to Reproduce

Create these files:

# MODULE.bazel
module(
    name = "example",
    version = "0.0.0",
    compatibility_level = 1,
)

bazel_dep(name = "rules_python", version = "1.0.0")
bazel_dep(name = "bazel_skylib", version = "1.7.1")

python = use_extension("@rules_python//python/extensions:python.bzl", "python")

python.toolchain(
    is_default = True,
    python_version = "3.12.2",
)
use_repo(python, "python_3_12_2")

# my_script.py
import sys
import pathlib


def main():
    file = pathlib.Path(sys.argv[1])
    file.write_text("This is my generated data.")
    print(f"    Wrote to file at {file.resolve(strict=True)}", file=sys.stderr)
    print("    Here's proof. The written text was:", file=sys.stderr)
    print(f"    {file.read_text()}", file=sys.stderr)


if __name__ == "__main__":
    main()

# BUILD.bazel
load("@bazel_skylib//rules:run_binary.bzl", "run_binary")
load("@rules_python//python:defs.bzl", "py_binary")

py_binary(
    name = "my_script",
    srcs = ["my_script.py"],
)

run_binary(
    name = "run_my_script",
    tool = ":my_script",
    args = ["some_file.txt"],
    outs = ["some_file.txt"],
)

And, if you're using Bazelisk, a .bazelversion file with:

7.4.1

Then call:

bazel build //:run_my_script

Expected Result

The run_binary target should be able to use the file generated by py_binary.

Actual Result

Bazel errors out, saying that the output file was not created.

$ bazel build //:run_my_script 
INFO: Analyzed target //:run_my_script (1 packages loaded, 3 targets configured).
INFO: From RunBinary some_file.txt:
    Wrote to file at /usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/971d8fef15cecd42cbe8747d79f170f6/sandbox/linux-sandbox/3/execroot/_main/some_file.txt
    Here's proof. The written text was:
    This is my generated data.
ERROR: /usr/local/google/home/dthor/dev/skylib-example/BUILD.bazel:9:11: output 'some_file.txt' was not created
ERROR: /usr/local/google/home/dthor/dev/skylib-example/BUILD.bazel:9:11: RunBinary some_file.txt failed: not all outputs were created or valid
Target //:run_my_script failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.454s, Critical Path: 0.26s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
ERROR: Build did NOT complete successfully

[dougthor42]

I should also note that the same thing happens if using a sh_binary instead of py_binary.

# my_shell_script.sh
echo "    Writing to $1" >&2
echo "Hello world" > "$1"

sh_binary(
    name = "my_shell_script",
    srcs = ["my_shell_script.sh"],
)

run_binary(
    name = "run_my_script",
    tool = ":my_shell_script",
    args = ["some_file.txt"],
    outs = ["some_file.txt"],
)

$ bazel build //:run_my_script
INFO: Analyzed target //:run_my_script (0 packages loaded, 0 targets configured).
INFO: From RunBinary some_file.txt:
    Writing to some_file.txt
ERROR: /usr/local/google/home/dthor/dev/skylib-example/BUILD.bazel:14:11: output 'some_file.txt' was not created
ERROR: /usr/local/google/home/dthor/dev/skylib-example/BUILD.bazel:14:11: RunBinary some_file.txt failed: not all outputs were created or valid
Target //:run_my_script failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.126s, Critical Path: 0.02s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
ERROR: Build did NOT complete successfully

[fmeum]

The output path of the file won't be just some_file.txt, it will be a more complicated path starting with bazel-out.

Could you try setting args = ["$(execpath some_file.txt)"]? That should expand to the correct path to write the output to.

[dougthor42]

That's related to why I first started down this path. It seemed like $(execpath :some_target) wasn't being expanded, so I tried simplifying things to the bare minimum and found the issue in the original post.

OK so here's where it gets strange. Doing args = ["$(execpath some_file.txt)"] works for the example in the OP, thanks!

But doing the same thing in a different project shows that execpath isn't expanded. In the below, notice that the created file is called $(execpath some_file.txt) rather than .../bazel-out/k8-fastbuild/bin/some_file.txt:

$ tail -n 11 src/pyle_xc/layout/scripts/BUILD.bazel 
py_binary(
    name = "doug_script",
    srcs = ["doug_script.py"],
)

run_binary(
    name = "run_my_script",
    tool = ":doug_script",
    args = ["$(execpath some_file.txt)"],
    outs = ["some_file.txt"],
)
$ cat src/pyle_xc/layout/scripts/doug_script.py 
import sys
import pathlib


def main():
    file = pathlib.Path(sys.argv[1])
    file.write_text("This is my generated data.")
    print(f"  Wrote to file at {file.resolve(strict=True)}", file=sys.stderr)
    print("  Here's proof. The written text was:", file=sys.stderr)
    print(f"  {file.read_text()}", file=sys.stderr)


if __name__ == "__main__":
    main()
$ bazel clean --async; bazel --ignore_all_rc_files build //src/pyle_xc/layout/scripts:run_my_script --sandbox_debug
...
INFO: From RunBinary src/pyle_xc/layout/scripts/some_file.txt:
  Wrote to file at /usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dfb34a4360d7e3628ef90718c12cf15a/sandbox/linux-sandbox/15/execroot/_main/$(execpath some_file.txt)
  Here's proof. The written text was:
  This is my generated data.
ERROR: /usr/local/google/home/dthor/dev/pyle3-xc/src/pyle_xc/layout/scripts/BUILD.bazel:409:11: output 'src/pyle_xc/layout/scripts/some_file.txt' was not created
ERROR: /usr/local/google/home/dthor/dev/pyle3-xc/src/pyle_xc/layout/scripts/BUILD.bazel:409:11: RunBinary src/pyle_xc/layout/scripts/some_file.txt failed: not all outputs were created or valid
Target //src/pyle_xc/layout/scripts:run_my_script failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1.608s, Critical Path: 0.60s
INFO: 6 processes: 5 internal, 1 linux-sandbox.
ERROR: Build did NOT complete successfully
$ tree -L 2 ~/.cache/bazel/_bazel_dthor/dfb34a4360d7e3628ef90718c12cf15a/sandbox/linux-sandbox/15/execroot/_main/
/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dfb34a4360d7e3628ef90718c12cf15a/sandbox/linux-sandbox/15/execroot/_main/
├── $(execpath some_file.txt)
├── bazel-out
│   ├── k8-fastbuild
│   └── k8-opt-exec-ST-d57f47055a04
└── src
    └── pyle_xc

You'll also notice that I used --ignore_all_rc_files and cleaned bazel before running, so my ~/dev/pyle3-xc workspace should be the same as the clean one I generated for the original post.

Thanks for letting me rubber duck.