Conditional build step | Zig Programming Language | Page 1

wispy kite Feb 25, 2024, 2:39 PM

#

Hi! I am building zig for riscv32im target and I came across an issue with build script. I need to copy .data section from the executable elf file as a raw binary data to another file. But this should happen only if this section exists. AFAIK there is no option in build API to skip the build step if it fails, so the whole build fails for me when there is no .data section in elf.

How can I do that zig way?

My current build script:

const std = @import("std");
const CrossTarget = @import("std").zig.CrossTarget;
const Target = @import("std").Target;
const Feature = @import("std").Target.Cpu.Feature;

pub fn build(std_build: *std.Build) void {
    const features = Target.riscv.Feature;
    var disabled_features = Feature.Set.empty;
    var enabled_features = Feature.Set.empty;

    disabled_features.addFeature(@intFromEnum(features.a));
    disabled_features.addFeature(@intFromEnum(features.c));
    disabled_features.addFeature(@intFromEnum(features.d));
    disabled_features.addFeature(@intFromEnum(features.e));
    disabled_features.addFeature(@intFromEnum(features.f));

    enabled_features.addFeature(@intFromEnum(features.m));

    const query = CrossTarget{
        .cpu_arch = .riscv32,
        .os_tag = .freestanding,
        .abi = .none,
        .cpu_model = .{
            .explicit = &std.Target.riscv.cpu.generic_rv32
        },
        .cpu_features_sub = disabled_features,
        .cpu_features_add = enabled_features
    };

    const target = std_build.resolveTargetQuery(query);

    const elf = std_build.addExecutable(.{
        .name = "main.elf",
        .root_source_file = .{ .path = "src/main.zig" },
        .target = target,
        .optimize = .ReleaseFast
    });

    elf.setLinkerScriptPath(.{ .path = "src/linker.ld" });

    std_build.installArtifact(elf);

    const code_bin = std_build.addObjCopy(
        elf.getEmittedBin(),
        std.Build.Step.ObjCopy.Options {
            .only_section = ".text",
            .format = .bin
        }
    );

    const copy_code_bin = std_build.addInstallBinFile(
        code_bin.getOutput(),
        "code.bin"
    );

    std_build.default_step.dependOn(&copy_code_bin.step);

    const ram_bin = std_build.addObjCopy(
        elf.getEmittedBin(),
        std.Build.Step.ObjCopy.Options {
            .only_section = ".data",
            .format = .bin
        }
    );

    const copy_ram_bin = std_build.addInstallBinFile(
        ram_bin.getOutput(),
        "ram.bin"
    );

    std_build.default_step.dependOn(&copy_ram_bin.step);
}

hollow peak Feb 26, 2024, 4:32 AM

#

so you want it to silently emit an empty file if the section doesn't exist?

wispy kite Feb 26, 2024, 10:25 AM

#

hollow peak so you want it to silently emit an empty file if the section doesn't exist?

Basically yes

hollow peak Feb 26, 2024, 12:21 PM

#

wispy kite Basically yes

what you can do is either try to get a .allow_nonexistent: bool into std.Build.Step.ObjCopy.Options merged into the stdlib

#

or you can make your own build step that catches the error

hollow peak Feb 26, 2024, 12:23 PM

#

hollow peak what you can do is either try to get a `.allow_nonexistent: bool` into `std.Buil...

oh right nvm forgot that the objcopy step probably calls real objcopy nowadays

wispy kite Feb 26, 2024, 8:54 PM

#

hollow peak what you can do is either try to get a `.allow_nonexistent: bool` into `std.Buil...

I thought maybe try emitting executable elf, and then try to find section with objdump, and objcopy only if it exists.

But now that I think about it writing my own build step sounds better

wispy kite Mar 5, 2024, 4:33 PM

#

I am trying to write my own objcopy build step in zig now. And after hours of research I am at the point of just giving up and writing a separate shell script to objcopy .data section if it exists. Would really appreciate some guidance here.

I cannot understand what I should pass to b.default_step.dependOn function and how to create my own build step.

cc @hollow peak

hollow peak Mar 5, 2024, 4:34 PM

#

wispy kite I am trying to write my own objcopy build step in zig now. And after hours of re...

oh

#

I was just thinking to call the objcopy step code

#

and just do

original_code(args) catch {
  create_file(args.path);
  return;
}

wispy kite Mar 5, 2024, 4:36 PM

#

but the problem is objcopy is not being executed during build.zig. it is just a constructor for build steps and all the errors are handled internally, so I cannot catch an error in build.zig. at least this is my understanding

hollow peak Mar 5, 2024, 4:37 PM

#

what do you mean

#

you can just have your makeFn call another steps makeFn?

wispy kite Mar 5, 2024, 4:38 PM

#

what is makeFn?

hollow peak Mar 5, 2024, 4:38 PM

#

https://ziglang.org/documentation/master/std/src/std/Build/Step.zig.html#L4

#

that's the function that's called to execute the step

#

const DefaultEmptyFileStep = struct {
  s: std.Build.Step,
  orig: *std.Build.Step,

  pub fn makeFn(step: *std.Build.Step) anyerror!void {
    const self = @fieldParentPtr(DefaultEmptyFileStep, "s", step);
    // ...
  }

  pub fn init(step: *std.Build.Step) @This() {
    var result = @This{};
    result.s = std.Build.Step.init(.{
      .id = .custom,
      .name = "idk",
      .owner = b,
      .makeFn = makeFn,
    });
    result.dependencies = step.dependencies; // Really fucking ugly to double-reference it here (possibly causing a double free), please dupe it instead
    return result;
  }
}

#

something like this

wispy kite Mar 5, 2024, 4:54 PM

#

hm, looks promising, but there is also this error:
use of undeclared identifier 'DefaultEmptyFileStep'

hollow peak Mar 5, 2024, 4:55 PM

#

weird

#

did you change its name

#

try with @This() instead

wispy kite Mar 5, 2024, 4:56 PM

#

@This{} did not compile, I changed it to @This()

hollow peak Mar 5, 2024, 4:57 PM

#

oh right

#

it should have said

@This(){
  .orig = step,
  .s = undefined,
}

#

I forgot to finish that part for some reason

wispy kite Mar 5, 2024, 4:59 PM

#

still the same error, btw do you have zed editor? it has collab feature I wanted to use and this seems like an opportunity lmao

hollow peak Mar 5, 2024, 4:59 PM

#

I don't

wispy kite Mar 5, 2024, 4:59 PM

#

ah, nvm then

hollow peak Mar 5, 2024, 4:59 PM

#

I can try to get it to compile

#

one sec

#

I assumed you would figure it out lol

wispy kite Mar 5, 2024, 5:00 PM

#

I will 100%

#

just takes whole lotta time

hollow peak Mar 5, 2024, 5:08 PM

#

I will try this tomorrow if you havn't figured it out by then

wispy kite Mar 5, 2024, 5:08 PM

#

sure, thanks a lot still!

hollow peak Mar 5, 2024, 5:08 PM

#

that sounds fair, right?

wispy kite Mar 6, 2024, 8:15 PM

#

hollow peak I will try this tomorrow if you havn't figured it out by then

sorry for the ping, but I haven't figured out yet

hollow peak Mar 6, 2024, 8:16 PM

#

dang

#

okay

#

what have you got and what's the error

wispy kite Mar 6, 2024, 8:19 PM

#

hollow peak what have you got and what's the error

I tried for about an hour yesterday, read sources and stuff, but your construction is still cryptic to me. I made a temporary solution using bash script and zig objdump -j .data for now. But I don't like it and want a rewrite. Let me dig up the code and run it so I can see the error

wispy kite Mar 6, 2024, 8:34 PM

#

const std = @import("std");
const CrossTarget = @import("std").zig.CrossTarget;
const Target = @import("std").Target;
const Feature = @import("std").Target.Cpu.Feature;
const AllocPrintError = @import("std").fmt.AllocPrintError;
const Allocator = std.mem.Allocator;

const CustomObjCopy = @import("CustomObjCopy.zig");

pub fn build(std_build: *std.Build) void {
    const features = Target.riscv.Feature;
    var disabled_features = Feature.Set.empty;
    var enabled_features = Feature.Set.empty;

    disabled_features.addFeature(@intFromEnum(features.a));
    disabled_features.addFeature(@intFromEnum(features.c));
    disabled_features.addFeature(@intFromEnum(features.d));
    disabled_features.addFeature(@intFromEnum(features.e));
    disabled_features.addFeature(@intFromEnum(features.f));

    enabled_features.addFeature(@intFromEnum(features.m));

    const query = CrossTarget{
        .cpu_arch = .riscv32,
        .os_tag = .freestanding,
        .abi = .none,
        .cpu_model = .{
            .explicit = &std.Target.riscv.cpu.generic_rv32
        },
        .cpu_features_sub = disabled_features,
        .cpu_features_add = enabled_features
    };

    var arena = std.heap.ArenaAllocator.init(std.heap.page_allocator);
    defer arena.deinit();
    const allocator = arena.allocator();
    const filenames = [_][]const u8{ "asm", "sort" };

    for (filenames) |filename| {
        const target = std_build.resolveTargetQuery(query);

        const source_path = format(allocator, "src/{s}.zig", .{filename});
        const elf_name = format(allocator, "{s}.elf", .{filename});

        const elf = std_build.addExecutable(.{
            .name = elf_name,
            .root_source_file = .{ .path = source_path },
            .target = target,
            .strip = true,
            .optimize = .ReleaseFast
        });

        elf.setLinkerScriptPath(.{ .path = "../linker.ld" });

        std_build.installArtifact(elf);

        const rom_bin = std_build.addObjCopy(
            elf.getEmittedBin(),
            std.Build.Step.ObjCopy.Options {
                .only_sections = &[_][]const u8{ ".text" },
                .format = .bin,
            }
        );

        const rom_bin_path = format(allocator, "{s}.rom.bin", .{filename});

        const copy_rom_bin = std_build.addInstallBinFile(
            rom_bin.getOutput(),
            rom_bin_path
        );

        std_build.default_step.dependOn(&copy_rom_bin.step);

        const DefaultEmptyFileStep = struct {
          s: std.Build.Step,
          orig: *std.Build.Step,

          pub fn makeFn(step: *std.Build.Step) anyerror!void {
            const self = @fieldParentPtr(DefaultEmptyFileStep, "s", step);
              // ...
          }

          pub fn init(step: *std.Build.Step) @This() {
            var result = @This(){
                .orig = step,
                .s = undefined,
            };
            result.s = std.Build.Step.init(.{
              .id = .custom,
              .name = "objcopy",
              .owner = std_build,
              .makeFn = makeFn,
            });
            result.dependencies = step.dependencies; // Really fucking ugly to double-reference it here (possibly causing a double free), please dupe it instead
            result.orig = step;
            return result;
          }
        };

        
    }
}

fn format(allocator: Allocator, comptime fmt: []const u8, args: anytype) []u8 {
    const source_path = std.fmt.allocPrint(allocator, fmt, args) catch |err| {
        std.debug.panic("Alloc print error {}", .{err});
    };

    return source_path;
}

this code gives an error: use of undeclared identifier 'DefaultEmptyFileStep'
I also don't understand how I am supposed to use the DefaultEmptyFileStep to include it in the build. Do I pass it to std_build.default_step.dependOn function?

#

sorry for the mess with indents, zig plugin for intellij is apparently bad

hollow peak Mar 6, 2024, 8:44 PM

#

I got it working before I saw your message

#

📎 build.zig

#

oh right, you probably want to do it slightly differently

#

so that it makes an output when it succeeds

#

📎 build.zig

#

I suppose you don't need the original step either since the generated file has it

#

even more minimalist

#

📎 build.zig

#

is this working for you

wispy kite Mar 6, 2024, 9:15 PM

#

the same error as before, and also this one: no field named 'only_section' in struct 'Build.Step.ObjCopy.Options'
we have different zig versions because ObjCopy.Options was changed recently
my zig version is 0.12.0-dev.3152+90c1a2c41

hollow peak Mar 6, 2024, 9:20 PM

#

yep

#

you gotta use only_sections instead

wispy kite Mar 6, 2024, 9:47 PM

#

waht about the first error?

hollow peak Mar 6, 2024, 9:49 PM

#

as I said, it's called only_sections

wispy kite Mar 6, 2024, 11:19 PM

#

I fixed only_sections error and updated code to compile for my target arch
It compiles and runs, but there is only elf file emitted, no objcopied file appears. Even though there is .data section in elf file
I am debugging this right now, but here is my build function just in case (I didnt change the DefaultEmptyFileStep)

pub fn build(b: *std.Build) !void {
    const features = Target.riscv.Feature;
    var disabled_features = Feature.Set.empty;
    var enabled_features = Feature.Set.empty;

    disabled_features.addFeature(@intFromEnum(features.a));
    disabled_features.addFeature(@intFromEnum(features.c));
    disabled_features.addFeature(@intFromEnum(features.d));
    disabled_features.addFeature(@intFromEnum(features.e));
    disabled_features.addFeature(@intFromEnum(features.f));

    enabled_features.addFeature(@intFromEnum(features.m));

    const query = CrossTarget{
        .cpu_arch = .riscv32,
        .os_tag = .freestanding,
        .abi = .none,
        .cpu_model = .{
            .explicit = &std.Target.riscv.cpu.generic_rv32
        },
        .cpu_features_sub = disabled_features,
        .cpu_features_add = enabled_features,
    };

    const target = b.resolveTargetQuery(query);

    const exe = b.addExecutable(.{
        .name = "test_objdump",
        .root_source_file = .{ .path = "src/stack.zig" },
        .target = target,
        .optimize = .ReleaseFast,
        .strip = true,
    });

    exe.setLinkerScript(.{ .path = "../linker.ld" });

    b.installArtifact(exe);

    const oc = exe.addObjCopy(.{
        .basename = "file",
        .format = .bin,
        .only_sections = &[_][]const u8{ ".data" },
        .pad_to = 0x1000,
    });

    const oc_opt = try DefaultEmptyFileStep.init(&oc.output_file);

    b.default_step.dependOn(&oc_opt.step);
}

hollow peak Mar 7, 2024, 1:28 AM

#

# rm -rf zig-{cache,out}
# zig build
# find zig-cache/o | grep test_objdump
zig-cache/o/5cf588f41859443c8a9e0bb2d3f0ac72/test_objdump.o
zig-cache/o/5cf588f41859443c8a9e0bb2d3f0ac72/test_objdump
# find zig-cache/o | grep file
zig-cache/o/328bd57922099c8ec0920fd390a549a1/file

#

with the code I sent except dumping .data

#

also not doing &.{".data"} is weird

wispy kite Mar 7, 2024, 1:35 AM

#

why does it put it inside the zig-cache folder? can it emit this file alongside elf in zig-out/bin?

hollow peak Mar 7, 2024, 1:37 AM

#

wispy kite why does it put it inside the zig-cache folder? can it emit this `file` alongsid...

of course it can

#

you have to tell it to do that

#

by telling the builder to install it

#

but that's outside of the scope of your question

#

I didn't know you wanted to do that, thought you just wanted to use the section from some other build step

wispy kite Mar 7, 2024, 1:39 AM

#

mb, needed to be more precise in original post

hollow peak Mar 7, 2024, 1:39 AM

#

https://ziglang.org/documentation/master/std/#A;std:Build.addInstallBinFile

#

use this function

#

you can construct a LazyPath from the oc_opt.output_file

wispy kite Mar 7, 2024, 1:57 AM

#

hm, this

b.addInstallBinFile(std.Build.LazyPath.relative(oc_opt.output_file.path.?) , "file.bin")

tells about attempt to use null value on path.?
I assume .? just unwraps the optional, not 100% familiar with all the zig syntax yet

#

edit: oc_opt

hollow peak Mar 7, 2024, 2:00 AM

#

wispy kite hm, this ``` b.addInstallBinFile(std.Build.LazyPath.relative(oc_opt.output_file....

you can't do that

#

the output path is populated later

#

use .{.generated = &oc_opt.output_file}

wispy kite Mar 7, 2024, 2:03 AM

#

works! I don't believe such a simple task took me so long even with the immense help of yours. thanks ❤️

hollow peak Mar 7, 2024, 2:03 AM

#

no problem

#

I would recommend trying to become more familiar with the build system before trying to wrestle with it

wispy kite Mar 7, 2024, 2:05 AM

#

my primary way to get familiar is just dive in and do the task, but this API was too overwhelming

#Conditional build step