Writing C/C++ Build System

[cursive jacinth]

Hello fellow C++ community. I decided to write my own build system for myself (and maybe for others who will find it usefull) which should be as simple to use and as powerful as possible. I want everything be defined in single build file and right now need to sum up other people opinion on this. Right now I came up to this:

(predefined statements)
compiler, standart, output_dir, obj_dir, src_dir, include_dir, lib_dirs

Example config using these:

compiler = "clang++" //if it is in the path, otherwire privide path to executable
standard = "c++20"
output_dir = "build/bin"
obj_dir = "build/obj"
src_dirs = {  "../src" }
include_dirs = { }
lib_dirs = { "SDLmain.lib" "SDL.lib" }

(User defined variables)

Example:

path_to_sdl = "path/to/sdl"

include_dirs = { ${path_to_sdl}/include }

Right now I wrote file lexer and until it is not too late I want to know

What are your thoughts on this design? What is missing, what can be improved? I will glad to hear and implement you ideas and suggestions! Thanks!

[jade fiber]

Is this is a toy project, or something you want to be actually useful?

If you want it to be useful, you need a selling point. What other build systems do wrong that yours does right?

[cursive jacinth]

@jade fiber yep. I want it to be usefull buy my imagination is soo dead xD.

Oooh I got you! Never thought of it! Thats a really good idea I will look into it. Thanks!

[jade fiber]

Or if you don't make the config file C++, at least make it a common format such as JSON

[jade fiber]

This is this first thing people gonna ask

[cursive jacinth]

#include "../src/makepp.hpp"

int main() {
  Maker build;
  build.SetCompiler("clang++").SetStandard("c++20")
       .SetBuildDir("../build");

  build.AddExecutable("main1")
       .AddSources({"hello_world.cpp"});

  //build.AddLibrary("libmylib.a");
  build.Make();
}

Going as is. Working on cross-compile rn.

[cursive jacinth]

Exactly

wdym?

I already use them here

you pass sources as vector

or you meant smth else?

#pragma once

#include <iostream>
#include <vector>
#include <filesystem>

#include "target.hpp"

class Maker {
  private:
    std::vector<Target> _targets;
    std::string _compiler = "clang+";
    std::string _build_dir = "build";
    std::string _standard = "c++17";
    std::string _output_extension_exe = "";
    std::string _output_extension_lib = ".a";

  public:
    Maker() {
      DetectPlatform();
    }

    Target& AddExecutable(const std::string& name) {
      this->_targets.push_back(Target(name));
      return this->_targets.back();
    }

    Target& AddLibrary(const std::string& name) {

    }

    Maker& SetCompiler(const std::string& compiler) {
      this->_compiler = compiler;
      return *this;
    }

    Maker& SetStandard(const std::string& standard) {
      this->_standard = standard;
      return *this;
    }

    Maker& SetBuildDir(const std::string& build_dir) {
      this->_build_dir = build_dir;
      return *this;
    }

    void Make() const {
      if (!this->_build_dir.empty())
        std::filesystem::create_directories(this->_build_dir);
      
      for (const auto& target : this->_targets) {
        std::string output_path = this->_build_dir + (this->_build_dir.empty() ? "" : "/") + target.GetName();
        std::string command = this->_compiler + " -std=" + this->_standard + " -o " + output_path;

        for (const auto& src : target.GetSources()) {
          command += " " + src;
        } 

        for (const auto& include_dir : target.GetIncludeDirectories()) {
          command += " -I" + include_dir;
        }

        for (const auto& dep : target.GetDependencies()) {
          command += " " + dep;
        }

        std::system(command.c_str());
      }
    }

    void DetectPlatform() {
      
    }
};

I want to make it full headers. So people can compile build file very simple with just 1 command without links

[jade fiber]

Another thing that would be nice to have is respecting the standard environment variables (CC, CXX, CFLAGS, CXXFLAGS, etc)

[cursive jacinth]

Can't setting these just be replaces with methods of Maker class? Like

.SetCompilerFlags(const std::vector<std::string>& flags)

[forest saddle]

works

that will be cool

...writing c++

[jade fiber]

Thing is, when I build some random library, I want its build system to respect the conventions for setting compiler and flags

I definitely don't want to edit its build script to change the flags

[cursive jacinth]

But how then would you suggest to set them along with method?

[jade fiber]

.SetCompiler should probably be ignored if the env variable is set

The flags env variables should be appended to the end of flags you have hardcoded. I think

[cursive jacinth]

I think i can use std::getenv to retrieve environment and parse exisiting flags

Will work on it aswell thanks!

[cursive jacinth]

like this

_compiler = std::getenv("CXX") ? std::getenv("CXX") : _compiler;

So constructor of Maker class will automatically set all flags

[hidden owl]

Be careful to use a language that is turing complete for the build system. I can guarantee there will be people that will do their software in the build system, also, if the language is C++ for the buildsystem, does that mean that I would need to compile the build files first to build the actual project?

[cursive jacinth]

@jade fiber yes you will need to do such as

clang++ main.cpp -o main

I dont see problem here cuz it is simple and straightforward

[jade fiber]

IMO non-turing-complete build systems are uncool. You'll inevitably want to do something non-trivial, and then you have to resort to shell scripts or whatever

[cursive jacinth]

My build system is basically a cpp headers. So why they won't just use c++ itself?

[hidden owl]

The main advantage of build systems is basically to care less about the compiler and platform and let almost everything automated, of course you are still needing to care about edge cases but they are rare.

If you need to compile the build system you are going to need to care about the compiler, platform and etc all the time, because the build system itself is a binary file. also it makes CI more complicated to use.

[cursive jacinth]

How are you supposed to even run c++ without a compiler?

If you are writing it for sure you know at least how to compile a single file into exec

[jade fiber]

🤷‍♂️ I don't see a problem with building the build system file one. Heck, you could attach a shell script that automatically selects the compiler to do that

[hidden owl]

Care about the compiler, as in, set the compiler variables and etc

[jade fiber]

Being cross-platform isn't hard, you just cl /EHsc main.cpp or g++ main.cpp on MSVC and everywhere else respectively

[hidden owl]

I've been coding in C++ for a couple months, collaborated to a library that I was using and were having issues with emscripten and not a single time I had to call the compiler directly. At maximun I just added a couple arguments from Cmake/meson

[cursive jacinth]

@hidden owl But you just need to compile single build file. No need to compile it for different architectures

Cross-compile handled in library itself

[cursive jacinth]

Just

compiler file_name.cpp -o output_name

[hidden owl]

how are you going to distribute the headers of your build system? Also would they need to be put together with the project in the repository?

What if there is a bug in the build system, how would I need to update it?

[cursive jacinth]

Distributing as headers and share lib. You will need headers in separate directory

[hidden owl]

if there's a share lib, wouldn't it be better to have a bootstrapped build system where the build system compiles and setup itself?

[jade fiber]

I'd make it header-only, personally

It's not like you'd need to rebuilt it often

[hidden owl]

I bet people would put in the include folder and then the build system will build system together with the app

[jade fiber]

🤷‍♂️

[cursive jacinth]

I meant both headers and sharelib for diff users

[hidden owl]

It would be funny

[jade fiber]

My hot take is that nobody will ever need anything other than header-only for this

But that's me

[hidden owl]

Use modules

So people struggle to find a way to install gcc-13

[cursive jacinth]

Just replace folder where you store headers

[cursive jacinth]

It might be usefull for game engines actually

[hidden owl]

How so?

[jade fiber]

Btw, the reason why I have opinions on all this is because I've been contemplating making a build system similar to this one

[cursive jacinth]

Include the setted up build system files for different targets

[hidden owl]

https://tenor.com/view/manly-tangled-kissing-disney-unicorn-gif-5669699

[cursive jacinth]

This might become real 😉

[willow echo]

if you're planning on making it a C and C++ thing then don't assume clang++ everywhere

you're going to need to be able to execute C and C++ compile commands, using different compilers, and link the objects

[cursive jacinth]

Right now I am targeting c++ only. I will later think how to implement C support but I can just make C headers

[cursive jacinth]

it was just example btw

[willow echo]

but before i say anything else: this looks really interesting. you're underestimating the huge number of corner cases and weird situations a build system will run into, but that's completely OK. in fact you should ignore much of the what-ifs now if you feel like exploring the design

[willow echo]

ok, no worries

[cursive jacinth]

also do compiler's flags are too different from each other? like clang/gcc/mingw

[willow echo]

i'll show you a pet build system of mine, implemented in GNU make, not as a suggestion in any way but just to show how some projects will mix C and C++ translation units

[cursive jacinth]

Okay! Thank you!

[willow echo]

it's a huge space of variations, but don't be put off by that. if you're in a creative mood just go with that, there's time for corner cases later

@cursive jacinth probably easiest to just have a look at the output of my github CI. https://github.com/imaami/libcanth/actions/runs/10535559287

unfortunately the verbose build option is off in that run, let me start one for you with the compile commands visible

[cursive jacinth]

that will be so nice~

[cursive jacinth]

I am just doing this for the first time. I am also very interested in build process cuz our year c++ university course didn't cover that much so I want to discover it

[willow echo]

i find the whole thing interesting too, there's no shortage of stuff to look into

i have a compiler feature introspection thingy in my build system. overall it's highly cursed, it uses extremely horrible GNU make syntax and features

[cursive jacinth]

ooh

[willow echo]

@cursive jacinth this run will have verbose output https://github.com/imaami/libcanth/actions/runs/10540983993

[cursive jacinth]

I will look into that, thanks

[willow echo]

i do a test build with each one of these compilers: gcc-7, gcc-8, gcc-9, gcc-10, gcc-11, gcc-12, gcc-13, gcc-14, clang-6, clang-7, clang-8, clang-9, clang-10, clang-11, clang-12, clang-13, clang-14, clang-15, clang-16, clang-17, clang-18, clang-19, clang-20

so a few of them

the compiler flags with all the different warning settings etc. aren't any representation of some absolute "correct" set of flags for the compiler versions. they're based on what language standards and features i'm using

btw, i'm explicitly only using the C23 standard of C, or as close to it as I can, so everything in my build only really applies to this specific choice of language standard

the C++ side isn't really used in any proper way so I haven't spent as much time on it

[cursive jacinth]

ye I can see

[willow echo]

but don't look too long, don't get exhausted 🙂

focus on the compiler flags you use now and work with those

[cursive jacinth]

haha

[cursive jacinth]

I would also add -Wall -Wextra

it seems to be usefull

[willow echo]

I think I do have those in every build config as a baseline

[cursive jacinth]

especially as I can see for clang

[cursive jacinth]

yes

[willow echo]

clang has -Weverything but it literally enables every useful and useless warning, a lot of them need to be overridden with -Wno- flags to make the result reasonably useful

it's probably not a good idea to use the -Weverything -Wno-... approach with clang though, it takes fiddling

-Wall -Wextra is good

[cursive jacinth]

okay

right now I think I will focus on clang mainly untill most of the usefull features will be done

[willow echo]

sounds like a good strategy

[cursive jacinth]

gonna test build system on my sdl2 + opengl + imgui project

[willow echo]

a build system literally based on C++ classes is a really interesting concept imho

[cursive jacinth]

some kind of stress test with lots of libs involved

[cursive jacinth]

thanks to @forest saddle who suggested it

[willow echo]

testing a build system is just doing all the permutations of configurable parameters and seeing what happens 😄

[cursive jacinth]

xD

[willow echo]

i still don't do that btw, i need to add that to my CI. current number of builds in ubuntu-24.04 is "only" like 46

it'll grow with a multiple of jesuschrist when i permute all the things

[cursive jacinth]

last CI i wrote was back ago in october. Ig I already forgot everything

[willow echo]

my background is in bash scripting so i fall back on writing scripts usually, i'm going to slowly move more stuff into a collection of bash utility functions and make the actual github workflow file much smaller

now there's a lot of inline bash there for no other reason than not bothering to dedup it yet

[cursive jacinth]

oh

one day the time will come 😅

[cursive jacinth]

By the way how much time did you spend on your build system?

[willow echo]

Too much

[jade fiber]

Where are you getting the binaries of all those compilers btw?

Do you build them from source? I consistently have problems with linux distros not updating the compilers fast enough. Or not letting you install different versions of one (ubuntu doesn't let you have different libc++'s, I think?)

[willow echo]

It's all im the github worlflows file

The installatiom of all of them

[jade fiber]

Ah, you don't test libc++, and the rest is simple 😐

Dunno what I expected

[willow echo]

Just adding repos and running apt basically

To an extent yes

[cursive jacinth]

Added preset default toolchains for corresponding systems

  struct ToolchainConfig {
    std::string compiler;
    std::string archiver;
    std::string linker_flags;
    std::string output_extension_exe;
    std::string output_extension_lib;
  };

std::unordered_map<std::string, ToolchainConfig> _toolchains;

this->_toolchains["windows-x86_64"] = {"x86_64-w64-mingw32-g++", "x86_64-w64-mingw32-ar", "", ".exe", ".lib"};
      this->_toolchains["windows-x86"] = {"i686-w64-mingw32-g++", "i686-w64-mingw32-ar", "", ".exe", ".lib"};

      this->_toolchains["macos-x86_64"] = {"clang++", "ar", "", "", ".dylib"};
      this->_toolchains["macos-arm64"] = {"clang++", "ar", "", "", ".dylib"};

      this->_toolchains["linux-x86_64"] = {"g++", "ar", "-L/usr/lib/x86_64-linux-gnu", "", ".a"};
      this->_toolchains["linux-arm64"] = {"aarch64-linux-gnu-g++", "aarch64-linux-gnu-ar", "-L/usr/lib/aarch64-linux-gnu", "", ".a"};

Also now it automatically parse env variables as @jade fiber suggested and automatically detects current platform and architecture trying to build with preset toolchains if user didn't set compiler manually.

I think now its time to add some sort of cross compile. I think each Maker class should be set to needed arch

Yep. I can use it for testing