#Move Modules under the Terasology Repo

1 messages · Page 1 of 1 (latest)

jagged blaze
#

I know we have all this tooling for managing repositories but why have it at all. its just developer overhead where you juggling 30+ repositories. I would rather just have all the core modules in the main repo.

jagged vigil
#

This is a very important topic. My personal opinion is quite well known already - but if the desire is minimum scope game focus this is a very legit path forward.

It is not the path I would take personally, partly as there's less meaning for me to be involved (since I thrive on sometimes-not-needed complexity) as well as potential less meaning for the overall project (the foundation of which was on extensibility etc)

However, it remains very valid depending on the goals and those who would join a journey making that a reality. I'm not going to impose my personal weirdness on the future and fully acknowledge that might not be good for viability anyway.

leaden ember
#

I want to add two questions to the discussion of "mono repo vs multi-repo":

  1. Would a decision for mono repo imply getting rid of the concept of modules? (edit: already short-term)

My current assumption is "no", it would only get rid of the technical aspect of having separate repos for modules.

  1. Would a decision for mono repo imply that all (current (in and out of the minimized scope) and future) modules would live in the engine repo? (edit: already short-term)

From what Michael wrote:

have all the core modules in the main repo
it sounds to me like "no", only the "first class citizen" modules (aka iota?) would live in the engine, "second class citizen" modules (aka omega-iota?) would continue living in separate repos and "third class/party citizen" modules (aka external ones) would live in whoever else's space - provided we still want to cater to modders in the future.

Now that IMO sounds like a lot of complexity, because we need to fetch modules from different places and would probably have different logic for handling the individual "classes" of modules. That's complexity I'd like to avoid for our short-term minimized scope. If we want to do it like that, then for the minimized scope I'd propose to focus on first class modules (iota?) living in engine and throw out everything else for the time being. once we got the house in order and want to expand again (with/into second class modules), then (and only then) we can revive that part. Does that sound reasonable?

plush prism
#

I think that there should be no difference between "second class citizen" and "third class/party citizen" except for the location of the source code, at least when it comes to building them. For third-party mods to work a module needs to define it's dependencies and should be build against those dependencies, including the engine.

I strongly believe that the same should hold for "fist class citizen" modules, having them in the main repo just makes it easier to bump versions and ensure that everything fits together. But if we (eventually, long-term) aim for extensible modularity also these core modules should follow this approach.

Dropping extensible modularity (for now) would allow us to just build and bundle one thing, not caring about module versions at all, and in fact ignoring them. If we do this cut, Terasology becomes somewhat "closed" to outside modules (technically you could still write modules, but whether something works or not is closer to pure luck than proper configuraiton).

#

All this might come down to the bigger picture on what are we doing in CI? IMHO, we should be able to draw simple flowcharts and similar diagrams to depict what is happening when, and any detour or additional source (of information or data) migth indicate (unnecessary?) compexity we might want to cut.
In particular, I want to be able to do all the steps locally as well, for simpler debugging, emergency cases where CI is not available, stuff like that.

#

@jagged vigil I'm pretty sure there are still a ton questions regarding CI and automation, and I'd love to see elegant and comprehensible solutions applied across the board for Terasoloy 🤓 Some questions that might get us started:

  • what are the artifacts we are producing?
  • how do we build the engine?
  • how do we build a single module?
  • how do we bundle a game release?
    • where is this bundle defined?
  • where are the respective pipelines and jobs defined?
  • how can we have reproducible builds?
leaden ember
# plush prism I think that there should be no difference between "second class citizen" and "...

there should be no difference between "second class citizen" and "third class/party citizen" except for the location of the source code
yup, makes sense 👍

Dropping extensible modularity (for now) would allow us to just build and bundle one thing, not caring about module versions at all, and in fact ignoring them
do you think that this stands in contrast to "(eventually, long-term) aim for extensible modularity" or is there a way that we can make this cut for now but will still not block our potential aim for extensible modularity long-term?

leaden ember
plush prism
#

I believe cutting it completely now would lock us in on the "we just build one thing" approach, and it will be quite hard to introduce module versioning and such again later.
Thus, I believe this is a decision we can make, but we need to be conscious about implications for mid- to long-term efforts.

jagged blaze
#

it's more of a time saving measure, we wrote a lot of tools to just create piles and piles of PR's for all these small repositories and the process of merging and kicking off a huge build chain is time consuming. I think @leaden ember can attest to that. if we can just simplify the workflow so we can move stuff in faster then I want to do that. its developer time saved, since we're a small group we shouldn't have to manage a burden that we don't necessarily need to want to have :?

jagged vigil
#

Looks like I might have a bit more to respond to in here than doable mid-workday 🙂

Real quick on my stance: I've grown to dislike the monorepo approach at a professional level but I do get the value and it is definitely the right fit in some cases. I just like having lots of little things in various shapes to then fit into a series of shaped holes in tooling rather than a huge thing full of pointy bits you then try to wrap tooling around. Monorepo really ideally should go with something like Bazel or be for projects that don't want or need extensibility or excessive use of microservices and such.

That stance is probably a consequence of being a tooling nerd. It was fine back when I was more active and was able to keep the automation working decently. I don't want to force anybody to take n actions just because there are n modules involved - tooling should be able to do that in 1 action. If it doesn't then that's a problem, and at present it doesn't.

Hobby-wise I liked it for the simplicity of architecture at the cost of logistics effort, with absolutely everything content-related torn out of the engine, which was a differentiator vs something like Minecraft (and possibly MineTest and others?) and would probably help support better organization around automated testing and release management.

On whether you could collapse down into monorepo with a set of core modules and re-expand later: that sounds very painful, but on the other hand it is a valid approach to at least help stabilize and focus code. I can appreciate that possibility while still being against it on a personal level.

I'm seeing something similar playing out with backstage.io which is struggling with their extensibility since it wasn't a focus at first and they began with monorepo and telling 3rd parties to just submit PRs with new plugins to the upstream repo. They have a whole separate setup tree that comes "blank" of plugins then you have to literally copy paste or sub-clone (is that a thing? I forget - just grab specific subdirs) to get the right pieces. Coincidentally my company may be trying to get in on helping them straighten all that out - but there is this weirdness around what counts as a core plugin, what might be marketplace, and how would independent 3rd parties do stuff on their own. Not to mention a pile of "engine" stuff (like various UI/UX topics) still lives in the main repo and can't easily be customized.

So multi-repo with a minimal engine it is sort of a theoretical "better" IMHO but a huge challenge to actually practice. A functioning approach like that is one of my cornerstones for the "identity" of Terasology, but that becomes a different and maybe bigger topic. A part of the personal plan I was working towards was a focus on improving tooling here.

For CI I think you only really need the complexity Jenkins affords if doing the wild multi-repo thing. Mono repo could just be a GitHub action. One of the "fancy" things that also would get into identity and developer experience would be the partial past effort of being able to solely clone a single module and actually run the whole game from that module source directly to exclusively dev on the module.

#

That wall of text ended up coming fairly easily anyway, maybe I generally should write near noon rather than midnight 🤔

leaden ember
jagged vigil
#

For the quick response to the 2 Qs above I'd say no and no - you could aim long term for the "current" approach and live temporarily or partially in monorepo. It just comes down to levels of pain and effort I think 🤔

leaden ember
jagged vigil
#

Not sure what you mean? Sorry

#

over the short term? Or that it would happen very soon?

leaden ember
#

I'm wondering if we can have a monorepo with curated (first class) modules within that monorepo and second class / 3rd party (difference is only location of code) modules outside of it

#

so that for now we can focus on that monorepo and a small set of curated modules + a dummy second class module outside of the mono repo to ensure logistics work and don't lose sight of other extensibility-related aspects until we get to fully embrace them again in the mid- to long-term (at which point we would still keep the monorepo)

jagged vigil
#

Okay got it. I think we sort of had that years ago after we split out some modules but still kept some Core stuff in the engine repo, which then later were excised in the name of more pure architecture, but yeah we probably chopped into pieces that were too tiny

It worked OK'ish but was indeed weird to treat some modules differently. It is doable.

plush prism
#

Would a decision for mono repo imply getting rid of the concept of modules? (edit: already short-term)
No, not necessarily. While it is an option to go that way, we can certainly keep the concept of modules as is.

Would a decision for mono repo imply that all (current (in and out of the minimized scope) and future) modules would live in the engine repo? (edit: already short-term)
No, not necessarily.

However, this is more closely tied to the question what artifacts we want to produce. If we want to avoid complexity around building modules outside of the engine repo, the answer would be "yes", I guess 🤷‍♂️

#

Monorepo really ideally should go with something like Bazel
I think Bazel makes most sense if you have a mixed technology stack (might be interesting in case we introduce more C++ or other "native" code for rendering purposes), but with a pure Java (or JVM) codebase using good ol' Gradle to produce multiple build artifacts might just be fine.

I don't want to force anybody to take n actions just because there are n modules involved - tooling should be able to do that in 1 action. If it doesn't then that's a problem, and at present it doesn't.
There's a difference between processes that can be automated by tooling, and processes that require human interaction. When we require reviews before merging, that's nothing that tooling can take away from us. Since we split one logical change over multiple repos, we now need to handle multiple reviews either way.
We can address issues like this from different angles. For instance, change the process to not require manual reviews, now automation works (for instance, direct push to multiple repos supported by IntelliJ). We could make use of existing tooling and/or a combination of self-made tooling (for instance, in a clean workspace check out all PRs belonging to that logical change, make a local review, and use gh pr review to approve all PRs at once if happy).

leaden ember
#

avoid complexity around building modules outside of the engine repo
if we want to avoid that completely, we would need to decommit extensibility.
if we don't want to decommit extensibility then we will have some complexity around building modules outside of the engine repo - monorepo or not.
what I'm not clear about is how big the diff in complexity is for monorepo compared to multi-repo in this case.

plush prism
#

I'm wondering if we can have a monorepo with curated (first class) modules within that monorepo and second class / 3rd party (difference is only location of code) modules outside of it
Yes we can.

👍 We have a single repository to work with and can focus on whatever module lineup we have in there.

🤷‍♂️ We'd have to decide what to build (artifacts), and depending on this, we might gain some insights for "mini-mono-repos" (as we could reuse knowledge and code to build multiple modules from one repo), or we'd tie the build pipeline even closer to some structure in the repo, which would make it harder to separate later.

👎 We would have to immediately invest into the CI setup (or wouldn't we?). The CI setup for the module-only repos will likely be different and is yet another thing to learn for new contributors and even ourselves.

jagged vigil
#

On manual review: yeah I would necessarily tie this to declaring "logical groupings" of change/PRs with a primary PR you review and "pull" using tooling to then ideal also readily show changes locally, review that way, and either hit the button on the main PR then automation merges the rest or you just push locally 🤖

leaden ember
#

CI setup for the module-only repos will likely be different and is yet another thing to learn for new contributors and even ourselves
that's something we'll always have to deal with if we commit to extensibility, right?

leaden ember
# plush prism > I'm wondering if we can have a monorepo with curated (first class) modules wit...

We'd have to decide what to build (artifacts)
if we were to say that we want to build individual modules into artifacts, then theoretically the CI setup for first and second class and 3rd party modules could be quite similar with the main differentiator being the location of the code (subdir in engine monorepo, root-level dir in module-only repo under our control, root-level dir in module-only repo under 3rd party control), right?
(security-concerns of 3rd party put aside for the moment)

dense peak
#

Although this is a bit more nuanced than mono-repo vs multi-repo, I am actually supportive of moving the "core" modules into same repo as the engine, purely to make managing changes easier. Keep the module system but just place the modules that fit our core vision adjacent to the engine. I think that part of the build is actually in a fairly good state, at least from the perspective that I used it as a base for modernising the build in Destination Sol.

Since Bazel has come-up a few times here, I should probably add that I don't think it would be a great idea. I work with Bazel and C++ in my job and it's a pain to set-up - you practically need a full-time build person to maintain the build scripts (no offence intended). That usually ends-up being my task, so I know very well (I am a developer, not a build scripter). It does not work well with distributed projects, or any code that you don't directly control.

trim spear
#

you could also 'simulate' a monorepo through using git submodules.

#

of course fluent git kung-fu is required to properly operate that.

jagged blaze
trim spear
#

Of course, having a high level of modularity needs maintenance effort.
(I did not read all of the wall of text tbh, 😉 )

I would now go and collect all the advantages and disadvantages you all see into a table, easy to memorize, to make a informed decision how to proceed.

#

But keep in mind also a big mono repo may be a pain to maintain.

jagged blaze
trim spear
#

yes, at work a bunch of projects are in a monorepo, it's a pain.
The others are multi-repo based, with proper dependency management. They have also their problems, but maintaining a repo and a CI build are none of those...

#

of course you could say that's all hysterically grown... true, with starting from zero it might be different...

#

but we're also not starting from zero...

jagged blaze
#

each module is very small so the effort of updating a single module that does one specific thing is more work right?

#

I can see submodules working if you say pull in an RHI abstraction for your engine

#

modules are tightly coupled to the engine so it makes sense to have them in the main repo

plush prism
#

its still difficult to update and push all thsoe indvidual repositories.
I had thought that update and push kinda works equally well as with our current setup, but that's not the bottleneck, but more the reviews and building on CI...
Also, including modules as git submodules in the engine repo sounds like the wrong around - you'd have your own dependencies as submodules, not your dependants, wouldn't you? I had thought about a "workspace repo" as a place to reference the engine and a bunch of modules as submodules at some point, but didn't see any real advantage ...

#

modules are tightly coupled to the engine so it makes sense to have them in the main repo
Not sure what exactly "tight coupled" means here. We have a dependency on engine features, but most integration is via the ECS, isn't it?

jagged blaze
#

some of the core modules are very tightly bound to each other and the engine where if you make an API change in the engine you have to touch piles and piles of modules.

#

none the less I would like to have a core set in the main repository so we can make lots of changes fast without having to do the whole process of pushing to 20 or so modules reviewing piles oof PRs and merging all of them

#

I'm more for a submodule if its like some core library we're using and we might change it to some smaller degree. like if we use the DirectXShader compiler you don't want to bring in that huge dependency. upstream tends to introduce more changes so manging to update your copy with upstream becomes a challenge if you embed it into your project.

dense peak
# jagged blaze none the less I would like to have a core set in the main repository so we can m...

none the less I would like to have a core set in the main repository so we can make lots of changes fast without having to do the whole process of pushing to 20 or so modules reviewing piles oof PRs and merging all of them
Managing all of those separate module repos is an experience I recall with not a great deal of fondness either. It was worse when you tried to patch everything in the omega line-up on-top.

jagged blaze
#

we can use LFS if have to store a lot of assets

jagged vigil
#

Real quick on Bazel: I was really just trying to call out some sort of monorepo specific tooling as being ideal if you truly have a good use case for monorepo. I suspect there are a lot of not-really-natural monorepos that are struggling either because ill-fit tooling or not being a 100% fit and commitment to the monorepo style (which could easily happen over time as the project scales)

Much like off-the-shelf game engines maybe never being an ideal fit for voxel games it may be that the nature of a heavily extensible game is incompatible with monorepo unless you have some really nifty logistics and custom automation - at which point it could really just as well be multi or some sort of hybrid.

But either you start with a committed fit and ideal tooling, or you have to put in the work to make something custom yourself. I think. So a big question either way is whether we'll be able to make whichever approach into an amazing developer experience.

leaden ember
#

it may be that the nature of a heavily extensible game is incompatible with monorepo unless you have some really nifty logistics and custom automation - at which point it could really just as well be multi or some sort of hybrid.
from what I understood from the discussion so far and the vote on extensibility options on #1125059784656302080 message, we're currently aiming at monorepo for engine and whatever we want to ship as part of our game (initial scope only a few modules for basic gameplay, then extend scope bit by bit) plus the technical means to allow for extensibility (but without ourselves providing alternatives/variants ourselves) and potentially a kind of dummy module to ensure those means work.

not sure we need "really nifty" (read: complicated or complex) logistics for that. but that's exactly what I'd like to brainstorm with @plush prism and you after / while diagramming our current CI state and what a target CI state for the above-mentioned current aim could look like - the comparison of which should help us with identifying a migration path and assess how feasible, actionable, and hard it would be.

jagged vigil
#

Okay, over in the other spot you just mentioned current state. I'm certainly curious and happy to try brainstorming future state. It is going to be hard for me to get excited and motivated about a logistics approach that goes against my nature though, so to say.

We did the monorepo engine approach long ago. I'm not necessarily opposed if that's where the project should go, and yeah we can resume the mildly awkward approach of different logistical setups for different kinds of modules. It just would feel like going backwards to me, you know? That one is a personal/skill thing. It may make more sense there to just start from scratch with something totally different like an entirely GitHub Actions based thing. Which I have barely used so I probably wouldn't be a great resource for it.

leaden ember
#

approach of different logistical setups for different kinds of modules
not sure the logistical setup would be that different. mostly the location of the source code of the module would differ, but that's it I think. but maybe the picture in our minds differs so we're talking about different things, unaware of it.
I hope that the diagramming exercise helps in clarifying some potentially underlying assumptions of each of us that are not that visible at the moment.

leaden ember
#

I see (B) as decoupled architecture. We write modules for one coherent gameplay. Those modules have common base. Same base as for "external" modules. Same rules.

Like Spring. Services and modules in spring lives with one ruleset. (Almost, several "core" modules like webmvc/ scheduling gives extension for modules, there i suggesting subsystems: adding audio, network, rendering and other core functions)
@queen talon do I understand correctly, that the subsystems you propose are that "common base" and that you would have them continue to live in the engine? what about the "modules for one coherent gameplay". would they live in the engine (monorepo) or outside of it (multi-repo as currently)?

queen talon
leaden ember
#

okay, thanks for the input

leaden ember
#

<@&1125136359733067817> part of our current milestone is developing a migration path for a potential mono-repo approach. while we're making good progress with the code-wise milestone elements, I feel like we kind of ceased talking about this point here...
questions that come to my mind would be:

  • which modules do we want to move into the engine repo?
  • where should they live? (/modules dir in engine?)
  • what do we need in terms of CI-wise changes to make this work?
  • do we see any other dependencies?
  • do we see potential blockers that could occur?
dense peak
# leaden ember <@&1125136359733067817> part of our current milestone is developing a migration ...

My thoughts are:

  • All the modules corresponding to gameplay areas that we want to focus on can go into the engine repository (Core Gameplay, Core Worlds, etc.)
  • Let's get TutorialMinimalEngineDemo working again, if possible. I'm sure that I've patched it back into a working state privately many times before because it's a great test when the engine's acting abnormally.
  • Eventually (although not as a first step) it might be worth consolidating some modules based on gameplay groupings, rather than minimalist functional ones (a pretty dependency tree does not always lead to the least cognitive overhead in comprehending how it all fits together)
  • Modules should continue to live in <workspace_root>/modules
  • CI changes are probably not needed, as unit tests should be automatically picked-up by gradle (although I could be wrong here)
  • I can't think of any clear blockers/dependencies right now - it's just not been a focus as things work at the moment, even if in a very fragile way
  • As an optional extra, it would be nice to preserve the pre-existing git history during the move

Everyone else is welcome to their own opinions on this too, of course.

jagged blaze
#

my two cents on generic interfaces are that you tend to provision for very general cases. I don't think its optimial in a lot of cases because you will provision for a lot of future use cases. if you have N possible cases you might use a map when the starting set is only lets say 4. switching over an enum vs doing a lookup in a map. I think modules around content make more sense then trying to generalize some generic interface.

I guess there are things where you might want to link things between modules where I think like a inventory module would make sense. have a core module that provides the basic interfaces for the game and other modules can sit underneath that. decide what needs to be extended and extend it as so.

#

I'm kind of suggesting a more monolith approach to tons of tiny self contained systems. by that nature you handle for the most generic cases. I think then you can find where there are good breaking poionts to pull stuff apart.

#

Core Gameplay, Core Worlds can just be merged all together to start with maybe?

leaden ember
leaden ember
plush prism
leaden ember
#

heyho @dense peak @trim spear, @plush prism started writing up a roadmap regarding mono repo migration including a lot of concerns we might want to reiterate on ⬆️

#

thanks for this @plush prism !

grave bobcatBOT
#

Gave +1 Slime Tokens to @plush prism

trim spear
#

Reiterate in text form (forum) or talkin?

plush prism
#

We can discuss here and write an update on that issue when we reach a conclusion or some other noteworthy insight