We used to run on-demand AWS / Google | Unreal Source | Page 1

fiery hornet Oct 11, 2025, 11:50 PM

#

our setup is not super complex really. we have a p4 trigger that calls some lambda functions, which turns on an ec2 machine. that machine runs a script, which gets some info in a dynamodb table about the CL to build and packages the game, and uploads it to S3. that's it

#

our scripts do kind of "reinvent" the wheel a bit, since it's doing a ton of things that are handled by Horde and all, so we're looking to migrate to it, but more importantly find ways to accelerate the builds

#

while it's not a major thing that they have tasked me to do, I am also looking to see if we can reduce costs and make some things more efficient, since I know we don't have a huge budget.

#

I'm pretty sure that paying some machines at e.g. OVH would be less expensive than our current AWS setup.
There's also a security aspect, which AWS handles pretty nicely with their VPC, so I'd have to figure out how to replicate that probably, but it shouldn't be too hard

#

I have the before/after cost if you’re interested.
@muted flicker I would be interested! (I'm not sure the thread pinged you, so just mentioning you directly!)

#

I'm also thinking of writing an implementation for Backblaze storage in Horde, since it's a lot less expensive

muted flicker Oct 13, 2025, 1:08 PM

#

fiery hornet > I have the before/after cost if you’re interested. <@139515614896193537> I wou...

ah cheers! I dug up the billing receipts.
We used Google Cloud originally (3 on-demand virtual machines, 1 always-online master virtual machine) which had a cost of about $800 USD/month. I dont have the exact machine specs but they were pretty beefy with CPU and ram to reduce compile times.

We then bought 2 local machines, $1,800 each. These built considerably faster since:

They were always-online and didnt have to wait for a machine to spin up
They were able to have a GPU installed, which helped our smokescreens run faster (technically not required with -buildmachine but we saw significant improvements with one installed regardless)
We were able to exceed previous CPU and ram limits (now at 128gb whereas Google previously had a max limit of 96)

With them being always-online, we plugged em into an outlet that measured their power consumption. It runs us about $350 USD/month on power and internet. We recouped the up-front machine cost after about 8-9 months

muted flicker Oct 13, 2025, 1:08 PM

#

fiery hornet our scripts do kind of "reinvent" the wheel a bit, since it's doing a ton of thi...

I haven't used Horde yet, but we're also very interested in swapping away from our custom tooling to something more standardized

muted flicker Oct 13, 2025, 1:09 PM

#

fiery hornet our setup is not super complex really. we have a p4 trigger that calls some lamb...

What do your compile / upload times look like currently?

muted flicker Oct 14, 2025, 7:27 AM

#

@fiery hornet in case the thread didn't ping you either polishcow

fiery hornet Oct 14, 2025, 2:47 PM

#

muted flicker What do your compile / upload times look like currently?

a bit over 20 mins for packaging the client, and about the same for the server.

#

uploads time seem to be 2 minutes? our logs aren't very clear about that. it's all internal to AWS, and in the same region anyways, so it should be fast

muted flicker Oct 14, 2025, 2:50 PM

#

ahh thats a huge benefit of having it all on AWS

fiery hornet Oct 14, 2025, 2:50 PM

#

Client is 10GB, Server around 2

muted flicker Oct 14, 2025, 2:50 PM

#

Do you package executables for the development team also (Development Editor)?

fiery hornet Oct 14, 2025, 2:51 PM

#

The game is packaged in the Test config. We have some precompiled binaries for UGS too, only on code pushes. They take around 10 mins to build.

#

Which feels quite long for an editor rebuild 😅

muted flicker Oct 14, 2025, 2:53 PM

#

wow 10min is quite impressive for a rebuild

#

even on my development machine its around 20min for a rebuild. What are the machine specs?

fiery hornet Oct 14, 2025, 2:55 PM

#

muted flicker wow 10min is quite impressive for a rebuild

really? on another (albeit a bit smaller) project, I had editor rebuilds (with submits!) down to under 5 mins. And this one ran on my old Ryzen 5600/64GB RAM machine

fiery hornet Oct 14, 2025, 2:55 PM

#

muted flicker even on my development machine its around 20min for a rebuild. What are the mach...

tbh it's not a clean rebuild!

#

PCBs are incremental

fiery hornet Oct 14, 2025, 2:55 PM

#

muted flicker wow 10min is quite impressive for a rebuild

but the spec of the build servers we have on AWS is 48 cores/96GB RAM

muted flicker Oct 14, 2025, 2:58 PM

#

fiery hornet really? on another (albeit a bit smaller) project, I had editor rebuilds (with s...

Is that just project rebuild, or full engine rebuild w/ editor?

fiery hornet Oct 14, 2025, 2:59 PM

#

hmmm, could be just project rebuild, since we didn't modify the engine, it probably doesn't have to recompile it fully?

muted flicker Oct 14, 2025, 2:59 PM

#

ohhhh

fiery hornet Oct 14, 2025, 3:00 PM

#

yeah, that's why I'm thinking 10 mins for this project is quite long

muted flicker Oct 14, 2025, 3:00 PM

#

that makes sense. We make frequent engine edits on our end

#

When you run your build, do you know how many actions its building? It'll display something like this:
12>------ Building 2400 action(s) started ------

fiery hornet Oct 14, 2025, 3:02 PM

#

couldn't tell you for my current project, as the logs aren't easily accessible right now, but the other project was 415 actions

muted flicker Oct 14, 2025, 3:07 PM

#

Gotcha. Do you know if you're spending the most time compiling, cooking or packaging?

fiery hornet Oct 14, 2025, 8:51 PM

#

I asked the team that's been there for a longer time and will report back later!

fiery hornet Oct 20, 2025, 8:42 PM

#

So, after investigating our latest server (LinuxArm64) build log

Build:    272.93s / 4.55 min -- 23 actions (mostly our game's Runtime module it seems)
Cook:     790.99s / 13.18 min

#

In the cook log, I have some stuff about GEs being applied, and some from the PIE log category, which seems suspicious. This is happening during what seems to be part of the WP building or something.

#

I also found like

#

310GB of Logs that were related to WP, almost 12k of them

#

@muted flicker pinging to make sure you see the messages:)

#

(thanks for the help btw)

#

Here's what LogCookStats outputted

📎 cookstats.txt

muted flicker Oct 21, 2025, 2:17 PM

#

Oh that's a very long cook time. Ours is at most a couple of minutes

#

it looks like it spent the vast majority of that in cooking just 20 SkeletalMesh assets. That seems very disproportionate. @fiery hornet do you have any SkeletalMesh assets that are overly complex or doing anything crazy?

#

It also looks like from the log you might have zen servers set up. If you're on 5.6 you can take advantage of incremental cooks via the -cookincremental launch parameter, although results have been mixed it looks like

#

TickCookOnTheSidePrepareSaveTimeSec also seems pretty high. What are you using for storage?

#

ill see if I can get one of our machines to also output LogCookStats for comparison

fiery hornet Oct 21, 2025, 2:35 PM

#

we're still on 5.5, so we're specifically not doing incremental cooking yet

#

we'll be skipping 5.6 and waiting for 5.7.1 (or ideally 5.7.2)

fiery hornet Oct 21, 2025, 2:37 PM

#

muted flicker It also looks like from the log you might have zen servers set up. If you're on ...

we might have Zen for cooked output, it was something I was hoping to enable to help, but I did notice stuff in the logs and some Zen files too

fiery hornet Oct 21, 2025, 2:38 PM

#

muted flicker ``TickCookOnTheSidePrepareSaveTimeSec`` also seems pretty high. What are you usi...

it's using a 1.2TB EBS volume. I can check the throughput and IOPS in a couple of minutes

fiery hornet Oct 21, 2025, 2:40 PM

#

muted flicker ill see if I can get one of our machines to also output LogCookStats for compari...

it's usually outputted near the end of the cooking process i think

muted flicker Oct 21, 2025, 2:47 PM

#

yeah, we dont currently store the logs so ive gotta update the machines once they're done with their current job 😄

#

On our end it takes about 5min for both cooking and packaging combined, since we use the BuildCookRun batch

Starting packaging...
BuildCookRun -project="Game" -noP4 -platform=Win64 -clientconfig=Development -serverconfig="Development" -cook -AdditionalCookerOptions="-cookprocesscount=8" -stage -archive -nodebuginfo -archivedirectory="C:/SteamworksSDK/tools/ContentBuilder/content/Game/" -unrealexe="C:\Game\Binaries\Win64\GameEditor-Cmd.exe"
Packaging complete, took 315 seconds.

muted flicker Oct 21, 2025, 2:56 PM

#

fiery hornet it's using a 1.2TB EBS volume. I can check the throughput and IOPS in a couple o...

what volume type are you using? gp3?

fiery hornet Oct 21, 2025, 3:08 PM

#

muted flicker what volume type are you using? gp3?

yep,

#

I also noticed that for some reason, we are using a DebugGame editor executable

muted flicker Oct 21, 2025, 3:10 PM

#

oh that would for sure cause some slowdown hueh

#

Try with -unrealexe= so its explicit. We've also had issues in the past where it would incorrectly assume a different executable

fiery hornet Oct 21, 2025, 3:11 PM

#

it is with -unrealexe 🙃

#

the person who set it up before me must've set it incorrectly or something

#

    cmd = \
        'dotnet ' + \
        ubt_uat_bundle.ubt_exec_path + \
        ' -Target="GameNameEditor Win64 DebugGame"' + \
        ' -Target="ShaderCompileWorker Win64 Development -Quiet"' + \
        ' -WaitMutex'
    try:
        subprocess.check_call(
            cmd,
            shell=True,
            stderr=subprocess.DEVNULL,
            stdout=subprocess.DEVNULL,
            creationflags=subprocess.HIGH_PRIORITY_CLASS
        )

#

that's our editor compile command haha, it seems it gets compiled every time we package the game too, but I didn't think it would need to be done manually

muted flicker Oct 21, 2025, 3:17 PM

#

That's strange. On our team we compile Development (for Steam testing w/ console) and Shipping but not DebugGame, we only ever use that on local machines

fiery hornet Oct 21, 2025, 3:17 PM

#

yeah, that's what we did at my previous studio too

#

I'll have to test the changes locally first, but I think I should be able to compile Development instead of DebugGame without breaking anything

muted flicker Oct 21, 2025, 3:18 PM

#

yep

fiery hornet Oct 21, 2025, 3:19 PM

#

muted flicker it looks like it spent the vast majority of that in cooking just 20 SkeletalMesh...

about this, not sure, I'll ask around. I think we have some MetaHuman stuff, and I know we use Nanite for static meshes

muted flicker Oct 21, 2025, 3:20 PM

#

ive never messed with metahumans 🤷‍♂️ it could be they just take longer to cook

#

but from the log it seems disproportionate, even compared to static meshes which cooked fast

fiery hornet Oct 21, 2025, 3:24 PM

#

yeah for sure

#

it's still "just" 191s of the total 790s of cooking time, which tells me something else is taking a while

#

the CookStats say 595s

#

but the full log says this:

Execution of commandlet took:  9m 55s (595.80 seconds)
LogCore: Display: reporting session name in FinishDestroy():
LogOnline: Display: OSS: Removing 1 named interfaces
LogCore: Display: reporting session name in FinishDestroy():
LogCore: Display: reporting session name in FinishDestroy():
LogCore: Display: reporting session name in FinishDestroy():
LogShaderCompilers: Display: Shaders left to compile 0
LogShaderCompilers: Display: Shaders left to compile 0
LogStudioTelemetry: Display: Shutdown StudioTelemetry Module
Took 790.98s to run UnrealEditor-Win64-DebugGame-Cmd.exe, ExitCode=0

#

I wonder if the editor startup time is like 200s

#

oh yeah it's 100% the editor startup lmao

#

LogGameFeatures: Display: Game feature 'GameNameCore' transitioned successfully. Ending state: Active [Active, Active]
LogGameFeatures: Display: Total built in plugin load time 97.4926s
LogGameFeatures: Display: There were 1 built in plugins that took longer than 3.0000s to load. Listing worst offenders.
LogGameFeatures: Display:   97.2902s    GameNameCore

#

The Zen Server seems to be for the local DDC

#

from what I'm seeing in logs

#

not for cooked output

#

but I might be wrong

fiery hornet Oct 21, 2025, 3:45 PM

#

just double checked, and indeed, we are not using Zen Store for cooked output

#

which is weird, because there is a log about it

#

it's disabled in our project settings, and the project params that get printed say ZenStore=False, but I get 3 lines referencing things about it (e.g. LogZenStore: Display: Establishing oplog 'GameName.12345f/EditorDomain')

#

BuildCookRun -ScriptsForProject=c:/workspace/Projects/GameName/GameName.uproject -project=c:/workspace/Projects/GameName/GameName.uproject -unrealexe=c:/workspace/Engine/Binaries/Win64/UnrealEditor-Win64-DebugGame-Cmd.exe -archivedirectory=c:/workspace/Projects/GameName/Packaged -noP4 -utf8output -nocompileeditor -skipbuildeditor -cook -stage -archive -package -build -pak -iostore -compressed -prereqs -nocompile -nocompileuat -target=GameNameServer -platform=LinuxArm64 -server -serverconfig=Test -noclient -archivemetadata

#

and that's the commandline that gets executed for the packaging process

#

previous place I was at, we ran this to build:
BuildCookRun -project=C:\workspace\Games\GameName\GameName.uproject -build -configuration=Shipping -targetplatform=Win64 -target=GameNameSteam -cook -unversionedcookedcontent -stage -pak -compressed -prereqs -package -iostore -utf8output -buildmachine -unattended -noP4 -nosplash -stdout -NoCodeSign -CrashReporter

#

so I'm thinking I probably don't need to manually build the editor first, since that command line will compile it

muted flicker Oct 21, 2025, 3:56 PM

#

fiery hornet Here's what LogCookStats outputted

here's our output, pretty significant difference. We also dont have the SkeletalMesh spike

📎 message.txt

fiery hornet Oct 21, 2025, 3:57 PM

#

oh yeah interesting, there's no skeletal mesh being built at all it seems

muted flicker Oct 21, 2025, 3:57 PM

#

fiery hornet ``` LogGameFeatures: Display: Game feature 'GameNameCore' transitioned successfu...

ah xdd
This could be caused by having a lot of always-loaded assets

fiery hornet Oct 21, 2025, 3:57 PM

#

yeah I think we have some pretty gnarly dependencies. I briefly talked about it yesterday during a meeting

#

and was told we probably have a bunch of them

muted flicker Oct 21, 2025, 3:58 PM

#

gotcha

#

im noticing in general you have x10 the packages

#

maybe even more

#

yours

LogCookStats: Display: Package.Load
LogCookStats: Display:     NumRequestedLoads=6606
LogCookStats: Display:     NumPackagesLoaded=214441
LogCookStats: Display:     NumInlineLoads=207835
LogCookStats: Display:     LoadPackageTimeSec=298.445320
LogCookStats: Display: Package.Save
LogCookStats: Display:     NumPackagesIterativelySkipped=0
LogCookStats: Display:     NumPackagesSaved=17315
LogCookStats: Display:     SavePackageTimeSec=66.691399

ours

LogCookStats: Display: [CookWorker 2]: Package.Load
LogCookStats: Display: [CookWorker 2]:     NumRequestedLoads=1768
LogCookStats: Display: [CookWorker 2]:     NumPackagesLoaded=5878
LogCookStats: Display: [CookWorker 2]:     NumInlineLoads=4110
LogCookStats: Display: [CookWorker 2]:     LoadPackageTimeSec=36.540159
LogCookStats: Display: [CookWorker 2]: Package.Save
LogCookStats: Display: [CookWorker 2]:     NumPackagesIterativelySkipped=0
LogCookStats: Display: [CookWorker 2]:     NumPackagesSaved=1826
LogCookStats: Display: [CookWorker 2]:     SavePackageTimeSec=4.549163

#

214k packages seems very excessive haha, is that number accurate?

#

You may be cooking non-gameplay assets, I would recommend checking your "maps to cook" list

#

You're also not using cookworkers, try adding -AdditionalCookerOptions="-cookprocesscount=8" to your launch parameters

fiery hornet Oct 21, 2025, 4:12 PM

#

I think our game started based on Lyra, so that probably inflates some numbers. I was told that we were using the MapsToCook list, but...

CookAll=False
CookPartialGC=False
CookInEditor=False
CookMapsOnly=False
Deploy=False
IterativeDeploy=False
FastCook=False
LogWindow=False
Manifests=False
MapToRun=
NoClient=True
NumClients=0
NoDebugInfo=False
SeparateDebugInfo=False
MapFile=False
NoCleanStage=False
NoXGE=False
MapsToCook=
MapIniSectionsToCook=
Pak=True
IgnorePaksFromDifferentCookSource=False
IoStore=True
SkipIoStore=False
ZenStore=False

muted flicker Oct 21, 2025, 4:16 PM

#

hueh

fiery hornet Oct 21, 2025, 4:17 PM

#

unless I am missing something else in the logs haha

muted flicker Oct 21, 2025, 4:18 PM

#

Try adding just a few known maps to the MapsToCook list (startup + titlescreen + lobby + gameplay) and see if the game will run

fiery hornet Oct 21, 2025, 4:18 PM

#

ah, we have some maps specified in the DefaultGame.ini

#

some of them seem to be non-gameplay though

#

I assume that's why it isn't showing up in the list there

muted flicker Oct 21, 2025, 4:20 PM

#

How'd you get the above list?

fiery hornet Oct 21, 2025, 4:22 PM

#

it gets printed at the start of the log

#

right under Project Params **************

#

it does say "Loading ini files for GameName.uproject" before, but I just looked in the DefaultGame.ini and we do have maps specified

muted flicker Oct 21, 2025, 4:24 PM

#

during cook or when launching editor?

fiery hornet Oct 21, 2025, 4:24 PM

#

before the build command even

muted flicker Oct 21, 2025, 4:27 PM

#

Gotcha, I dont see that on my end but I might just be looking in the wrong place. I assume if the engine is printing that the value is empty, its probably empty. What header is your MapsToCook entries under in DefaultGame.ini, and do you see those values in Project Settings in editor?

fiery hornet Oct 21, 2025, 4:28 PM

#

under /Script/UnrealEd.ProjectPackagingSettings, and yeah they are in editor too

#

some of them have the prefix "TestMap_" so pretty sure they aren't needed for gameplay haha

#

I'll have to verify with the team

#

we also have some suspicious stuff in DirectoriesToAlwaysCook

muted flicker Oct 21, 2025, 4:31 PM

#

ahh good catch

#

We dont use DirectoriesToAlwaysCook on our end

fiery hornet Oct 21, 2025, 4:31 PM

#

fiery hornet 310GB of Logs that were related to WP, almost 12k of them

by the way, would you happen to know what the logs in Saved\Logs\WorldPartition are for?

muted flicker Oct 21, 2025, 4:33 PM

#

We dont use world partition on our end, but I know its Unreal's new "open world" system. Does your team use world partition levels?

#

in general we've been very conservative with new features, we've stayed clear of lumen, nanite and world partition until all the growing pains are sorted out nod

fiery hornet Oct 21, 2025, 4:36 PM

#

yeah we use WorldPartition a lot since our levels are quite large

muted flicker Oct 21, 2025, 4:37 PM

#

makes sense, not sure I can help much with that, but it seems unrelated to the long cook times

fiery hornet Oct 21, 2025, 4:39 PM

#

yeah it's probably not related, but it did fill up our drives with 310GB worth of logs hahaha

muted flicker Oct 21, 2025, 4:40 PM

#

xdd

fiery hornet Oct 21, 2025, 4:42 PM

#

Our C++ compile times also seem weirdly slow to me

#

------ Building 23 action(s) started ------
** For GameNameServer-LinuxArm64-Test **
[1/23] Compile Module.GameNameCoreRuntime.14.cpp
...
[22/23] Link (lld) GameNameServer-LinuxArm64-Test
        1 file(s) moved.

[23/23] WriteMetadata GameNameServer-LinuxArm64-Test.target (UBA disabled)
Trace file written to C:/workspace/Engine/Programs/AutomationTool/Saved/Logs/UBA-UnrealPak-Win64-Development.uba with size 20.7kb
Total time in Unreal Build Accelerator local executor: 251.96 seconds
Total execution time: 272.72 seconds

#

seems to me that over 4.5 mins for 23 actions is quite slow

#

for a 24 cores / 48 threads machine with 96GB of RAM

muted flicker Oct 21, 2025, 4:52 PM

#

lemme compare with our builds

#

yeah thats pretty slow

#

1>------ Building 51 action(s) started ------
1>** For GameEditor-Win64-Development **
1>[1/51] Compile [x64] GameComponent.cpp
1>[47/51] Link [x64] GameEditor-Game.lib
1>[48/51] Link [x64] GameEditor-GameEditor.dll
1>   Creating object G:\Unreal Engine\Game\Game\Binaries\Win64\GameEditor-GameEditor.exp
1>[49/51] Link [x64] GameEditor-Game.dll
1>   Creating object G:\Unreal Engine\Game\Game\Binaries\Win64\GameEditor-Game.exp
1>[50/51] WriteMetadata GameEditor.target (UBA disabled)
1>[51/51] cmd Executing post build script (PostBuild-1.bat) (UBA disabled)
1>Sentry: Start debug symbols upload
1>Sentry: Automatic symbols upload is not required for Editor target. Skipping...
1>Trace file written to G:/Unreal Engine/Game/Engine/Programs/UnrealBuildTool/Log.uba with size 15.7kb
1>Total time in Unreal Build Accelerator local executor: 44.47 seconds
1>Total execution time: 49.73 seconds
1>Done building project "Game.vcxproj".
========== Build: 1 succeeded, 0 failed, 11 up-to-date, 0 skipped ==========
========== Build completed at 18.54 and took 50,553 seconds ==========

#

Are you able to compile on your local machine? Are you seeing similar compile times?

#

Also what does your Engine/Saved/UnrealBuildTool/BuildConfiguration.xml look like?

fiery hornet Oct 21, 2025, 5:06 PM

#

muted flicker Are you able to compile on your local machine? Are you seeing similar compile ti...

much faster on my machine, which has less cores (16/32). not for the same target (I was compiling the standalone here instead of the server), but it also compiled the editor, so in total I had like 75 actions, and it took just a little over 142s (76s of which were linking the executable)

#

I have a Ryzen 9 7950X

#

not sure what the c7i.12xlarge instances have

fiery hornet Oct 21, 2025, 5:08 PM

#

fiery hornet ``` LogGameFeatures: Display: Game feature 'GameNameCore' transitioned successfu...

also, this takes 42-45s on my machine, both in Development and DebugGame configuration

#

so there's definitely something odd with our build machine specs

muted flicker Oct 21, 2025, 5:09 PM

#

in general we also saw reduced performance with remote machines, but it might be something that can be resolved

fiery hornet Oct 21, 2025, 5:10 PM

#

muted flicker Also what does your ``Engine/Saved/UnrealBuildTool/BuildConfiguration.xml`` look...

empty on my machine, probably the same on the build machines

muted flicker Oct 21, 2025, 5:10 PM

#

Do you think it could be bottlenecked by RAM?

fiery hornet Oct 21, 2025, 5:10 PM

#

maybe if it's really slow, but they have more RAM than my PC. I have 64GB and the build machines have 96

muted flicker Oct 21, 2025, 5:10 PM

#

Gotcha

fiery hornet Oct 21, 2025, 5:12 PM

#

I think my Ryzen 5600 machine might be faster lol. A build with 9-10 actions on it took 67s for the compilation (not the same game though)

muted flicker Oct 21, 2025, 5:13 PM

#

We've sometimes seen that UBT will start more processes than the machine can reasonably facilitate, causing it all to slow down overall. But im not sure thats whats happening here. You could try creating this BuildConfiguration.xml and see if that has any impact:

<?xml version="1.0" encoding="utf-8" ?>
<Configuration xmlns="https://www.unrealengine.com/BuildConfiguration">
  <TaskExecutor>
    <ProcessorCountMultiplier>0.5</ProcessorCountMultiplier>
    <MaxProcessorCount>32</MaxProcessorCount>
  </TaskExecutor>
  <ParallelExecutor>
    <ProcessorCountMultiplier>0.5</ProcessorCountMultiplier>
    <MaxProcessorCount>32</MaxProcessorCount>
  </ParallelExecutor>
</Configuration>

Basically limiting it to only spawn one processor per two cores.

muted flicker Oct 21, 2025, 5:13 PM

#

fiery hornet I think my Ryzen 5600 machine might be faster lol. A build with 9-10 actions on ...

yeahhh we also maxed out what was offered by Google Cloud when we used it lol

#

and even then it didnt compare to our local machine hardware

fiery hornet Oct 21, 2025, 5:13 PM

#

muted flicker We've sometimes seen that UBT will start more processes than the machine can rea...

Determining max actions to execute in parallel (24 physical cores, 48 logical cores)
  Executing up to 24 processes, one per physical core
Using Unreal Build Accelerator local executor to run 23 action(s)

#

pretty sure it already does that haha

#

unless you mean using 12 processes

muted flicker Oct 21, 2025, 5:14 PM

#

12 yeah

fiery hornet Oct 21, 2025, 5:15 PM

#

fiery hornet much faster on my machine, which has less cores (16/32). not for the same target...

just checked my own log, and I was using a 1.5 count multiplier here, interestingly, so my 16 cores was running 24 processes

muted flicker Oct 21, 2025, 5:15 PM

#

ahh gotcha

fiery hornet Oct 21, 2025, 5:15 PM

#

I think that's my local UBT config

muted flicker Oct 21, 2025, 5:15 PM

#

we used BuildConfiguration.xml heavily in UE4 but it hasnt really been a problem in UE5

fiery hornet Oct 21, 2025, 5:16 PM

#

I have a custom BuildConfiguration.xml in my appdata that I forgot about

<?xml version="1.0" encoding="utf-8" ?>
<Configuration xmlns="https://www.unrealengine.com/BuildConfiguration">
    <BuildConfiguration>
        <!--<bUseAdaptiveUnityBuild>true</bUseAdaptiveUnityBuild>
        <bAdaptiveUnityDisablesOptimizations>true</bAdaptiveUnityDisablesOptimizations>
        <bAdaptiveUnityDisablesPCH>true</bAdaptiveUnityDisablesPCH>-->
        <bAllowUBALocalExecutor>false</bAllowUBALocalExecutor>
    </BuildConfiguration>
    <ParallelExecutor>
        <!-- (reserve some cores for my use during builds) -->
        <!-- 1.5 * 16 = 24 of 32 == 8 free logical cores on my Ryzen 9 7950X (16 physical, 32 logical cores) -->
        <ProcessorCountMultiplier>1.5</ProcessorCountMultiplier>
    </ParallelExecutor>
</Configuration>

#

the UBA local executor is broken by windows right now when cross-compiling

muted flicker Oct 21, 2025, 5:16 PM

#

ah lol

#

Are you able to remote in and check CPU / RAM usage during compile on the EC2 machine?

fiery hornet Oct 21, 2025, 5:17 PM

#

yeah, I'll wait until the next check-in, because triggering a build manually is not easy

#

Horde will be a nice addition/replacement

muted flicker Oct 21, 2025, 5:18 PM

#

i really wanna swap to horde also

#

no Git support though, only P4 pensivecowboyratbutt

fiery hornet Oct 21, 2025, 5:19 PM

#

I was using TeamCity at my previous job, and it's really nice, but here we use a lot of what Epic provides, so I'm keeping the same philosophy here. It also has some really nice features that TC lacks

fiery hornet Oct 21, 2025, 5:19 PM

#

muted flicker no Git support though, only P4 <:pensivecowboyratbutt:810012229189632031>

yeah, but at least there are some alternatives coming up. I talked with some folks from Diversion and they have Horde and UGS support in testing

Diversion

Diversion: Best Version Control for UE | Perforce Alternative

Diversion: The best version control for Unreal Engine, Game Development, Virtual Production and 3D Visualization. A scalable, easy alternative to Perforce, Git and SVN for large files, UE, Unity, Godot, Blender, Twinmotion, 3ds Max, Maya, Houdini, SketchUp, Substance, ZBrush etc.

#

They're really responsive on Discord too, which is nice

muted flicker Oct 21, 2025, 5:21 PM

#

oh nice, we looked at options at the start of the project and there weren't many. I think Plastic SCM was the closest to "Git for game developers" until they got bought by Unity

#

ive not heard of Diversion, gotta check it out

fiery hornet Oct 21, 2025, 5:21 PM

#

yeah they are fairly new

#

heard about it from the recent UE game jam, but I may have stumbled upon it before too. It's a really user-friendly software and it's cheaper than P4

#

no self-hosting yet (other than enterprise), but they are planning on going open-source eventually

muted flicker Oct 21, 2025, 5:22 PM

#

yeah the "no servers" part of their pitch trailer seems like a red flag hueh so hopefully they offer self-hosting soon

#

I had to use P4 during the Unreal 3 days and its scarred me ever since, our artists hated it so much. not to mention the cost

fiery hornet Oct 21, 2025, 5:23 PM

#

they really pride themselves on the ease of use, and there's nothing easier than just signing up and starting to use it :P

muted flicker Oct 21, 2025, 5:23 PM

#

yeah true lol

fiery hornet Oct 21, 2025, 5:24 PM

#

muted flicker I had to use P4 during the Unreal 3 days and its scarred me ever since, our arti...

I'm too young to know that, but I've heard it's gotten much better hahaha

#

the pricing hasn't though

muted flicker Oct 21, 2025, 5:24 PM

#

xdd

#

what does your team use for VC then?

fiery hornet Oct 21, 2025, 5:24 PM

#

P4 haha

muted flicker Oct 21, 2025, 5:24 PM

#

ahh gotcha lol

fiery hornet Oct 21, 2025, 5:25 PM

#

at my previous studio too, and that's also what I used in university for game dev

#

I use Git for other personal projects, but Unreal stays in P4 land for me hahaha

muted flicker Oct 21, 2025, 5:25 PM

#

makes sense, Epic is also pushing for it hard

fiery hornet Oct 21, 2025, 5:26 PM

#

I understand why too. Git isn't the greatest at dealing with large files, though it's better than before

muted flicker Oct 21, 2025, 5:27 PM

#

yeah you need Git LFS to make it work, and the out-of-the-box Git plugin isnt the greatest

fiery hornet Oct 21, 2025, 5:27 PM

#

I've learned how to wrangle P4 server management despite the documentation being... not great

#

my biggest annoyance is them providing a simple CLI (p4ctl) for Ubuntu, but not for Debian, and I didn't know that at first, so I created systemd units manually on Debian

#

on the bright side, I know more about how it works now hahaha

muted flicker Oct 21, 2025, 5:28 PM

#

LMAO

fiery hornet Oct 21, 2025, 5:28 PM

#

the worst part is that it works very well on Debian, so I can only assume it's laziness to not provide the packages

muted flicker Oct 21, 2025, 5:29 PM

#

yeah they probably just want to limit their releases

#

gotta love poorly documented traps

fiery hornet Oct 21, 2025, 5:30 PM

#

Some stuff is in their documentation, and other things are better found in their helpdesk Q&As, that can't really be searched and can only be found with pain by using the correct keywords on Google/DuckDuckGo/whatever

#

I found a cool project today though for setting up a P4 server: https://github.com/alefriant/Alehundred-VCS-Toolkit/tree/EnglishVersion9.2

GitHub

GitHub - alefriant/Alehundred-VCS-Toolkit at EnglishVersion9.2

A powerful, open-source toolkit to simplify the setup and management of a professional, self-hosted version control system on low-cost ARM hardware. - GitHub - alefriant/Alehundred-VCS-Toolkit at E...

#

anyways, I've setup 3-4 P4 servers, and migrated 2 of them, so I have a fairly good understanding of them now. I won't be migrating our current one for a while though, if ever. It's not a great experience

#

It's also not super expensive despite being on AWS

muted flicker Oct 21, 2025, 5:33 PM

#

yeah makes sense, migrating VC is usually a next-project kinda thing

muted flicker Oct 21, 2025, 5:34 PM

#

fiery hornet I found a cool project today though for setting up a P4 server: https://github.c...

i have never considered hosting VC on a raspberry pie before, that's quite interesting lol

fiery hornet Oct 21, 2025, 5:36 PM

#

the recent ones are really powerful too, so it's more than good enough haha

#

I have a homelab for most of my things though

muted flicker Oct 21, 2025, 5:36 PM

#

thats sick

#

i wanted to get into raspberry pie during covid, but obviously everyone else had the same idea so they ran out almost immediately hueh

#

you sound very experienced when it comes to sys admin work, is that your role within your team?

fiery hornet Oct 21, 2025, 5:53 PM

#

I got brought in as a DevOps Engineer, so close enough. But I mostly know a lot from messing around with my own machines, as well as a previous job where I helped with web servers management

#

I surprisingly don't have a lot of professional experience :P

#

just a lot of "special interests" hahahaha

fiery hornet Oct 21, 2025, 5:55 PM

#

muted flicker ah <:xdd:1387162685770174676> This could be caused by having a lot of always-lo...

the more I think about it, the more this makes sense, because I remember the first time I opened the editor it took over 10 minutes and peaked at 42-43GB of RAM usage lol

muted flicker Oct 21, 2025, 6:19 PM

#

fiery hornet I surprisingly don't have a lot of professional experience :P

thats mostly everyone in gamedev hueh we're all just winging it

muted flicker Oct 21, 2025, 6:19 PM

#

fiery hornet the more I think about it, the more this makes sense, because I remember the fir...

part of that might also be first-time shader compiling so its hard to tell. if the editor takes a long time to boot consistently though (including a significant stutter when starting PIE) then its probably always-loaded content

#

though i do think your whack cook times is likely due to cooking unnecessary assets

#

let me know once you have the data on CPU / RAM utilization, slow compiles was our main motivator for moving to local machines, so im interested in seeing if that pain point can be resolved

fiery hornet Oct 21, 2025, 6:25 PM

#

muted flicker part of that might also be first-time shader compiling so its hard to tell. if t...

yeah there definitely was some shader compilation happening, but I don't think it was all that

fiery hornet Oct 21, 2025, 6:26 PM

#

muted flicker let me know once you have the data on CPU / RAM utilization, slow compiles was o...

it would be very nice to resolve it. we have a bunch of AWS credits from what I've been told, so it's advantageous for us to stay there

#

have you tried octobuild for C++ compilation?

#

https://github.com/octobuild/octobuild

GitHub

GitHub - octobuild/octobuild: Compiler cache for Unreal Engine

Compiler cache for Unreal Engine. Contribute to octobuild/octobuild development by creating an account on GitHub.

muted flicker Oct 21, 2025, 6:27 PM

#

a reduction of 60-90% sounds insane

#

this sounds almost too good to be true LMAO, there's gotta be a catch

#

I will definitely try it though

#

how did you find it?

fiery hornet Oct 21, 2025, 6:28 PM

#

https://aws-games.github.io/cloud-game-development-toolkit/v1.1.5/assets/packer/build-agents/linux/index.html

Packer templates for Unreal Engine Linux build agents - Cloud Game ...

Cloud Game Development Toolkit | Docs

#

while looking into that toolkit, since I'll probably be planning our new build pipeline with it as a base

muted flicker Oct 21, 2025, 6:30 PM

#

good find, have you tried it yet?

#

octobuild I mean

fiery hornet Oct 21, 2025, 6:31 PM

#

not yet, been mostly planning, researching and getting used to the new project in the past weeks. I'll be spinning up a testing build machine later this week to try it as well as zen store and a bunch of other things to accelerate our build pipeline

muted flicker Oct 21, 2025, 6:34 PM

#

zen store should help your cook times since it eliminates cooking unchanged assets. since our cook times are so low its not really a priority (im lazy)

#

i will definitely try octobuild though and run some measurements

fiery hornet Oct 21, 2025, 6:36 PM

#

muted flicker zen store should help your cook times since it eliminates cooking unchanged asse...

only if incremental cook is enabled right? we won't be using it since it's apparently not super stable in 5.5

#

zen might help anyways by reducing filesystem overhead though

muted flicker Oct 21, 2025, 6:40 PM

#

oh right you're on 5.5

#

and yeah true. Although you should probably reduce the number of packages anyway hueh

#

also im interested, what do you guys do for shipping executables to developers? package to S3 with an app for devs to download? or have them all compile locally

fiery hornet Oct 21, 2025, 6:49 PM

#

we use UnrealGameSync (P4 only again haha)

#

the engineering team compiles locally, but the rest of the team uses precompiled binaries with UGS

#

they are stored in P4 for now, but will be in a bucket once Horde is setup

#

it's a really great tool

muted flicker Oct 21, 2025, 6:50 PM

#

ahh makes sense

#

we ended up having to roll our own version of UGS, effectively works the same, just sends binaries to S3 that devs then download

fiery hornet Oct 21, 2025, 6:52 PM

#

yeah, it depends pretty heavily on P4 by default

#

did you just plug Git support in UGS?

muted flicker Oct 21, 2025, 6:53 PM

#

nah we made our own app that works effectively the same way, including the version files

#

we have too many custom tools hueh thats why im interested in Horde to bring at least some standardization to our toolset

#

honestly next project we might bite the bullet and just go for P4 if all tools are still heavily P4-reliant

fiery hornet Oct 21, 2025, 6:56 PM

#

Horde is easy to extend which is really nice too

#

the whole build pipeline aspect of it is a plugin

muted flicker Oct 21, 2025, 7:00 PM

#

nice, honestly git and p4 arent that different, if the p4 parts are isolated and its just a matter of swapping commands, then it might be feasible

fiery hornet Oct 21, 2025, 7:02 PM

#

Horde works a lot with p4's concept of streams, which are kind of like branches, but also not really

#

but it's definitely much easier to adapt than UGS would be

muted flicker Oct 21, 2025, 7:30 PM

#

yeah, i only see about 270 mentions of Perforce in the horde repo, which isn't too bad

#

Diversion also has their own guide for working with Horde, so presumably the same approach they took could also be applied to Git
https://docs.diversion.dev/unreal/horde_setup

Diversion Documentation

Set up Horde with Diversion (Windows, UE5 Source) - Diversion Docum...

Quickstart for wiring Diversion SCM to Horde and preparing a UE5 source repository for Horde CI.

#

unless Diversion is just a P4 frontend hueh

fiery hornet Oct 21, 2025, 7:31 PM

#

nah it's all custom for sure hahaha

#

it's free to try too

#

I think it has some similarities to both P4 and Git

muted flicker Oct 21, 2025, 7:32 PM

#

that might be a good way to experiment with horde without needing P4 then

fiery hornet Oct 21, 2025, 7:32 PM

#

branches like git, but better binary file support like P4

#

also not storing the whole project on the PC, similar to how P4 does it

muted flicker Oct 21, 2025, 7:35 PM

#

nice

#

ill have to give it a try over the weekend when ive got spare time

fiery hornet Oct 21, 2025, 7:37 PM

#

yeah I need to give it a better try too

#

I suggested it to some friends for a game jam though, and so far haven't heard any cries for help, which tells me it must be pretty easy to use

muted flicker Oct 21, 2025, 7:38 PM

#

LMAO thats metal, learning all new VC for a limited-time game jam

fiery hornet Oct 21, 2025, 7:39 PM

#

going back to my cooking log for a moment, would you happen to know a way to print/view what assets are being cooked? we're trying to see what skeletal meshes are getting cooked

fiery hornet Oct 21, 2025, 7:39 PM

#

muted flicker LMAO thats metal, learning all new VC for a limited-time game jam

yeah hahahaha

#

it's similar enough to P4 in usage that it must've been pretty easy

muted flicker Oct 21, 2025, 7:39 PM

#

gooood question, lemme cross-reference my log

#

its probably not a comprehensive list, but im seeing a lot of
LogSkeletalMesh: Display: [CookWorker 0]: Waiting for skinned assets to be ready 0/1 (SKM_Dog_01) ...
which might provide a few meshes that are relevant

#

lemme check the source and see if they have any command that verbosely outputs everything

fiery hornet Oct 21, 2025, 7:41 PM

#

I'm seeing this too, but it definitely doesn't log 20 of them

fiery hornet Oct 21, 2025, 7:43 PM

#

fiery hornet yeah hahahaha

I also heard about Anchorpoint, which is technically just a UI for Git https://www.linkedin.com/company/anchorpoint-software/posts/
Which seems to have some nice QoL features in their tool.

muted flicker Oct 21, 2025, 7:43 PM

#

looks like if you set LogCook to verbose, you'll get this:
UE_LOG(LogCook, Verbose, TEXT("Enqueing cook request, Filename='%s', Platform='%s'")

fiery hornet Oct 21, 2025, 7:44 PM

#

ah neat!

muted flicker Oct 21, 2025, 7:44 PM

#

fiery hornet I also heard about [Anchorpoint](https://www.anchorpoint.app/), which is technic...

oh I saw this in a reddit thread also actually!

fiery hornet Oct 21, 2025, 7:44 PM

#

I never remember the flag to change the log verbosity

muted flicker Oct 21, 2025, 7:44 PM

#

yeah its a bit obscure, there's both a .ini and launch command

#

for .ini, add the following to DefaultEngine.ini

[Core.Log]
LogCook=verbose

fiery hornet Oct 21, 2025, 7:45 PM

#

of course there is haha

muted flicker Oct 21, 2025, 7:46 PM

#

muted flicker oh I saw this in a reddit thread also actually!

we use Fork since it was the first Git client that our artists could actually understand (plus its free 🙂 ), but since Anchorpoint has a trial I can run it for a bit and see if its worth paying the subscription

fiery hornet Oct 21, 2025, 7:46 PM

#

yeah I heard good stuff about Fork!

#

I used GitKraken and Github Desktop a bit in the past too, but never really got used to UIs enough

#

I do like lazygit as a TUI

muted flicker Oct 21, 2025, 7:47 PM

#

GitKraken used to be the Git client back in the day

fiery hornet Oct 21, 2025, 7:48 PM

#

muted flicker On our end it takes about 5min for both cooking and packaging combined, since we...

I noticed you have both -clientconfig and -serverconfig flags. Do you package both the client and the server at the same time? did you split the build step and cooking step into 2 parts for that?

muted flicker Oct 21, 2025, 7:48 PM

#

lazygit looks, uh, interesting hueh

muted flicker Oct 21, 2025, 7:49 PM

#

fiery hornet I noticed you have both -clientconfig and -serverconfig flags. Do you package bo...

both at the same time yeah! only requires one cook.

#

we run our compile seperately from cook, since we need to upload the executables to S3

fiery hornet Oct 21, 2025, 7:49 PM

#

interesting, didn't know that could be done! probably more efficient that way

muted flicker Oct 21, 2025, 7:49 PM

#

but after the compile we just run BuildCookRun to cook and package

fiery hornet Oct 21, 2025, 7:50 PM

#

So BuildCookRun with build only 2 times, then BuildCookRun for cook and package for both targets at once?

#

oh I think we wouldn't be able to do that on our side actually, since our servers are LinuxArm64, clients are Win64

muted flicker Oct 21, 2025, 7:52 PM

#

almost! instead of BuildCookRun for build, we run BuildUBT.bat -> BuildUAT.bat -> GenerateProjectFiles.bat -> Build.bat

#

we could probably run BuildCookRun, I forgot why we didnt lol

muted flicker Oct 21, 2025, 7:52 PM

#

fiery hornet oh I think we wouldn't be able to do that on our side actually, since our server...

im not sure that matters for cooking?

fiery hornet Oct 21, 2025, 7:53 PM

#

pretty sure Cooking is platform specific no?

#

in any case, I'll be testing it eventually

muted flicker Oct 21, 2025, 7:53 PM

#

Hmm

#

It might be, im unsure

fiery hornet Oct 21, 2025, 7:54 PM

#

I think I tried just plopping a Windows pak file in a Linux build before and it didn't work

muted flicker Oct 21, 2025, 7:55 PM

#

oh pak might be different yeah

#

but that's part of the package process, not cook iirc

fiery hornet Oct 21, 2025, 7:55 PM

#

hmmm yeah maybe actually

muted flicker Oct 21, 2025, 7:56 PM

#

I do see that Game/Saved/Cooked/ has per-platform subfolders (like /Windows/) so you're probably right

fiery hornet Oct 21, 2025, 8:05 PM

#

You've mentioned console builds right? How hard are they to setup? Is it mostly a new compiler and some cook differences?

muted flicker Oct 21, 2025, 8:07 PM

#

oh ive only done console builds for UE3. When I mentioned console it was the in-game console thats available in Development builds

fiery hornet Oct 21, 2025, 8:56 PM

#

ahhh right yeah!

fiery hornet Oct 22, 2025, 12:16 AM

#

fiery hornet yeah, I'll wait until the next check-in, because triggering a build manually is ...

we do seem to be using a most of the cores, when compiling the editor at least

#

it seems to be sitting idle after 2-3 minutes

#

still running the BuildEditorAndTools.xml graph in the background though, maybe one of the step is really slow

#

one core is used more than the others

#

#

I can't look at it for the whole build right now, but I think single-core performance might just be bad on those machines

#

because the compilation parts are relatively fast, but the linking is very slow

#

would explain the slower performance than my CPU despite having more cores

fiery hornet Oct 22, 2025, 5:17 PM

#

muted flicker looks like if you set LogCook to verbose, you'll get this: ``UE_LOG(LogCook, Ver...

turns out this didn't get printed for some reason, but I found GCookProgressDisplay, which can be set by a cvar, so this will probably work, testing right now

muted flicker Oct 22, 2025, 5:26 PM

#

ah sorry didnt see your new messages until now

muted flicker Oct 22, 2025, 5:27 PM

#

fiery hornet we do seem to be using a most of the cores, when compiling the editor at least

this looks pretty solid. Like you said, it might be a single-core performance issue during linking

fiery hornet Oct 22, 2025, 5:27 PM

#

that 2.4GHz is pretty low hahaha

muted flicker Oct 22, 2025, 5:28 PM

#

yeahhh

#

here's what we got on our local machine

fiery hornet Oct 22, 2025, 5:29 PM

#

I assume core usage is only that high when compiling right?

#

cooking is probably less cores

muted flicker Oct 22, 2025, 5:30 PM

#

correct yeah its in the middle of a compile

#

lemme check if we have one thats cooking rn

fiery hornet Oct 22, 2025, 5:30 PM

#

found out there's some new AWS instances that are a lot more powerful for the single core perf

#

https://runs-on.com/benchmarks/aws-ec2-instances/m8a.large/

RunsOn

Benchmarks for m8a.large (AWS)

CPU benchmark for the m8a.large EC2 instance type, as well as the availability, pricing, and spot interruption percentage in various regions.

#

new from this month

#

I'll be testing them eventually probably

fiery hornet Oct 22, 2025, 5:34 PM

#

fiery hornet turns out this didn't get printed for some reason, but I found `GCookProgressDis...

ok yeah, this prints a TON more info

[/Script/UnrealEd.CookerSettings]
cook.displaymode=7

muted flicker Oct 22, 2025, 5:34 PM

#

oh nice, good catch

#

we dont have a machine thats cooking rn but ive put it in the pipeline and will give stats once its going

#

Were you able to identify why there's 200k packages that require cooking?

#

I was thinking about it and I wonder if you have One File Per Actor enabled in world settings for your levels?

fiery hornet Oct 22, 2025, 5:35 PM

#

not yet, but I suspect that some of the maps and always-cook directories aren't needed

#

yeah, IIRC it's necessary for WorldPartition

#

or at least better

#

that might explain the packages actually

muted flicker Oct 22, 2025, 5:35 PM

#

ohhh ok, that might be the source then, and it could be valid.

fiery hornet Oct 22, 2025, 5:36 PM

#

we need to get rid of our "Asset has been saved with empty engine version." warnings, they are polluting the logs so much right now

#

we think they're from a pre-UGS workflow where the engine wasn't versioned correctly

muted flicker Oct 22, 2025, 5:36 PM

#

oh thats an interesting warning, ive not seen that before

#

gotcha

fiery hornet Oct 22, 2025, 5:37 PM

#

mhm, when you don't have a Build.version file it does that

muted flicker Oct 22, 2025, 5:37 PM

#

fiery hornet https://runs-on.com/benchmarks/aws-ec2-instances/m8a.large/

this sounds promising, definitely share results once you have them. You could probably just create a snapshot of an existing volume for quick setup

fiery hornet Oct 22, 2025, 5:37 PM

#

fiery hornet mhm, when you don't have a Build.version file it does that

https://dev.epicgames.com/documentation/en-us/unreal-engine/versioning-of-assets-and-packages-in-unreal-engine

Epic Games Developer

Versioning of Assets and Packages in Unreal Engine | Unreal Engine ...

Using customized serialization code and versioning to control how Objects are loaded from Assets and Packages in Unreal Engine.

fiery hornet Oct 22, 2025, 5:39 PM

#

muted flicker this sounds promising, definitely share results once you have them. You could pr...

yeah, I'll probably do that in the coming weeks, have a bunch of other things to do first. I think we might even be able to go to a lower core count and still get better speeds than we do now, just because of the better single-core perf

fiery hornet Oct 22, 2025, 5:48 PM

#

muted flicker Were you able to identify why there's 200k packages that require cooking?

I think we don't actually cook 200k packages

#

once the cook is done, it prints "Packages Cooked: 16960"

#

so I think maybe the 200k is actors being loaded, but not individually cooked?

#

also, grep is so fast, just plucked all 44k lines of LogCook from the 342k line cook log instantly

fiery hornet Oct 22, 2025, 7:36 PM

#

muted flicker i will definitely try octobuild though and run some measurements

just so you know, I am pretty sure it isn't compatible with UBA, if you are using it. It masks as an Incredibuild service so it can be automatically be picked up by UE
https://github.com/octobuild/octobuild/issues/135
https://github.com/EpicGames/UnrealEngine/blob/5.5.4-release/Engine/Source/Programs/UnrealBuildTool/Configuration/BuildConfiguration.cs#L47

GitHub

Octobuild compatibility with UBA · Issue #135 · octobuild/octobuild

Hi everyone, Just checked out this awesome project and stepped into an issue with my UBA not working while Octabuild(labeld as Incredibuild, is this a mistake?) is active. Is this expected behavior...

muted flicker Oct 23, 2025, 5:21 PM

#

fiery hornet just so you know, I am pretty sure it isn't compatible with UBA, if you are usin...

ahh good to know!

muted flicker Oct 23, 2025, 5:22 PM

#

fiery hornet once the cook is done, it prints "Packages Cooked: 16960"

17k sounds way more reasonable. Were the 20 SkeletalMeshes consistently showing up under DDC resource stats, or was it just a one-time occurance?

fiery hornet Oct 23, 2025, 5:24 PM

#

oh we 100% are cooking SKM when we don't need it. a lot of the "Instigators" were StartupPackage, so it sounds like some classes are loaded at engine startup and the dependency tree ends up reaching some SKM

muted flicker Oct 24, 2025, 4:41 PM

#

I got some stats from cooking, looks like we're also not seeing full utilization so that is mostly expected

fiery hornet Oct 24, 2025, 5:39 PM

#

muted flicker I got some stats from cooking, looks like we're also not seeing full utilization...

how much memory are your 8 cooking processes using? I couldn't even get 4 processes on my local machine with 64GB of RAM (though I did have a bit of stuff in the background). 4 processes ended up being slower than 2 or even 1 lol

muted flicker Oct 24, 2025, 5:53 PM

#

I think that 25GB number is probably accurate, but its probably not the peak

#

im gonna run a cook shortly, ill see what it spikes to

muted flicker Oct 24, 2025, 6:12 PM

#

if you've got a lot of nanite and metahuman, exceeding 64GB RAM makes kinda sense xdd those things are big bois

fiery hornet Oct 24, 2025, 6:27 PM

#

yeah we have lots of big bois indeed hahaha

#

Having 2 cook processes will probably help our build machines though

#

you ever had a cook worker get stuck @muted flicker ?

muted flicker Oct 24, 2025, 6:47 PM

#

oh thats a weird error, never seen that

#

is that while cooking locally?

muted flicker Oct 24, 2025, 7:01 PM

#

muted flicker im gonna run a cook shortly, ill see what it spikes to

yeah looks like we never exceed 25GB RAM

fiery hornet Oct 24, 2025, 7:33 PM

#

muted flicker is that while cooking locally?

no, it was on a testing ec2 instance

#

but I think it was just stuck on a particular package

#

it happened again with 3 cook workers, so I just added the config to output the cooked packages and see what's getting stuck

fiery hornet Oct 24, 2025, 7:35 PM

#

muted flicker yeah looks like we never exceed 25GB RAM

I wish haha

#

with 3 cook processes, we reach ~82

muted flicker Oct 24, 2025, 8:17 PM

#

MONKA

#

we run with 8 and dont even see those numbers haha

muted flicker Oct 24, 2025, 8:23 PM

#

fiery hornet no, it was on a testing ec2 instance

i assume it runs fine without multi-core cook?

fiery hornet Oct 24, 2025, 10:25 PM

#

muted flicker i assume it runs fine without multi-core cook?

I think it might have been a fluke or maybe something related to me enabling zen store for the first time, because I ran multiple cooks back to back after and it was fine. It turns out my EBS volumes weren't fully "initialized" either

muted flicker Oct 24, 2025, 11:14 PM

#

that would do it lol

muted flicker Oct 25, 2025, 1:48 PM

#

I gave octobuild a shot, seems to be unable to compile for some reason

fatal error C1083: Cannot open compiler intermediate file

might be a permissions issue

#

oh its a known issue with whitespace paths pain
https://github.com/octobuild/octobuild/issues/138

GitHub

Windows build fails if there is a whitespace in path · Issue #138 ...

Building the project with Octobuild 1.7.1 installed throws 0>#19 15/92: SharedPCH.UnrealEd.Project.ValApi.Cpp20.cpp @ 2s 0>ERROR: Task failed with exit code: 2 0>SharedPCH.UnrealEd.Project...

fiery hornet Oct 26, 2025, 12:12 AM

#

ah yeah hahahaha

#

i've had the habit of no whitespace for years now

muted flicker Oct 26, 2025, 8:55 PM

#

good habit evidently hueh

#We used to run on-demand AWS / Google