Discussion: Importing scripts from other mods | Factorio | Page 1

analog osprey Sep 1, 2024, 2:43 PM

#

Original message:

I have a problem.
When you import a module from other mod, if that module imports a file from his mod, the pathfinding is kinda broken.

#

justarandomgeek wrote:

yeah that's a known issue if they're not all fully qualified require names, it's fixed for 2.0 i beleive

#

Managed to write a hook for require() which transforms non-absolute imports to their absolute form, while leaving a chance for reserved imports to work (like for __core__.lualib)

#

Main logic:

---- 
--- Creates a require proxy which tracks the current context.
--- 
--- @param current_mod  ModName?
--- @param current_file ModuleID?
--- @return RequireProrotype
function importlib.make_require_proxy(current_mod, current_file)
    --- @param module ModuleID
    --- @return Module
    local function result(module)
        local parsed_result = importlib.parse_import_string(current_mod, current_file, module)
        logging.print(logging.levels.TRACE, "require(%q) [%s, %s] -> %s", module, current_mod or '<no-mod>', current_file or '<no-file>', parsed_result.absolute_link)
        
        local old_require = require --- @type RequireProrotype
        require = importlib.make_require_proxy(parsed_result.mod, parsed_result.file)
        
        local result --- @type Module
        if (parsed_result.absolute_link or parsed_result.is_relative)
        then result = _true_require(parsed_result.absolute_link)
        else
            success, result = pcall(_true_require, parsed_result.absolute_link)
            if (not success) -- Fallback for the '__core__.lualib'
            then result = _true_require(parsed_result.file)
            end
        end
        
        require = old_require
        return result
    end
    
    return result
end

hard kernel Sep 1, 2024, 2:48 PM

#

you're only tryign one path? make sure you try the whole search (require somethign that doesn't exist to see the whole search list. do it from inside a subdir.)

analog osprey Sep 1, 2024, 2:50 PM

#

hard kernel you're only tryign one path? make sure you try the whole search (require somethi...

I transform all import strings to their absolute form, respecting relative imports.

hard kernel Sep 1, 2024, 2:50 PM

#

there are three different ways that transform can happen, depending on what exists

analog osprey Sep 1, 2024, 2:50 PM

#

hard kernel Sep 1, 2024, 2:50 PM

#

you only tried one of them

#

require something that doesn't exist (while inside a subdir) to see the whole search list

analog osprey Sep 1, 2024, 2:52 PM

#

1. require('__othermod__.module') -> unchanged
2. require('module') -> require('__currentmod__.module') OR fallback to require('module')
3. require('./relative-module') -> require('__currentmod__.currentdir.relative-module')

hard kernel Sep 1, 2024, 2:52 PM

#

that second one is wrong

#

there's more options than that

#

require soemthing that doesn't exist while inside a subdir to see them

analog osprey Sep 1, 2024, 2:53 PM

#

I did it

hard kernel Sep 1, 2024, 2:53 PM

#

while in a subdir?

#

also that third one is wrong because it's nto legal at all

analog osprey Sep 1, 2024, 2:53 PM

#

Yes. The 2nd branch has fallback to the unmodified import

hard kernel Sep 1, 2024, 2:54 PM

#

analog osprey Yes. The 2nd branch has fallback to the unmodified import

your'e still missing one

analog osprey Sep 1, 2024, 2:54 PM

#

hard kernel also that third one is wrong because it's nto legal at all

Legal

hard kernel Sep 1, 2024, 2:54 PM

#

not in factorio

analog osprey Sep 1, 2024, 2:54 PM

#

hard kernel not in factorio

It works just fine

analog osprey Sep 1, 2024, 2:55 PM

#

hard kernel your'e still missing one

No because that case is already handled by the OG require, it will look for the current directory, it will skip looking on the root of the module

hard kernel Sep 1, 2024, 2:56 PM

#

analog osprey It works just fine

so you haven't actualyl run this in factorio yet then?

#

🤔 wait hang on, this error message doesn't match the test it actually does...

analog osprey Sep 1, 2024, 2:57 PM

#

??

hard kernel Sep 1, 2024, 2:57 PM

#

oh, yes it does, it pulled teh dots out earlier

#

yeah, factorio's require explicitly does not allow things that start wth . / or \

analog osprey Sep 1, 2024, 2:58 PM

#

Only .. is disallowed

hard kernel Sep 1, 2024, 2:58 PM

#

i'm looking at its code

#

it disallows . / and \ too

analog osprey Sep 1, 2024, 2:59 PM

#

#

I can show you it running if you don't believe me

analog osprey Sep 1, 2024, 3:10 PM

#

hard kernel while in a subdir?

Maybe I was not precise enough in describing. The part on the left is the called require while part on the right is called to the Factorio's true require, which by itself can lookup on different files, thus covering the relative-import case while in subdir.

hard kernel Sep 1, 2024, 3:11 PM

#

then you've changed the search order by doing that

analog osprey Sep 1, 2024, 3:13 PM

#

Files in the current directory are checked 1st? If you, you are correct.

hard kernel Sep 1, 2024, 3:14 PM

#

do the test i suggested to see teh list to confirm, but yeah pretty sure current dir is first then mod root

analog osprey Sep 1, 2024, 3:14 PM

#

combinators-bridge: [DEBUG] Bridge prototypes.mods.90-data-final-fixes for mod utility-combinators was not imported, registration skipped. Error: __combinators-bridge__/meta/importlib.lua:12: module prototypes.mods.90-data-final-fixes.utility-combinators not found;  no such file __combinators-bridge__/prototypes/mods/90-data-final-fixes/utility-combinators.lua no such file __combinators-bridge__/prototypes/mods/90-data-final-fixes/utility-combinators.lua no such file __core__/lualib/prototypes/mods/90-data-final-fixes/utility-combinators.lua

#

From my logs (i check if the optional file exist)

hard kernel Sep 1, 2024, 3:15 PM

#

i can't read taht as a big blob like that

analog osprey Sep 1, 2024, 3:15 PM

#

Give me a second

#

current mod:      combinators-bridge
current module:   meta.importlib
requested import: prototypes.mods.90-data-final-fixes.utility-combinators
search paths:
- __combinators-bridge__/prototypes/mods/90-data-final-fixes/utility-combinators.lua
- __combinators-bridge__/prototypes/mods/90-data-final-fixes/utility-combinators.lua
- __core__/lualib/prototypes/mods/90-data-final-fixes/utility-combinators.lua

#

That list of search paths changes when I'm inside the other mod

#

Okay, strange.

----
--- @param module ModuleID
local function test_require(module)
    local ok, err = pcall(require, module)
    if (not ok)
    then print(("[TRACE]: test-imports: require(%q) -> %s"):format(module, err:gsub("no such file", "\n - ")))
    end
end

test_require('module')
test_require('thismod.module')
test_require('./relative')
test_require('__combinators-bridge__.module')
test_require('__compaktcircuit__.module')
test_require('__inactive__.module')

error("Stop loading")

[TRACE]: test-imports: require("module") -> module module not found;  
 -  __combinators-bridge__/local/module.lua 
 -  __combinators-bridge__/module.lua 
 -  __core__/lualib/module.lua
[TRACE]: test-imports: require("thismod.module") -> module thismod.module not found;  
 -  __combinators-bridge__/local/thismod/module.lua 
 -  __combinators-bridge__/thismod/module.lua 
 -  __core__/lualib/thismod/module.lua
[TRACE]: test-imports: require("./relative") -> module ./relative not found;  
 -  __combinators-bridge__/local/./relative.lua 
 -  __combinators-bridge__/./relative.lua 
 -  __core__/lualib/./relative.lua
[TRACE]: test-imports: require("__combinators-bridge__.module") -> module __combinators-bridge__.module not found;  
 -  __combinators-bridge__/module.lua
[TRACE]: test-imports: require("__compaktcircuit__.module") -> module __compaktcircuit__.module not found;  
 -  __compaktcircuit__/module.lua
[TRACE]: test-imports: require("__inactive__.module") -> Path __inactive__/module.lua does not match any enabled mod.

hard kernel Sep 1, 2024, 3:32 PM

#

starting with ./ isn't supposed to work at all, but the check for it appears to be broken (it should be the same error as starting with / or \ if you want to see it)

analog osprey Sep 1, 2024, 3:32 PM

#

:D

#

But require('.relative') are OK?

hard kernel Sep 1, 2024, 3:33 PM

#

no, that also shouldn't work, but that one might actually trigger teh check as it currently is

analog osprey Sep 1, 2024, 3:33 PM

#

[TRACE]: test-imports: require(".relative") -> module .relative not found;  
 -  __combinators-bridge__/local/relative.lua 
 -  __combinators-bridge__/relative.lua 
 -  __core__/lualib/relative.lua

hard kernel Sep 1, 2024, 3:34 PM

#

start one with a / to see the error for what it's supposed to be disallowing

analog osprey Sep 1, 2024, 3:38 PM

#

OOOPS

#

Looks like a big security issue

hard kernel Sep 1, 2024, 3:45 PM

#

well the good news is it's not really a security issue unless you have sensitive info in .lua files (it'll clobber teh extension if you try somethign else, and it won't actually load anything but lua code), but yeah taht definitely shouldn't work 🤔

analog osprey Sep 1, 2024, 3:46 PM

#

Should I open a bug report, or you would do that?

hard kernel Sep 1, 2024, 3:46 PM

#

i've mentioned it to devs (in another discord channel) but if you want to do a formal bug report for it too that certainly won't hurt

analog osprey Sep 1, 2024, 3:56 PM

#

https://forums.factorio.com/viewtopic.php?f=7&t=115137

Factorio Forums

Lua: `require()` absolute and relative path checks are not working

I was discussing how import statements work with @justrandomgeek on Discord (link), and he noticed that the relative imports I use in my code should not work…

analog osprey Sep 1, 2024, 4:28 PM

#

So, this should be more correct?

hard kernel Sep 1, 2024, 4:29 PM

#

yeah, you can optionally assemble the __core__/lualib path into a resolved path yourself too if you want, but it shouldn't make much difference at that point

#

and i guess maybe optionally also do some extra shuffling to hide the re-mapipng youv'e done from the entries that end up in package.loaded

analog osprey Sep 1, 2024, 4:30 PM

#

What if Factorio would be patched to also include __spac-age__.misc by default? I don't want that to break my code

analog osprey Sep 1, 2024, 4:31 PM

#

hard kernel and i guess maybe optionally also do some extra shuffling to hide the re-mapipng...

Can you explain?

hard kernel Sep 1, 2024, 4:32 PM

#

require caches the loaded results in package.loaded keyed by the name that was passed to it and reuses those if they exist before tryign to re-load the file

#

but sinceyou changed the names you're passing to real require, you might need to move some of those entries around and/or check for them first yourself if you (or the code you're loading) care about that behavior

analog osprey Sep 1, 2024, 4:34 PM

#

Hm. Thanks for the advice.

hard kernel Sep 1, 2024, 4:34 PM

#

and/or potentially make separate "sandbox" copies of it for the various mods you're cross-loading, depending on how much they overlap

analog osprey Sep 1, 2024, 4:35 PM

#

hard kernel and/or potentially make separate "sandbox" copies of it for the various mods you...

That's in the different method.
I didn't succeed in sandboxing globals yet, but that's on my list

hard kernel Sep 1, 2024, 4:36 PM

#

that one is a bit harder (simhelper used to do it, but i used unsafe debug stuff there which is now gone unless you use teh special switch to get it back)

analog osprey Sep 1, 2024, 4:37 PM

#

#

This guards all events, both load-time and runtime.
And this function is guaranteed to run at most ones on each requested module.

hard kernel Sep 1, 2024, 4:40 PM

#

https://github.com/justarandomgeek/factorio-simhelper/blob/master/modloader.lua this is how i did it before we could load mod scripts in sims

analog osprey Sep 1, 2024, 4:43 PM

#

Looks like black magic

hard kernel Sep 1, 2024, 4:43 PM

#

very much so yes

analog osprey Sep 1, 2024, 4:43 PM

#

Powered by the Dark Gods even more than the codebase I'm currently working on.

hard kernel Sep 1, 2024, 4:44 PM

#

the main trick is that i used the debug hook to catch the laoded mod's control.lua just before it starts running and swap out the _ENV binding it got with one i control

#

which lets me lie to it about what global variables exist and what they hold

#

and, importantly, lie differently to different mods!

#

but my _ENV sandbox is pretty thin, i dont' hide the outer stuff fully so i coudl be lazy about copying it all in to each sandbox

analog osprey Sep 1, 2024, 4:57 PM

#

The only good thing in all these meta-imports and hook manipulations is the fact they are all only one-time events, and the slightly increased load time would still be nothing compared to the SE load time (where I plan to use my beautiful bridge).

#

Wait a second.
Is this your mod?
https://github.com/justarandomgeek/factorio-utility-combinators

GitHub

GitHub - justarandomgeek/factorio-utility-combinators

Contribute to justarandomgeek/factorio-utility-combinators development by creating an account on GitHub.

hard kernel Sep 1, 2024, 4:59 PM

#

yes

analog osprey Sep 1, 2024, 4:59 PM

#

xD

#

That's one of the mods I'm trying to bridge.

#

Since you are active person, can I instead send a PR?

#

On my bridge mod, I updated the descriptions & added PickerDollies support.
With the power of PR I would be able to do even more!

analog osprey Sep 2, 2024, 7:38 PM

#

@hard kernel am I correct that errors trhown at Lua are not localized? Is it safe to parse the error string?

hard kernel Sep 2, 2024, 7:40 PM

#

lua can throw any value at all. if i'ts a LocalisedString, it will be translated when it crosses vm boundaries and become a plain string.

#

the vm and game only ever throw plain strings, but mods can throw any kind of nonsense they want

#Discussion: Importing scripts from other mods