🔒 Converting python packages as standalone executable using nuitka

[wooden badger]

I have create a CLI application as a python package using poetry. I found that nuitka can be used to distribute as standalone executable. My target environment is a Debian 11 where no debian packages can be installed (not even build essential). My issue is when I invoke the binary file created by nuitka in dist folder. I get FileNotFoundError: Shared library with base name 'llama' not found.

I have also opened an issue in nuitka. All the supporting information is present there: https://github.com/Nuitka/Nuitka/issues/3094

The traceback leads to this particular function in llama-cpp/ : https://paste.pythondiscord.com/VSFA

I also tried with Pyinstaller which is actually more convenient in many use cases but in mine, I have to manually install debian packages like libc6 and libstdc++ (and I also get a missing _cffi_backend.cpython.cpython-39m-x86_64-linux-gnu.so error).

Has anyone worked in llama_cpp or CLI tool as package? I am stuck with this issue for days.

[fiery saffron]

@wooden badger

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[versed ocean]

Dumb question: have you tried manually putting the library (probably libllama.so or something like that) next to the executable, or perhaps passing a path to it in LD_PRELOAD? If that works, it'd be a simple-ish workaround.

[wooden badger]

I tried adding that shared library in the <package>.dist/ folder directly after creating with --standalone flag. Funny thing is I forced nuitka to recognize these .so files using package configuration in their YAML file. You can check the issue I mentioned that nuitka stdout shows clearly this:

Nuitka-Plugins:dll-files: Found 2 files DLLs from llama_cpp installation.

and also is added to the dist folder already

Still I get the FileNotFoundError @versed ocean

[versed ocean]

Hmm, and what happens if you run it like LLAMA_CPP_LIB=path/to/libllama.so ./mypackage.bin --help?

(I'm thinking maybe the custom path resolution that llama uses to load that library doesn't play well with where nuitka puts it.)

[wooden badger]

Well this I have to try. Let me check it out

[wooden badger]

Ok so this actually fixes the problem but yeah now I get another issue (typical devs problem).

[versed ocean]

Aha, great

As for why it doesn't work out of the box - you're saying nuitka puts these shared libraries into dist? The snippet you posted searches for them under lib:

_base_path = pathlib.Path(os.path.abspath(os.path.dirname(__file__))) / "lib"

so maybe that's the reason.

[fiery saffron]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.