unable to install libsndfile1 | Railway | Page 1

bleak gull Dec 15, 2022, 5:48 PM

#

My deployment depends on the python package soundfile. When I try to run the server, soundfile is an import and it crashes with the following message: OSError: cannot load library 'libsndfile.so' when trying to import soundfile. Most of what I found points to libsndfile1 being properly installed.

I've tried adding to aptPkgs but that fails because it says its not found when installing. I've tried manually installing in the build phase with "sudo apt-get update", "sudo apt-get install -y libsndfile1" and this installs it, but the application crashes saying it's not found. I got the same error as above when I tried installing without sudo: package not found from apt-get.

radiant summitBOT Dec 15, 2022, 5:48 PM

#

Project ID: 2b6639e4-c4ef-4e45-91a0-f37672379918

#

The installation of libsndfile1 failed due to a version mismatch between the library and the system's GLIBC version.

bleak gull Dec 15, 2022, 5:49 PM

#

2b6639e4-c4ef-4e45-91a0-f37672379918

#

"The installation of libsndfile1 failed due to a version mismatch between the library and the system's GLIBC version." I have seen something like this in the console when trying to install without sudo

radiant summitBOT Dec 15, 2022, 6:09 PM

#

Nixpacks and apt packages are different, so they are not equivalent.

south atlas Dec 15, 2022, 6:09 PM

#

Whoops

#

Anyway you can configure them both via the env vars listed here. https://nixpacks.com/docs/configuration/environment

Environment | Nixpacks

App source + Nix packages + Docker = Image

#

Have a play round with a few different combos

bleak gull Dec 15, 2022, 6:17 PM

#

#9 0.574 apt-get: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found

timber quest Dec 15, 2022, 9:01 PM

#

@bleak gull - going to tag in @plush wagon who may be able to help you debug installing system pkgs

#

I am working on guides here, but whenever someone needs a core lib, it usually runs into a footgun like situation. Something we wish to fix.

plush wagon Dec 15, 2022, 9:14 PM

#

Do you have a sample repo to test on @bleak gull ?
I also see that you have successfully installed libsndfile and it is now crashing due to missing libcudart

bleak gull Dec 15, 2022, 9:42 PM

#

OSError: cannot load library 'libsndfile.so': libsndfile.so: cannot open shared object file: No such file or directory

2022-12-15 20:14:42.771003: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /nix/store/c8zdsf7csk8msjb914bhccdhq0ycmqys-gcc-11.3.0-lib/lib:/nix/store/nbd9kijlxi8ppm70g1m85j7z31kbpqjx-zlib-1.2.12/lib:/usr/lib

2022-12-15 20:14:42.771058: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine

#

the issue is still libsndfile.so. libcudart is a warning from tensorflow that can be ignored - as shown above.

#

I can open up the repo. it will be rhelsing/basic-pitch-server

#

Nevermind, it's already public: https://github.com/rhelsing/basic-pitch-server/

GitHub

GitHub - rhelsing/basic-pitch-server

Contribute to rhelsing/basic-pitch-server development by creating an account on GitHub.

timber quest Dec 15, 2022, 9:48 PM

#

Gotcha this helps. Really dumb question I would be interested to hear from you. Why Railway for these types of workloads, I always find it interesting when someone picks us over say, a Google Tensor Compute Instance

plush wagon Dec 15, 2022, 9:49 PM

#

bleak gull ``` OSError: cannot load library 'libsndfile.so': libsndfile.so: cannot open sha...

Can you please link to that deployment with this error? I can't see that on the project id you provided

plush wagon Dec 15, 2022, 9:56 PM

#

plush wagon Can you please link to that deployment with this error? I can't see that on the ...

ah nvm I see the edited error

bleak gull Dec 15, 2022, 10:45 PM

#

@timber quest with railway, I have had to deal with the least amount of BS. It just works and gets out of the way. I dont trust google, they make things overly complicated and lock you in.

#

@timber quest and this model is very quick, runs sub 100ms.

#

For the time being, I'm trying to set up a service that I can use while developing, not in a production capacity. I've been running it off my other laptop, but thats gotten annoying haha

bleak gull Dec 17, 2022, 12:51 AM

#

Any thoughts on what I might try next?

timber quest Dec 17, 2022, 1:00 AM

#

We just switched to an Ubuntu base image I believe. Can ya try another deploy?

timber quest Dec 17, 2022, 1:18 AM

#

Also, unfortunately JR's been busy with the Nixpacks 1.0 rollout so we haven't had the time to look deeply into this issue sorry. Might be sometime till we get resolution for ya.

#

In the meantime, I am raising a ticket and collecting all of the information.

radiant summitBOT Dec 17, 2022, 1:18 AM

#

Flagging this thread. A team member will be with you shortly.

bleak gull Dec 17, 2022, 1:18 AM

#

No worries. Is the base image something that can be set manually in my config?

timber quest Dec 17, 2022, 1:21 AM

#

Its something that we change on your behalf but this is a JR question, pinging him internally to get an answer for you.

plush wagon Dec 17, 2022, 5:14 AM

#

It is possible to set it manually https://nixpacks.com/docs/configuration/file#build-image. However this likely isn't what you are looking for. Sorry I haven't had that much time to look into what packages to install. I made a bit of progress but kept hitting cuda errors. I think just missing some apt/nix package. You might be able to find the correct one here? https://search.nixos.org/packages?channel=unstable&from=0&size=50&sort=relevance&type=packages&query=cudaPackages
I have very little experience in the cuda/tensorflow world though so sadly can't give much immediate help. Would love to add first class support for this though. So please let us know if you get it building with nixpacks. Alternatively you can just add a Dockerfile to your repo and we will build with that

Configuration File Reference | Nixpacks

App source + Nix packages + Docker = Image

hushed hamlet Dec 17, 2022, 5:53 PM

#

just for the reference, I am facing the same issue while deploying a FastApi service which uses some audio input/output

plush wagon Dec 17, 2022, 6:37 PM

#

What libraries are it using?

hushed hamlet Dec 17, 2022, 6:40 PM

#

plush wagon What libraries are it using?

spleeter

#

but basically it seems it is asking to install libsndfile on the system where the application is running

#

and i am not sure how can we install packages on the system on railway

#

trying NIXPACKS_APT_PKGS env var to install libsndfile, will update here if that works

#

nope, not working still the same issue

bleak gull Dec 17, 2022, 11:07 PM

#

I'm going to play with some different base images and see if I can have any luck

bleak gull Dec 18, 2022, 12:43 AM

#

I think Im a bit closer. Still no luck. libsndfile.so gets installed here: /lib/x86_64-linux-gnu/libsndfile.so but the error is saying it cant be found from the virtual environment: /opt/venv/lib/python3.9/site-packages/_soundfile_data/libsndfile.so -- Is there a way to make the system libraries always available to the virtual environments? I think that is the issue. I think maybe using export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH or export LD_LIBRARY_PATH=/lib/x86_64-linux-gnu/:$LD_LIBRARY_PATH

bleak gull Dec 18, 2022, 1:07 AM

#

That was it: set LD_LIBRARY_PATH to /lib/x86_64-linux-gnu/:/usr/local/bin:/usr/lib/

#

aptPkgs = ["...", "libsndfile1-dev", "libsndfile1"]

#

with buildImage = 'ghcr.io/railwayapp/nixpacks:ubuntu-1671044363'

hushed hamlet Dec 19, 2022, 7:17 AM

#

@bleak gull hey did it worked for you? if so, can you help me out too please?

molten hawk Dec 19, 2022, 7:44 AM

#

Their message says to set 3 variables in nixpacks.toml

LD_LIBRARY_PATH = /lib/x86_64-linux-gnu/:/usr/local/bin:/usr/lib/

aptPkgs = ["...", "libsndfile1-dev", "libsndfile1"]

buildImage = 'ghcr.io/railwayapp/nixpacks:ubuntu-1671044363'

#

@hushed hamlet Maybe start there?

hushed hamlet Dec 19, 2022, 7:47 AM

#

molten hawk Their message says to set 3 variables in nixpacks.toml LD_LIBRARY_PATH = /lib/...

mean I should create a nixpacks.toml file in my project repo with these variables?

molten hawk Dec 19, 2022, 7:49 AM

#

Yes

hushed hamlet Dec 19, 2022, 3:56 PM

#

Still, build is failing with error #11 ERROR: executor failed running [/bin/bash -ol pipefail -c python -m venv /opt/venv && . /opt/venv/bin/activate && pip install -r requirements.txt]: exit code: 1 ----- > [stage-0 7/8] RUN --mount=type=cache,id=s/d4e470c9-cadf-479f-86b7-4e8863137acb-/root/cache/pip,target=/root/.cache/pip python -m venv /opt/venv && . /opt/venv/bin/activate && pip install -r requirements.txt: ----- executor failed running [/bin/bash -ol pipefail -c python -m venv /opt/venv && . /opt/venv/bin/activate && pip install -r requirements.txt]: exit code: 1 Error: Docker build failed

hushed hamlet Dec 19, 2022, 5:00 PM

#

#11 77.31       creating build/temp.linux-x86_64-cpython-38
#11 77.31       gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/nix/store/frwb4ki3yl7c4bl7jhxk3h035rx8g158-libxcrypt-4.4.33/include -fPIC -DNPY_NO_DEPRECATED_API=NPY_1_13_API_VERSION -I/tmp/pip-build-env-atj4nhhm/overlay/lib/python3.8/site-packages/numpy/core/include -I/opt/venv/include -I/nix/store/zzfhcf101498slkcfgi085jqd33rjf95-python3-3.8.16/include/python3.8 -c _pysndfile.cpp -o build/temp.linux-x86_64-cpython-38/_pysndfile.o
#11 77.31       In file included from _pysndfile.cpp:806:
#11 77.31       pysndfile.hh:56:10: fatal error: sndfile.h: No such file or directory
#11 77.31          56 | #include <sndfile.h>
#11 77.31             |          ^~~~~~~~~~~
#11 77.31       compilation terminated.
#11 77.31       error: command '/root/.nix-profile/bin/gcc' failed with exit code 1
#11 77.31       [end of output]```

bleak gull Dec 19, 2022, 11:45 PM

#

Can you post your nixpacks.toml and requirements.txt

hushed hamlet Dec 20, 2022, 4:25 AM

#

requirements.txt:

�s�p�l�e�e�t�e�r�
�u�v�i�c�o�r�n�
�n�u�m�b�a�
�l�l�v�m�l�i�t�e�
�n�v�i�d�i�a�-�t�e�n�s�o�r�r�t�
�p�y�t�h�o�n�-�m�u�l�t�i�p�a�r�t�
�p�y�s�n�d�f�i�l�e�```

nixpacks.toml:
```LD_LIBRARY_PATH = '/lib/x86_64-linux-gnu/:/usr/local/bin:/usr/lib/'

aptPkgs = ["...", "libsndfile1-dev", "libsndfile1"]

buildImage = 'ghcr.io/railwayapp/nixpacks:ubuntu-1671044363'```

bleak gull Dec 20, 2022, 5:51 AM

#

From the error, it’s looking for sndfile.h - that’s a slightly different error from what I was getting

#

I’d try using python 3.9 and searching on nix packages for python39.pysoundfile and including that in your nixpacks

#

And then maybe removing that from requirements

hushed hamlet Dec 20, 2022, 6:02 PM

#

I removed pysndfile from requirements.txt and updated nixpacks.toml file like this:


aptPkgs = ["...", "libsndfile1-dev", "libsndfile1", "python3.9-soundfile"]

buildImage = 'ghcr.io/railwayapp/nixpacks:ubuntu-1671044363'```

But still getting this error:
```from . import core
File "/opt/venv/lib/python3.8/site-packages/librosa/core/__init__.py", line 6, in <module>
from .audio import *  # pylint: disable=wildcard-import
File "/opt/venv/lib/python3.8/site-packages/librosa/core/audio.py", line 8, in <module>
import soundfile as sf
File "/opt/venv/lib/python3.8/site-packages/soundfile.py", line 189, in <module>
_snd = _ffi.dlopen(_libname)
OSError: cannot load library 'libsndfile.so': libsndfile.so: cannot open shared object file: No such file or directory```

bleak gull Dec 21, 2022, 4:46 AM

#

buildImage = 'ghcr.io/railwayapp/nixpacks:ubuntu-1671044363'

[variables]
LD_LIBRARY_PATH = '/lib/x86_64-linux-gnu/:/usr/local/bin:/usr/lib/'

[phases.setup]
providers = ["...", "python"]
nixPkgs = ["..."]
aptPkgs = ["...", "libsndfile1-dev", "libsndfile1"]

[phases.build]
cmds = ["echo start build", "...", "echo done build"]

#

heres my exact nixpack.toml

#

Use python 3.9.0 if you can

#

add a file called .python-version with just 3.9.0 in it

#

see if that works

hushed hamlet Dec 21, 2022, 6:40 AM

#

bleak gull see if that works

it seems to have been worked, thanks!

#unable to install libsndfile1