#Where can I learn all the details about dynamic libraries in linux?

99 messages · Page 1 of 1 (latest)

nocturne relic
#

The only small things I found are dlopen's man page to learn the syntax of dlopen with RTLD_LAZY, dlsym and dlclose , and some of GCC's documentation showing the command line flags used to compile your own dynamic library, with C-style mangling of course, and literally nothing else.

I feel like this should be taught in university the same way linked lists are taught, but, of course, it is not, lmao.

There was some magical dynamic library called glibc and every single executable in linux was depending on it somehow, but I can't understand how it works or why it exists.

Also, every single large C/C++ library is defined by these dynamic libraries compiled by the author and, somehow, are portable for every single linux distro, I want to learn how that is possible.

torn bobcatBOT
#

When your question is answered use !solved to mark the question as resolved.

Remember to ask specific questions, provide necessary details, and reduce your question to its simplest form. For tips on how to ask a good question run !howto ask.

sour sand
#

well, I'm don't have the kind of deep understanding you seek, but I have some thoughts for you

#

as to "how" re: every distro, likely because Linux is basically still Linux regardless of distro

#

meaning, the kernel is still the linux kernel

#

surely you can compile and re-compile your own kernel with varying options, like there are some distros designed for music/audio/video production that have some "real time" type options built into the kernel, but it's still the linux kernel (as opposed to the BSD kernel)

#

and presumably this does not work if we're talking about e.g. moving something from Linux to BSD because there are notable breaking differences between the two

#

as for glibc, yeah that's a thing in Linux. glibc upgrades are maybe one of the biggest challenges in distro maintainers doing dist upgrades because pretty much literally everything depends on glibc

#

glibc is basically just The C Standard Library, specifically the one created and maintained by GNU

#

so, all the common stuff like malloc, free, etc

#

as for how shared libraries work on an internals level, then that's going to be found somewhere inside the implementations of both your compiler/stdlib and the kernel

#

as for whether that's actually officially documented anywhere else than by the source code, idk

nocturne relic
#

I just wanna learn how dynamic libraries work

sour sand
#

so like, the kernel probably enters this picture because e.g. virtual memory mapping

#

and also probably process handling

#

in that one process should have one copy of a given shared lib in memory, different separate processes should have different copies

sour sand
sour sand
#

also, there are more reasons to learn the kernel than just being a contributor

#

understanding what's really going on under the hood will help you write better C++ and help you better solve strange difficult to debug problems

nocturne relic
#

is that what you mean?

sour sand
#

there's a CppCon youtube about some really weird quirk that was discovered by someone at facebook

#

had to do with virtual memory mapping and page boundaries and the kernel

#

the person who fixed it surely needed to have kernel understanding to do so

nocturne relic
sour sand
#

I don't think so, I don't really remember

#

more like there was some C++ bug no one could figure out, until someone did

#

but it took deep knowledge of internals to fix it

#

well, I just searched, I can't find it and I'm loathe to slog through the thousands of CppCon youtubes by now

#

it's in there somewhere, watch enough CppCon and you'll find it 😉

dull saffron
#

well a bit more seriously, it's just overall cursed

#

I'm not sure what about dynamic libraries you want to learn, but on linux you'd typically turn to what ELF is and what a dynamic linker is

#

and how a program even gets started

nocturne relic
dull saffron
#

a program basically lists what "dynamic libraries" it needs, and it's the dynamic linker's job to load all of them

#

the end

#

the basic principle isn't really more complicated than that

#

the details are gory though

#

like how do you figure out if a found dependency is the right version of that dependency or not, which depends on a bunch of things and typically relies on some information that's embedded into the program/dynamic library so that they can be identified

#

like where the dynamic linker even goes to look for dependencies

nocturne relic
dull saffron
#

like the order in which dependencies are loaded

nocturne relic
#

that the ELF file needs?

dull saffron
#

which by the way can break a program

#

like how I absolutely hate that symbol interposition even is a thing in ELF

#

like how once you've loaded everything you still need to resolve the links to the functions properly

#

like how this doesn't account for dynamic loading techniques which occur during program execution instead of program startup

#

how you have a bunch of stupid issues that can also crop up with the unloading part

#

how this whole thing is also technically not compatible with the c++ standard if you "do it wrong :)"

dull saffron
#

iirc ldd prints out the full path of the file that would end up being loaded

sour sand
dull saffron
#

right there's also this "fun thing" about the dynamic loader being a variable that can be tuned by the program?

sour sand
#

I might have in interest in learning some of this

dull saffron
#

best source of motivation

#

but that aside I just mentioned a bunch of different topics

sour sand
#

well that's the motivation, but what about documentation? does such documentation exist or is it just "read the source"?

dull saffron
#

meh, I guess I'll try to pull up a document that talked about that, though last time I checked I wasn't completely in agreement with some of the conclusions/recommendations

#

if I can find the name again

#

but conclusions/recommendations aside, the process description seemed pretty good? wouldn't know, haven't double checked

#

section 1 of "how to write shared libraries" by ulrich drepper summarizes the loading process of ELF and should give you the basic vocabulary to look for more

nocturne relic
#

due to the ordering?

#

because multiple .so files have the same symbols sometimes?

#

and so, I gotta hack my way it by setting LD_LIBRARY_PATH ?

nocturne relic
sour sand
#

ah yes, good ol ulrich and everything every developer should know about how RAM circuits are designed and wired

dull saffron
sour sand
#

which is mostly everything most developers don't need to know

dull saffron
sour sand
#

could have said "some memory is fast, some is slow" book over

#

but I would therefor trust him to describe ELF

dull saffron
#

dunno, I never dug deeper than what I found there

#

I just ended up having some stupid issue and in the process of looking info I ended up finding that and it was mostly consistent with what I knew already, with a bunch of additional info

nocturne relic
dull saffron
#

no, you could have just looked it up 10 times in the time I took to answer this btw

#

well I mean it's not a book

#

it's a publication an article

nocturne relic
#

alright, thanks, but first I gotta go back to learning make, because dynamic libraries are interesting when I actually need them, but make is something I really need at work

#

!solved

torn bobcatBOT
#

Thank you and let us know if you have any more questions!

sour sand
#

to be fair, when googling this, there are thousands of results from sites I've never heard of

#

and you know, zero day exploits are a thing, spectre and meltdown are (were?) a thing

#

I'd suggest dartmouth is probably trustworthy

dull saffron
#

you could also go on his website and find the thing directly if you trust that kind of thing

sour sand
#

they did technically do the very first work on basically the very first OS I believe

dull saffron
#

even for "paid" publications it's common to find free versions of articles on the author's website

sour sand
#

idk, I saw some documentary on it, I'm pretty sure it was dartmouth

sour sand
dull saffron
#

well there are plenty of ways to go about getting a copy

dull saffron
sour sand
#

which also didn't come up in my search, but yeah ok