Patching Library Paths in Compiled Binaries

Toomanycats bio photo By Toomanycats

I was given a compiled library from a collaborator who isn’t keen on handing out source code. The programs worked in a Docker container but that wasn’t going to work for my lab’s distributed computing model.

I needed a way to change where the binaries expected the libraries to be. luckly my group’s IT manager came by and showed my how to do it.

The names have been Changed

I’ve changed the names of all the programs involved to avoid NDA complications and other unwanted public discloures.

The binary package is called, FooBarPackage that has two library dependencies, Intel and BazLib.

Fixing Compiled Binary Library Paths

FooBarPackage was shipped compiled with Intel and BazLib.

We need to be able to tell the programs inside of FooBarPackage to use LD_LIBRARY_PATH. The program ldd will show us which libraries the programs depend on and if they are found.

We’ll start fixing a single program contained in FooBarPackage, named FooBin.

ldd FooBarPackage/FooBin
    linux-vdso.so.1 =>  (0x00007ffc0faae000)
    BazLib_struct.so => not found
    BazLib_dt.so => not found
    /path/that/doesnot/exist/libIntel_foobaz.so => not found
    /path/that/doesnot/exist/libIntel_stuff.so => not found
    /path/that/doesnot/exist/libIntel_core.so => not found
    libIntel_stuff.so => not found
    libguide.so => not found
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff0dea00000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007ff0de6f7000)
    libm.so.6 => /lib64/libm.so.6 (0x00007ff0de3f5000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ff0de1de000)
    libc.so.6 => /lib64/libc.so.6 (0x00007ff0dde1c000)
    /lib64/ld-linux-x86-64.so.2 (0x00007ff0dec64000)

Library components like this one BazLib_struct.so can be fixed using an environment variable. But not /path/that/doesnot/exist/libIntel_foobaz.so.

We can fix the two BazLib_struct.so and BazLib_dt.so with this: /export LD_LIBRARY_PATH=/FooBarPackage_src/FooBarPackage/libs.

Once we do however, we’ll see MORE fully specified paths to Intel components. Fully specified paths are bad because LD_LIBRARY_PATH won’t fix them.

The Fix

The fix is to remove the directory name from those fully specified paths so that setting LD_LIBRARY_PATH will allow them to be found.

example /path/that/doesnot/exist/libIntel_foobaz.so will become libIntel_foobaz.so.

We must do that for all the binaries in FooBarPackage. But that’s not all….

The complex part of this, is that the libraries themselves, call each other, and use full paths. So we need to not only fix the binaries, but also the libraries themselves.

Examine the library files

for i in *.so;do
    echo $i
    ldd $i | grep 'path' | sort | uniq
done

output

BazLib_dt.so
    /path/that/doesnot/exist/libIntel_foobaz.so => not found
    /path/that/doesnot/exist/libIntel_stuff.so => not found
    /path/that/doesnot/exist/libIntel_core.so => not found

BazLib_struct.so
    /path/that/doesnot/exist/libIntel_foobaz.so => not found
    /path/that/doesnot/exist/libIntel_stuff.so => not found
    /path/that/doesnot/exist/libIntel_core.so => not found
libimf.so
libsvml.so

There’s only 3 so files to fix from two indirect calls coming from: BazLib_struct.so and BazLib_dt.so .

Intel directory

Check all the so files for bad paths:

for i in *.so;do
    ldd $i | grep "not found"
done

Note “not found” did not occur and hence we don’t need to fix these at all.

patchelf

Not found in the yum repository. Instead of adding another repo it was compiled from source. It was easy as I recall.

Patch the Binary FooBin

The utility patchelf is pretty easy to use. The changes are made in-place.

patchelf --replace-needed /path/that/doesnot/exist/libIntel_foobaz.so libIntel_foobaz.so FooBin
patchelf --replace-needed /path/that/doesnot/exist/libIntel_stuff.so libIntel_stuff.so  FooBin
patchelf --replace-needed /path/that/doesnot/exist/libIntel_core.so libIntel_core.so   FooBin

libs dir

Only two of these libraries need to be fixed, BazLib_struct.so and BazLib_dt.so.

patchelf --replace-needed /path/that/doesnot/exist/libIntel_foobaz.so libIntel_foobaz.so BazLib_dt.so
patchelf --replace-needed /path/that/doesnot/exist/libIntel_foobaz.so libIntel_foobaz.so BazLib_struct.so
patchelf --replace-needed /path/that/doesnot/exist/libIntel_stuff.so libIntel_stuff.so  BazLib_dt.so
patchelf --replace-needed /path/that/doesnot/exist/libIntel_stuff.so libIntel_stuff.so BazLib_struct.so
patchelf --replace-needed /path/that/doesnot/exist/libIntel_core.so libIntel_core.so  BazLib_dt.so
patchelf --replace-needed /path/that/doesnot/exist/libIntel_core.so libIntel_core.so BazLib_struct.so

The Intel Library

Luckily, these do not need any hacking at all.

Fix All Binaries

Now we need to run patchelf on the remaining binaries.

Make some variables for looping a bit later:

bad_dir="/path/that/doesnot/exist"
stuff="libIntel_stuff.so"
core="libIntel_core.so "
foobaz="libIntel_foobaz.so"

Make a list of binaries:

list=($(find . -executable -type f))

for bin in ${list[@]};do
    echo $bin
    ldd $bin | grep "path" | sort | uniq
done

output with some errors and non essential lines removed:

./segMasks
    /path/that/doesnot/exist/libIntel_core.so => not found
    /path/that/doesnot/exist/libIntel_foobaz.so => not found
    /path/that/doesnot/exist/libIntel_stuff.so => not found
./invertDisplacementField
    /path/that/doesnot/exist/libIntel_core.so => not found
    /path/that/doesnot/exist/libIntel_foobaz.so => not found
    /path/that/doesnot/exist/libIntel_stuff.so => not found
./reg
    /path/that/doesnot/exist/libIntel_core.so => not found
    /path/that/doesnot/exist/libIntel_foobaz.so => not found
    /path/that/doesnot/exist/libIntel_stuff.so => not found
./FooBin
    /path/that/doesnot/exist/libIntel_core.so => not found
    /path/that/doesnot/exist/libIntel_foobaz.so => not found
    /path/that/doesnot/exist/libIntel_stuff.so => not found
./dvRev
    /path/that/doesnot/exist/libIntel_core.so => not found
    /path/that/doesnot/exist/libIntel_foobaz.so => not found
    /path/that/doesnot/exist/libIntel_stuff.so => not found
./applyLocalScaling
    /path/that/doesnot/exist/libIntel_core.so => not found
    /path/that/doesnot/exist/libIntel_foobaz.so => not found
    /path/that/doesnot/exist/libIntel_stuff.so => not found
./averageImage
    /path/that/doesnot/exist/libIntel_core.so => not found
    /path/that/doesnot/exist/libIntel_foobaz.so => not found
    /path/that/doesnot/exist/libIntel_stuff.so => not found
./applyTransforms
    /path/that/doesnot/exist/libIntel_core.so => not found
    /path/that/doesnot/exist/libIntel_foobaz.so => not found
    /path/that/doesnot/exist/libIntel_stuff.so => not found
./restrictImage2Mask
    /path/that/doesnot/exist/libIntel_core.so => not found
    /path/that/doesnot/exist/libIntel_foobaz.so => not found
    /path/that/doesnot/exist/libIntel_stuff.so => not found
./roiDv
    /path/that/doesnot/exist/libIntel_core.so => not found
    /path/that/doesnot/exist/libIntel_foobaz.so => not found
    /path/that/doesnot/exist/libIntel_stuff.so => not found
./invertAffMatrix
    /path/that/doesnot/exist/libIntel_core.so => not found
    /path/that/doesnot/exist/libIntel_foobaz.so => not found
    /path/that/doesnot/exist/libIntel_stuff.so => not found

We can see it’s the same three libraries again.

Loop over the binaries

for prog in ${list}; do
    for lib in ${stuff} ${core} ${foobaz}; do
        echo "running $prog lib $lib"
        patchelf --replace-needed ${bad_dir}/${lib} ${lib} $prog
    done
done