Ruby extension wrapper around C library fails to load installed library

Question

The Ruby extension davenport-ruby to the C library davenport will not load properly on Ubuntu and Debian. It works alright on a development machine (MacOS), as demonstrated by the README of the smoke test Ruby project, dvt

The RubyGems loader (via bundler) compiles and installs the extension as follows:

~/dvt$ rm -rf ~/.bundle
~/dvt$ bundle install
Fetching gem metadata from https://rubygems.org/.
Using bundler 2.0.2
Fetching davenport 1.0.2.pre
Installing davenport 1.0.2.pre with native extensions
Bundle complete! 1 Gemfile dependency, 2 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
~/dvt$ bundle info davenport
  * davenport (1.0.2.pre)
    Summary: Ruby binding for the Davenport library
    Homepage: https://github.com/wbreeze/davenport-ruby
    Path: /home/deploy/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0\
      /gems/davenport-1.0.2.pre
~/dvt$ ls /home/deploy/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0\
      /gems/davenport-1.0.2.pre/lib/davenport_ruby/
davenport_ruby.so
~/dvt$

However attempting to run the program yields:

deploy@localhost:~/dvt$ ruby test.rb
Traceback (most recent call last):
    6: from test.rb:1:in `<main>'
    5: from /home/deploy/.rbenv/versions/2.6.3/lib/ruby/2.6.0/rubygems\
       /core_ext/kernel_require.rb:34:in `require'
...
    1: from /home/deploy/.rbenv/versions/2.6.3/lib/ruby/2.6.0/rubygems\
       /core_ext/kernel_require.rb:54:in `require':\
  libdavenport.so.0: cannot open shared object file: No such file or\
    directory - /home/deploy/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0\
      /gems/davenport-1.0.2.pre/lib/davenport_ruby/davenport_ruby.so\
(LoadError)

The library file, libdavenport.so.0 exists in /usr/local/lib. Making that part of the load path with, ruby -I /usr/local/lib test.rb yields the same result.

The library file, davenport_ruby.so exists in /home/deploy/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/davenport-1.0.2.pre/lib/davenport_ruby as shown here:

/home/deploy/.rbenv/versions/2.6.3/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require': libdavenport.so.0: cannot open shared\
object file: No such file or directory - /home/deploy/.rbenv/versions/2.6.3\
/lib/ruby/gems/2.6.0/gems/davenport-1.0.2.pre/lib/davenport_ruby\
/davenport_ruby.so (LoadError)
~/dvt$ ls -l ~/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/\
davenport-1.0.2.pre/lib/davenport_ruby/
total 72
-rwxr-xr-x 1 deploy deploy 69640 Jul 25 17:02 davenport_ruby.so
~/dvt$

The published 1.0.2.pre version of the Ruby davenport extension comes from this PR which link shows all of the code bits (and quite a lot of experiments). Is it some detail or piece of the extension gem not quite right?

davenport_ruby.so

The .so file is the shared library file compiled upon installation of the gem. It isn't present in the contents of the gem:

~/davenport-ruby/dltem[i1]$ gem unpack davenport-1.0.2.pre.gem
Unpacked gem: '/Users/dcl/davenport-ruby/dltem/davenport-1.0.2.pre'
~/davenport-ruby/dltem[i1]$ ls -R
davenport-1.0.2.pre davenport-1.0.2.pre.gem

./davenport-1.0.2.pre:
History.txt README.rdoc Rakefile    ext     lib

./davenport-1.0.2.pre/ext:
davenport_ruby

./davenport-1.0.2.pre/ext/davenport_ruby:
davenport_ruby.c    extconf.rb

./davenport-1.0.2.pre/lib:
davenport.rb
~/davenport-ruby/dltem[i1]$

When RubyGems installs the gem, does it detect the s.extensions << 'ext/davenport_ruby/extconf.rb' in the gem spec and execute that file with ruby? Does it run make for the resulting make file?

What will enable ruby test.rb to run without the error?

This gist contains a comparison of the unique global symbols in libdavenport.so.0 and davenport_ruby.so, and output from the gem environment and ruby -e 'puts $:.join("\n")' commands.

ldd

This question about the not found (LoadError) problem suggests using ldd to verify the linking. (There is a related question that is unanswered, but with a similar suggestion in the comment.)

Indeed, the extension library, although built, does not link to the installed libdavenport.so.0:

~/dvt$ bundle install
Fetching gem metadata from https://rubygems.org/.
Using bundler 2.0.2
Fetching davenport 1.0.2.pre
Installing davenport 1.0.2.pre with native extensions
Bundle complete! 1 Gemfile dependency, 2 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
~/dvt$ ldd ~/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/davenport-1.0.2.pre/lib/davenport_ruby/davenport_ruby.so
    linux-vdso.so.1 (0x00007fff2d3e3000)
    libdavenport.so.0 => not found
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f9a5be42000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9a5baa3000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f9a5c349000)
~/dvt$

ldconfig

This very old question over on AskUbuntu suggests the use of the ldconfig command. Since libdavenport.so.0 is installed in /usr/local/lib, the command was ldconfig /usr/local/lib. However to run it required a different login that can get root privileges. Switching back to the deploy account that is doing the installation of the Ruby program:

~/dvt$ ruby test.rb
Hola
[1, 3, 2, 4]
~/dvt$

Voila. It is now working. The question becomes how to get that to work with the deploy/install, and with a user that does not have root privileges. (Installing the library libdavenport required root privileges as well, although it was compiled and installed from source.)

Why would the C library `libdavenport.so.0` be calling for the extension library, `davenport_ruby.so` (which is present), and not the other way around? The C library references nothing in the extension, rather the extension references the C library. Could this be a name collision issue with the linking? — Douglas Lovell, Jul 25 '19 at 21:51

score 2 · Accepted Answer · answered Jul 30 '19 at 00:38

Running ldconfig /usr/local/lib as root turns out to be the answer. It causes the loader to be able to locate the installed libdavenport.so.0 library at runtime.

To the follow-on questions, the answer is that the make install probably won't be able to eliminate the need to run ldconfig on some systems. There is an old issue against another library that made the attempt, still open, at esnet/iperf. They didn't figure it out. If they did, they didn't update the issue. A newer issue against libcheck/check has similar struggles.

The current Gnu documentation for libtool has open in its implementation issues,

The install Makefile target should warn the package installer to set the proper environment variables (LD_LIBRARY_PATH or equivalent), or run ldconfig.

The "solution" in this case, is a documentation solution for the Davenport library, an update to the README made with this PR.

There's a nice short explanation about library linking and loading at Cprogramming.com (which is ad' supported but not painfully so; the article earns the reference).

Max · Answer 2 · 2019-07-30T15:33:21.313

libdavenport.so.0: cannot open shared object file: No such file or\
directory - /home/deploy/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0\
  /gems/davenport-1.0.2.pre/lib/davenport_ruby/davenport_ruby.so

This error means that the native extension davenport_ruby.so can't load Davenport's shared library libdavenport.so.0. You need to ensure that the shared library is in your load path using ldconfig or by adding its path to LD_LIBRARY_PATH.

Old incorrect answer:

Ruby C extensions have to be compiled for the platform you're running them on. If you just download some Ruby code that uses an extension, it won't work out of the box.

Did you build the extension using gem or bundle? Did you run that on the Ubuntu machine you're trying to run the script on?

The gem is not supposed to contain the .so. When you install the gem, it's supposed to build the .so and put it somewhere in Ruby's load path so that scripts can require it.

Assuming you actually installed the gem and it built the .so successfully, I can think of two possible problems

You installed the gem with the wrong tool or options, so it put the .so in the wrong place
You are running Ruby the wrong way (are you using rbenv or rvm?) so it's looking for the .so in the wrong place

Run gem environment and ruby -e 'p $:' to compare your gem installation directory with Ruby's load path.

Thank you for having a look. The question now has a listing of the gem content and questions about `davenport_ruby.so` and the extension build. The gem file was built using `gem build davenport.gemspec`. It was installed using bundler and a Gemfile. — Douglas Lovell, Jul 25 '19 at 21:41
Thank you very much for these tips about verifying the gem installation and native compile. The `ruby -e 'p $:'` command was a TIL. The gist linked at the bottom of the question now contains the output from both. It *looks like* the extension gem installs, compiles, and is the one loaded by the smoke test program. If the gem installation and loading looks good, perhaps the problem is in the C code itself or in the linking. — Douglas Lovell, Jul 26 '19 at 14:41
Upvoted in appreciation, and because the answer contains really great strategies for verifying the build and setup. Doing so ruled-out a great deal. — Douglas Lovell, Jul 30 '19 at 00:40

Ruby extension wrapper around C library fails to load installed library

davenport_ruby.so

ldd

ldconfig

2 Answers2