sox_ng wiki - Dynamic-Formats

Dynamic formats

For many of the file format handlers, you can say --with-whatever=dyn which doesn't include the handler in libsox but loads it up when/if sox_format_init() is called or sox_find_format() is called for a format that isn't built-in. There is also --with-dyn-default which makes dynamic all the format handlers that can be.

This was originally done to circumvent patents so that Debian could include sox and all the free format handlers in its main repositories and mp3, amrnb and amrwb in their non-free repo. Now that those patents have expired, that's a moot point but it makes the sox binary much smaller as the prime subjects for dynamic format handlers are the ones that include third-party libraries to encode and decode the formats the handle and tend to be large.

sox_format_init() scans the directory named by the compile-time define PKGLIBDIR (generated by src/Makefile.am and passed in CFLAGS as -DPKGLIBDIR= according to where SoX will be installed) and adds all the format handlers it finds there to the formats that it understands.

How it works

sox_init() doesn't load dynamic format handlers. That is done by sox_format_init() which scans the directory in which the plugin libraries live, usually /usr/lib/sox_ng, and uses libltdl to load all the plugins and update the list of format handlers that sox (or, rather, libsox) has.

sox_find_format() can also do this: first it scans the built-in handlers and if it doesn't find one that handles the format it needs, it loads the plugins and tries again, so calling sox_format_init() is not necessary; it loads them automatically when a format is required that's not built-in.

The following regards only sox_ng-14.7.0.1 onwards, not yet released.

The hairy case is multi-format handlers such as sndfile and ffmpeg. When these are dynamically loaded, we register not only the generic handler, -t sndfile, but also its other format handlers caf, paf, fap, xi and so on, according to the lsx_*_format_fn functions present in the plugin. Unfortunately, libltdl seems not to give you any way to list the public symbols in a dynamic library so instead we list the other format handlers' names after the generic one in its names field and init_format() goes through all these seeing if there is a symbol lsx_paf_format_fn or whatever. If there is, it registers it before the generic handler so that it is found in s_sox_format_tab[]' before the generic one and preferred when someone sayssox_ng foo.wav foo.xi`.

The even hairier one is mp3 because that is both a filename extension and the name of the plugin that handles mp1, mp2 and mp3 files with separate handler functions for each one. In the generic mp3 handler (which also handles mp3 files with lame) we list "mp3", "mp2", "mp1" in that order so that the mp1 and mp2 handlers take precedence over the mp3 one if they say -t mp2" or want to read or write a file whose name end in ".mp2". (This matters because mp2 can encode a more restricted range of sample rates than mp3).

How this will pan out when we let people use lame to decode mp2 and mp3 and start adding other MPEG encoders and decoders remains to be seen, but we can't break what the old -t mp3 does.

What needs changing to make a format handler dynamic

Each potentially dynamic format handler's code file affects:

configure.ac

  AC_OPTIONAL_FORMAT(whatever, WHATEVER,
    [AC_CHECK_HEADER(whatever.h,
       [AC_CHECK_LIB(whatever, whatever_init,
          WHATEVER_LIBS="$WHATEVER_LIBS -lwhatever", using_whatever=no)],
     using_whatever=no)])

which invokes a macro in m4/optional-fmt.m4 that * creates --with-whatever and --without-whatever options to ./configure, for which --with-whatever=dyn is a third option * adds #define HAVE_WHATEVER 1 if it's available and #define STATIC_WHATEVER 1 if it's not going to be a dynamic format handler * adds the appropriate include directory to WHATEVER_CFLAGS and the appropriate library to WHATEVER_LIBS

These are all used automatically and don't require any new code for each format except in:

src/formats.h

where, instead of

  FORMAT(whatever)

it has

  #if defined HAVE_WHATEVER && (defined STATIC_WHATEVER || !defined HAVE_LIBLTDL)
    FORMAT(whatever)
  #endif

so that it becomes a built-in format only if it's available but not dynamic (if you don't have libtltd, making dynamic format handlers is impossible).

src/optional-fmts.am

gains a paragraph for whatever, similar to all the other ones, that ensures that whatever.c is compiled into libsox if it's static, or into its format plugin otherwise.

Note that the source files for potentially dynamic format handlers are not listed in src/Makefile.am, as that happend here instead.

src/whatever.c

Knows nothing about all this. Everything is handled by the above three files.