Linking installed OpenBLAS libraries in FreeBSD

[lungsi]

OS: FreeBSD 12.1-RELEASE r354233 GENERIC amd64
OpenBLAS: 0.3.10,1
jblas: 1.2.5

Using mmul function in jblas returns

Execution error (UnsatisfiedLinkError) at org.jblas.NativeBlas/dgemm (NativeBlas.java:-2).
org.jblas.NativeBlas.dgemm(CCIIID[DII[DIID[DII)V

I presume it is because jblas is unable to link with the installed OpenBLAS libraries (pkg info -lx openblas)

openblas-0.3.10,1:
        /usr/local/include/cblas.h
        /usr/local/include/f77blas.h
        /usr/local/include/lapack.h
        /usr/local/include/lapacke.h
        /usr/local/include/lapacke_config.h
        /usr/local/include/lapacke_mangling.h
        /usr/local/include/lapacke_utils.h
        /usr/local/include/openblas_config.h
        /usr/local/lib/cmake/openblas/OpenBLASConfig.cmake
        /usr/local/lib/cmake/openblas/OpenBLASConfigVersion.cmake
        /usr/local/lib/libopenblas.a
        /usr/local/lib/libopenblas.so
        /usr/local/lib/libopenblas.so.0
        /usr/local/lib/libopenblasp-r0.3.10.a
        /usr/local/lib/libopenblasp-r0.3.10.so
        /usr/local/libdata/pkgconfig/openblas.pc
        /usr/local/share/licenses/openblas-0.3.10,1/BSD3CLAUSE
        /usr/local/share/licenses/openblas-0.3.10,1/LICENSE
        /usr/local/share/licenses/openblas-0.3.10,1/catalog.mk

Could somebody help me how I should link the installed OpenBLAS library?

[lungsi]

Additional Information

Looking at the source code ~/.m2/repository/org/jblas/jblas/1.2.5/jblas-1.2.3.jar/org//jblas/NativeBlastLibraryLoader.class the BLAS appear to be loaded here.

package org.jblas;

import org.jblas.exceptions.UnsupportedArchitectureException;
import org.jblas.util.LibraryLoader;
import org.jblas.util.Logger;

class NativeBlasLibraryLoader {
    NativeBlasLibraryLoader() {
    }

    static void loadLibraryAndCheckErrors() {
        try {
            try {
                System.loadLibrary("jblas");
            } catch (UnsatisfiedLinkError var3) {
                Logger.getLogger().config("BLAS native library not found in path. Copying native library from the archive. Consider installing the library somewhere in the path (for Windows: PATH, for Linux: LD_LIBRARY_PATH).");
                loadDependentLibraries();
                (new LibraryLoader()).loadLibrary("jblas", true);
            }

            double[] a = new double[1];
            NativeBlas.dgemm('N', 'N', 1, 1, 1, 1.0D, a, 0, 1, a, 0, 1, 1.0D, a, 0, 1);
        } catch (UnsatisfiedLinkError var4) {
            String arch = System.getProperty("os.arch");
            String name = System.getProperty("os.name");
            if (name.startsWith("Windows") && var4.getMessage().contains("Can't find dependent libraries")) {
                System.err.println("On Windows, you need some additional support libraries.\nFor example, you can install the two packages in cygwin:\n\n   mingw64-x86_64-gcc-core   mingw64-x86_64-gfortran\n\nand add the directory <cygwin-home>\\usr\\x86_64-w64-mingw32\\sys-root\\mingw\\bin to your path.\n\nFor more information, see http://github.com/mikiobraun/jblas/wiki/Missing-Libraries");
            }
        } catch (UnsupportedArchitectureException var5) {
            System.err.println(var5.getMessage());
        }

    }

    public static void loadDependentLibraries() {
        String arch = System.getProperty("os.arch");
        String name = System.getProperty("os.name");
        LibraryLoader loader = new LibraryLoader();
        if (name.startsWith("Windows") && arch.equals("amd64")) {
            loader.loadLibrary("libgcc_s_sjlj-1", false);
            loader.loadLibrary("libgfortran-3", false);
        } else if (name.startsWith("Windows") && arch.equals("x86")) {
            loader.loadLibrary("libgcc_s_dw2-1", false);
            loader.loadLibrary("libgfortran-3", false);
        } else if (name.equals("Linux") && arch.equals("amd64")) {
            loader.loadLibrary("quadmath-0", false);
            loader.loadLibrary("gfortran-4", false);
        }

    }
}

[martin-frbg]

Looks as if it "knows" to load other libraries that OpenBLAS depends on when jblas is installed on Windows or Linux, but does not
have corresponding settings for BSD. If there is ldd on FreeBSD you could try to run that on the installed libopenblas.so to see what it needs and modify the loadDependentLibraries function accordingly.

[lungsi]

Thanks so much @martin-frbg Your comment is very helpful but now I can't figure out what libname (string) should I use as argument in the NativeBlasLibraryLoader.class

Using ldd the library shared by the installed OpenBLAS were shown as

$ ldd /usr/local/lib/libopenblas.so
/usr/local/lib/libopenblas.so:
        libthr.so.3 => /lib/libthr.so.3 (0x80067d000)
        libgfortran.so.5 => /usr/local/lib/gcc9/libgfortran.so.5 (0x802e00000)
        libgomp.so.1 => /usr/local/lib/gcc9/libgomp.so.1 (0x803295000)
        libc.so.7 => /lib/libc.so.7 (0x80024a000)
        libquadmath.so.0 => /usr/local/lib/gcc9/libquadmath.so.0 (0x8034c9000)
        libm.so.5 => /lib/libm.so.5 (0x8006aa000)
        libgcc_s.so.1 => /usr/local/lib/gcc9/libgcc_s.so.1 (0x80370f000)
        libdl.so.1 => /usr/lib/libdl.so.1 (0x8006dc000)

In modifying the NativeBlasLibraryLoader.class with the additional conditions

        ...
        } else if (name.equals("FreeBSD") && arch.equals("amd64")) {
            loader.loadLibrary(libname, false);
            loader.loadLibrary(libname, false);
        }

I can't figure out what libname I should use as the arguments.

Should I use "libquadmath" and "libgfortran"?
That is,

        ...
        } else if (name.equals("FreeBSD") && arch.equals("amd64")) {
            loader.loadLibrary("libquadmath", false);
            loader.loadLibrary("libgfortran", false);
        }

PS: Prior to attempting the modification, after locating the shared libraries using the ldd command I have copied the libraries to /compat/linux and then created symbolic link as instructed in the FreeBSD Handbook.

[martin-frbg]

Yes, I would try it like that - if it does not work, you can always try adding the version number (libquadmath-0 and libgfortran-5) as in the entries for the other operating systems. Possibly you will also need loadLibrary lines for the libgcc_s and/or libgomp, libthr as well.

[brada4]

Obvious that Linux compatibility layer will not cross-breed FreeBSD ELF with Linux ELF. If you target compat (like having Linux Oracle JDK), you need Linux OpenBLAS and supporting library packages (CentoS 7 version can be acquired from Fedora EPEL https://fedoraproject.org/wiki/EPEL)

[martin-frbg]

Did not see the JDK version mentioned so was assuming OpenJDK which I believe is available as a native build for *BSD

[lungsi]

Yes @martin-frbg its FreeBSD's OpenJDK8

# pkg info openjdk8
openjdk8-8.252.09.1
Name           : openjdk8
Version        : 8.252.09.1
Installed on   : Sun Jul 12 09:51:09 2020 CEST
Origin         : java/openjdk8
Architecture   : FreeBSD:12:amd64
Prefix         : /usr/local
Categories     : devel java
Licenses       : GPLv2
Maintainer     : [email protected]
WWW            : https://openjdk.java.net/
Comment        : Java Development Kit 8
...

Also, to add further context I am not a native Java programmer I was trying to using the jblas library from Clojure. It was working fine until I noticed the error while calling the matrix multiplication function mmul: Error occurs when trying to multiply two matrices but multiplying two vectors is successful.

Here's further evidence of the installed OpenJDK

$ lein repl
nREPL server started on port 21838 on host 127.0.0.1 - nrepl://127.0.0.1:21838
REPL-y 0.4.4, nREPL 0.6.0
Clojure 1.10.0
OpenJDK 64-Bit Server VM 1.8.0_252-b09

[brada4]

It would be beneficial to clean out linux compat tree.

You are loading jblas.so from
https://github.com/jblas-project/jblas/tree/main/src/main/c/
Default is to use ATLAS, but you can rebuild to use others

  --build-type=...         One of 'lapack', 'atlas', 'openblas', 'nvblas'
                           (default is 'atlas')

[martin-frbg]

Perhaps running truss -t open on the java code can tell what exactly it is trying to load after finding libopenblas ? (I did not come across any tricks for making the UnsatisfiedLinkError more verbose)

[brada4]

LD_TRACE_LOADED_OBJECTS_ALL=Y
java ....

man 1 rtld

[lungsi]

Yes it will be helpful to know what libraries that are "needed" but I could not figure it out so I used the crude method of trying to load the libraries (mentioned above) as

        System.loadLibrary("openblas");
        System.loadLibrary("Atlas-0.5");
        System.loadLibrary("quadmath");
        System.loadLibrary("gfortran");
        System.loadLibrary("quadmath-0");
        System.loadLibrary("gfortran-5");
        System.loadLibrary("gcc_s");
        System.loadLibrary("libgomp");
        System.loadLibrary("libthr");

I continue to get the error. Having spent considerable time just so I can use matrix multiplication (i.e dgemm since this native method is being invoked) I am lost.

I even tried creating my own openblas interface but I got stuck at calling the native functions. I went in thinking the native functions are in libopenblas.so and I successfully loaded it with (clojure.lang.RT/loadLibrary "openblas") (or in Java static{ System.loadLibrary("openblas"); })

For instance

public class NativeBlas {
    private static int[] intDummy;
    private static double[] doubleDummy;
    private static float[] floatDummy;

    public NativeBlas() {
    }

    public static native void dgemm(char var0, char var1, int var2, int var3, int var4, double var5, double[] var7, int var8, int var9, double[] var10, int var11, int var12, double var13, double[] var15, int var16, int var17);

    static {
        System.loadLibrary("openblas");
        intDummy = new int[1];
        doubleDummy = new double[1];
        floatDummy = new float[1];
    }
}

Once OpenBlas is installed how do I access the native functions via the JVM?

Your manual describes the dgemm call tree to be

interface/gemm.c
        │
driver/level3/level3.c
        │
gemm assembly kernels at kernel/

Rather than (re)compile the source, How can I access dgemm from the installed library?

[martin-frbg]

The developer manual describes how the call is handled within OpenBLAS, which should be irrelevant to calling it from your code. The user manual has examples for calling the C interface (cblas_dgemm) or the Fortran interface (dgemm_) that used to be
the "native" code in the original reference implementation of BLAS. You will most likely want to call the C interface from Java,

[lungsi]

I just learned that the question I posed above is a problem of Foreign Function Interface: in java this is done with API's like JNI or JNR-FFI.

However, if I were to use the native methods of OpenBLAS in Clojure (via java) with the project setup as

/lein-native
|--build ---- script to compile c code; set JAVA_HOME as needed to compile on other machines
|--/doc
|--/resources
|--/src
|  |--/java
|  |  `--Mygemm.java ---- wrapper for C code
|  |--/lein_native ---- clojure code
|  |  `--core.clj ---- contains main, imports the java code and dynamically loads the native shared library
|  `--native ---- native source code
|     `--/linux
|        `--/x86_64
|           `--Mygemm.c -- native code calling ncurses
|           `--Mygemm.h
`--/target
   `--/bin
      `--/FreeBSD
         `--/x86_64
            `--libMyNative.so ---- Java requires native libaries be named in this format (extension will vary by OS)

and then run lein run, the build script looking something like

#!/bin/bash

#######################################################
##### CONSTANTS
#######################################################
OS=FreeBSD
ARCH=x86_64
SRC_DIR=src/native/
BIN_DIR=target/bin/
SRC_PATH=${SRC_DIR}$OS/$ARCH/
BIN_PATH=${BIN_DIR}$OS/$ARCH/
##### available user commands
CMD_BUILD=build
CMD_CLEAN=clean

#######################################################
##### VARIABLES
#######################################################
COMMAND=${1:-build}

#######################################################
##### FUNCTIONS
#######################################################
validate_inputs(){
  # check global "inputs" e.g. env variables
  if [ -z "$JAVA_HOME" ] && [ -d "$JAVA_HOME" ]; then
    echo JAVA_HOME not set, exiting
    exit 1
  fi
}

setup(){
  # create directory to hold compiled code
  if [ ! -d "$BIN_PATH" ]; then
    mkdir -p "$BIN_PATH"
  fi
}

##### command functions
build(){
  # compile
  g++ "$SRC_PATH/MyNative.c" -I "$JAVA_HOME/include" -I "$JAVA_HOME/include/FreeBSD" -l ncurses -shared -o "$BIN_PATH/libMyNative.so" -fPIC
}

clean(){
  rm -rf "$BIN_DIR"
}

#######################################################
##### LOGIC
#######################################################
validate_inputs
setup
eval $COMMAND

What do you think of placing OpenBLAS/interface/gemm.c -> Mygemm.c?

Also, can I still continue to take advantage of the call tree

interface/gemm.c
        │
driver/level3/level3.c
        │
gemm assembly kernels at kernel/

[martin-frbg]

No, please do not copy individual source files from OpenBLAS, this is not going to work. Use the regular API - your Mygemm wrapper should call the cblas_dgemm function that is in libopenblas.

[lungsi]

Gotcha! I just want to get my numerical simulations to work. I am just fishing for the most optimal solution :-)

It sounds like pointing to the libopenblas which is available after installing pkg install openblas

$ pkg info -lx openblas
openblas-0.3.10,1:
        /usr/local/include/cblas.h
        /usr/local/include/f77blas.h
        /usr/local/include/lapack.h
        /usr/local/include/lapacke.h
        /usr/local/include/lapacke_config.h
        /usr/local/include/lapacke_mangling.h
        /usr/local/include/lapacke_utils.h
        /usr/local/include/openblas_config.h
        /usr/local/lib/cmake/openblas/OpenBLASConfig.cmake
        /usr/local/lib/cmake/openblas/OpenBLASConfigVersion.cmake
        /usr/local/lib/libopenblas.a
        /usr/local/lib/libopenblas.so
        /usr/local/lib/libopenblas.so.0
        /usr/local/lib/libopenblasp-r0.3.10.a
        /usr/local/lib/libopenblasp-r0.3.10.so
        /usr/local/libdata/pkgconfig/openblas.pc
        /usr/local/share/licenses/openblas-0.3.10,1/BSD3CLAUSE
        /usr/local/share/licenses/openblas-0.3.10,1/LICENSE
        /usr/local/share/licenses/openblas-0.3.10,1/catalog.mk

ought to provide access to all the the BLAS and LAPACK functions (for instance, cblas_dgemm).

[brada4]

Could you run ldd against jblas.so that jblas build (tailored for OpenBLAS instead of their default ATLAS) produced?

[lungsi]

Since jblas is installed as a dependency in Clojures' project.clj - which is basically a .jar file pulled from maven locally into ~/.m2/repository/org/jblas/jblas/1.2.5 - there is no jblas.so.

So I don't know if if it's even possible to run ldd on jblas.

The dependencies defined inside the project.clj looks like

  :dependencies [[org.clojure/clojure "1.10.0"]
                 [org.jblas/jblas "1.2.5"]]

[brada4]

You need to use configure and make to build native library. Maven builds only java wrapper that says you are missing out native library. It is like first paragraph in their readme.md

[lungsi]

I see ...

If you want to build jblas from the sources including the native part, you need to set up quite a few things:

You will need some implementation of blas and lapack. jblas is tested with either plain lapack, or ATLAS (http://math-atlas.sourceforge.net/). You also need the Fortran sources for BLAS and LAPACK, available, for example from http://www.netlib.org/lapack/lapack-lite-3.1.1.tgz.

And probably do (in place of the strike through lines above) something like

$ git clone  [email protected]:mikiobraun/jblas.git
$ cd jblas
$ ./configure --static-libs --libpath=/opt/OpenBLAS/lib --libs=openblas --download-lapack --build-type=openblas
$ make
$ ant static-lean-jar

But what about this

In principle, all you need is the jblas-1.2.5.jar in your classpath. jblas-1.2.5.jar will then automagically extract your platform dependent native library to a tempfile and load it from there. You can also put that file somewhere in your load path ($LD_LIBRARY_PATH for Linux, %PATH for Windows).

Doesn't this mean that my installed OpenBLAS library path - presumably /usr/local/lib/libopenblas.so which is not in the "platform dependent native library tempfile path of jblas" - when loaded using, say, ldconfig -m /path/to/libs should work using just the jblas-1.2.5.jar?

[brada4]

I think the problem should be resolved over time around there:
jblas-project/jblas#127
PS I found it challenging to build jblas.so which turned out static lib on Linux
PPS no freebsd inside docker.

[mikiobraun]

Thanks for cross linking this thread, more information on here.

So first of all, jblas has only been tested on Linux, Windows and Mac, so I am not surprised it doesn't work with freebsd.

Second of all, you can also try it with configure --dynamic-libs. This does not try to link in static files but expects files to sit in the operating system. If you have an openblas installation that works, this should also work.

Getting to --static-libs, that is, copying all the necessary libraries into the jar would then be the next step.

Above you asked what the libname is, but that is OS dependent, so I also don't know how it works with freebsd.

Sorry I cannot be of more help!

[brada4]

Thank you for taking over, we were getting nuts already that nothing works, kudos for instant fix at other thread.