littler installation fails in these days

[eitsupi]

Since January 16, littler seems to be failing to install on rocker/r-ver:devel

 #18 9.975 using C compiler: ‘gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0’
#18 10.35 gcc -o r -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -I/usr/local/lib/R/include littler.c -L/usr/local/lib -Wl,--export-dynamic -fopenmp -L/usr/local/lib -L/usr/local/lib/R/lib -lR -lpcre2-8 -ldeflate -lzstd -llzma -lbz2 -lz -lrt -ldl -lm -licuuc -licui18n -lblas -llapack -Wl,-rpath,/usr/local/lib/R/lib -Wl,-rpath,/usr/local/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/jvm/java-21-openjdk-amd64/lib/server -Wl,-rpath,/usr/local/lib/R/lib -Wl,-rpath,/usr/local/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/jvm/java-21-openjdk-amd64/lib/server -Wl,-rpath,/usr/local/lib/R/lib -Wl,-rpath,/usr/local/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/jvm/java-21-openjdk-amd64/lib/server -Wl,-rpath,/usr/local/lib/R/lib -Wl,-rpath,/usr/local/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/jvm/java-21-openjdk-amd64/lib/server 
#18 10.53 /usr/bin/ld: cannot find -lzstd: No such file or directory
#18 10.53 collect2: error: ld returned 1 exit status
#18 10.53 make: *** [Makevars:31: r] Error 1
#18 10.53 ERROR: compilation failed for package ‘littler’

https://github.com/rocker-org/rocker-versioned2/actions/runs/12799426223

@eddelbuettel Any thoughts?

[eddelbuettel]

Uh-oh. Will take a look. Maybe have to with the changes to compression.

[eddelbuettel]

Hm. configure.ac follows RHOME so should use R-devel. From src/Makevars.in to src/Makevars is all computed based the installed version.

It builds under r-devel at CRAN too.

Works fine for me here (see below) under a freshly-remade R-devel. Can you check if you container has libzstd-dev ?

edd@rob:~/git/littler(master)$ rm -f src/r                                                                
edd@rob:~/git/littler(master)$ RD CMD INSTALL .                                                           
* installing to library ‘/usr/local/lib/R-devel/lib/R/library’
* installing *source* package ‘littler’ ...
** this is package ‘littler’ version ‘0.3.20.2’
** using staged installation                                                                             
checking for gcc... ccache gcc-14
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no                                                            
checking for suffix of object files... o                                                                 
checking whether the compiler supports GNU C... yes                                                      
checking whether ccache gcc-14 accepts -g... yes
checking for ccache gcc-14 option to enable C11 features... none needed
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking for getopt.h... yes
checking for sys/mkdev.h... no
checking for sys/sysmacros.h... yes
checking for setenv... yes
checking for clock_gettime... yes
checking for realtime... no
checking for gettimeofday... yes
checking for time... yes
checking if R was built as a shared library... yes
checking for install_name_tool... no
configure: creating ./config.status
config.status: creating src/Makevars
config.status: creating src/gitversion.h
config.status: creating src/config.h
config.status: src/config.h is unchanged
** libs
using C compiler: ‘gcc-14 (Ubuntu 14.2.0-4ubuntu2) 14.2.0’
ccache gcc-14 -o r -O3 -Wall -pipe -pedantic -I/usr/local/lib/R-devel/lib/R/include littler.c -L/usr/local/lib -Wl,--export-dynamic -fopenmp -L/usr/local/lib -L/usr/local/lib/R-devel/lib/R/lib -lR -lpcre2-8 -ldeflate -lzstd -llzma -lbz2 -lz -ltirpc -lrt -ldl -lm -licuuc -licui18n -L/usr/local/lib/R-devel/lib/R/lib -lRblas -L/usr/local/lib/R-devel/lib/R/lib -lRlapack -Wl,-rpath,/usr/local/lib/R-devel/lib/R/lib -Wl,-rpath,/usr/local/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/jvm/java-21-openjdk-amd64/lib/server -Wl,-rpath,/usr/local/lib/R-devel/lib/R/lib -Wl,-rpath,/usr/local/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/jvm/java-21-openjdk-amd64/lib/server -Wl,-rpath,/usr/local/lib/R-devel/lib/R/lib -Wl,-rpath,/usr/local/lib -Wl,-rpath,/usr/lib/x86_64-linux-gnu -Wl,-rpath,/usr/lib/jvm/java-21-openjdk-amd64/lib/server 
*
* new binary r installed in bin/ subdirectory
* consider adding a symbolic link from, e.g., /usr/local/bin
* on OS X, you may have to name this 'lr' instead
* see the littler-faq vignette for more details
*
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (littler)
edd@rob:~/git/littler(master)$ 

[eddelbuettel]

Can you show me what extSoftVersion() shows on your R-devel build?

> extSoftVersion()
                                          zlib                                          bzlib 
                                       "1.3.1"                           "1.0.8, 13-Jul-2019" 
                                            xz                                     libdeflate 
                                       "5.6.2"                                         "1.21" 
                                          PCRE                                            ICU 
                            "10.42 2022-12-11"                                         "74.2" 
                                           TRE                                          iconv 
                     "TRE 0.8.0 R_fixes (BSD)"                                   "glibc 2.40" 
                                      readline                                           BLAS 
                                         "8.2" "/usr/local/lib/R-devel/lib/R/lib/libRblas.so" 
> R.version
               _                                                 
platform       x86_64-pc-linux-gnu                               
arch           x86_64                                            
os             linux-gnu                                         
system         x86_64, linux-gnu                                 
status         Under development (unstable)                      
major          4                                                 
minor          5.0                                               
year           2025                                              
month          01                                                
day            18                                                
svn rev        87593                                             
language       R                                                 
version.string R Under development (unstable) (2025-01-18 r87593)
nickname       Unsuffered Consequences                           
> 

PS Mind you that doesn't even show zstd. Hm.

[eitsupi]

This is occurring on CI and I do not know what is on the image.
But I don't think there is a libzstd-dev as there have been problems in the past without it (#393).
Is that suddenly needed?

[eddelbuettel]

Sorry, I have no idea. #393 is ancient. I live on (always current) Ubuntu (though I often take three months to get to the new six-month cycle) and never had any issue whatsover with littler, R, or fst. As CRAN is ok too I suspect that issue is one of your making.

Oddly, I also do not seem to pull libzstd in when build r-base for Debian/Ubuntu:

Build-Depends: gcc (>= 4:4.9.2-2), g++ (>= 4:4.9.2-2), gfortran (>= 4:4.9.2-2), libblas-dev, liblapack-dev, tcl8.6-dev, tk8.6-dev, bison, groff-base, libncurses-dev, libreadline-dev, debhelper-compat (= 13), texinfo, libbz2-dev, liblzma-dev, libpcre2-dev, libcurl4-openssl-dev | libcurl4-dev, xdg-utils, zlib1g-dev, libpng-dev, libjpeg-dev, libx11-dev, libxt-dev, x11proto-core-dev, libpango1.0-dev, libcairo2-dev, libtiff-dev, xvfb, xauth, xfonts-base, texlive-base, texlive-latex-base, texlive-plain-generic, texlive-fonts-recommended, texlive-fonts-extra, texlive-extra-utils, texlive-latex-recommended, texlive-latex-extra, default-jdk [!arm !hppa !kfreebsd-any !hurd-any] | openjdk-10-jdk [!arm !hppa !kfreebsd-any !hurd-any], mpack, bash-completion, libtirpc-dev
edd@rob:~/deb/r-base(master)$ 

and it is not in the LIBS etc in R's Makeconf whereas a few other compressors are

edd@rob:~/deb/r-base(master)$ grep -- -lz /etc/R/Makeconf
LIBS =  -lpcre2-8 -ldeflate -llzma -lbz2 -lz -ltirpc -lrt -ldl -lm -licuuc -licui18n
edd@rob:~/deb/r-base(master)$ 

Maybe your local build of R-devel brings it in, somehow? This came up recently on the list and as I recall Simon may now be looking into compressing data archives with it?

Maybe check $RHOME/etc/Makeconf if -lzstd is in there now and hence 'spills' over to what gets computed for your littler?

[eitsupi]

Thank you, I see, so you are saying this is due to the R update?
Certainly adding libzstd-dev (#900) now makes the build succeed.

[eddelbuettel]

If you follow the r-devel list (or was it r-package-devel ?) you may have seen the thread, it started with Jeroen comparing packages sizes and possible compression gains.

I saw #900, this is the right move to plug this hole. I would still like to know how you created it so if you have a moment maybe look into answering what I asked 20 minutes ago and inspect the $RHOME/etc/Makeconf from your R build?

PS I just check ./configure --help and I see no option to either set or unset zstd. Truly bizarre.

[eitsupi]

Thanks for the details.
In the meantime, I've merged, so the image will be pushed to DockerHub as rocker/r-ver:devel after half a day or so.
(I can't afford to build and investigate locally at the moment)

[eitsupi]

@eddelbuettel It seems that -lzstd is included in the Makeconf file of R.

❯ docker run --rm -it rocker/r-ver:devel@sha256:9f7186ac309c653b06cba533ae7053ae84802492155e6db976b06adbbe42919d bash
Unable to find image 'rocker/r-ver:devel@sha256:9f7186ac309c653b06cba533ae7053ae84802492155e6db976b06adbbe42919d' locally
docker.io/rocker/r-ver@sha256:9f7186ac309c653b06cba533ae7053ae84802492155e6db976b06adbbe42919d: Pulling from rocker/r-ver
de44b265507a: Pull complete
9369bd839aab: Pull complete
1437124bba2c: Pull complete
0e8ebb621c8b: Pull complete
2b26f2834e22: Pull complete
244ad7ec74ef: Pull complete
f19f786a91a6: Pull complete
Digest: sha256:9f7186ac309c653b06cba533ae7053ae84802492155e6db976b06adbbe42919d
Status: Downloaded newer image for rocker/r-ver@sha256:9f7186ac309c653b06cba533ae7053ae84802492155e6db976b06adbbe42919d
root@3395a23a53b8:/# R --version
R Under development (unstable) (2025-01-18 r87593) -- "Unsuffered Consequences"
Copyright (C) 2025 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under the terms of the
GNU General Public License versions 2 or 3.
For more information about these matters see
https://www.gnu.org/licenses/.

root@3395a23a53b8:/# grep -- -lz $R_HOME/etc/Makeconf
LIBS =  -lpcre2-8 -ldeflate -lzstd -llzma -lbz2 -lz -lrt -ldl -lm -licuuc -licui18n

[eitsupi]

Seems come from this?
r-devel/r-svn@4550323

[eddelbuettel]

Yes I suspected just that and had suggested you check the file.

But that is homegrown binary so you are on your own -- it is not my .deb package or its ubuntu port. So I think something must have changed on your side of building things (or in the sources). I does not affect r-release or r-devel for me.

PS And I built against svn rev r87593 which is after the commit you reference.

[eddelbuettel]

Tried once more with update SVN source. No mention in the configure log when building. When done I do see zstd mentioned eg in R CMD build --help under --compression=:

  --compression=        type of compression to be used on tarball:
                        "gzip" (default), "none", "bzip2", "xz", "zstd"

but per the note in today's R-devel changes RSS (off NEWS)

CHANGES IN R-devel NEW FEATURES

• There is some support for ‘zstd’ compression of tarballs in ‘tar()’ and ‘untar()’. (This depends on OS support of ‘libzstd’ or by ‘tar’.)

it may be picking up my system-level zstd? I honestly do not know. In any event, not littler issue in the sense that littler just uses what R itself has encoded from its build.