makeself "--posix" argument in tar for increased compatibility actually broke compatibility in Alpine Linux (BusyBox)

[ntx-ben]

Using makeself v2.4.3 in Alpine Linux 3.12:

Header is 682 lines long
About to compress 301096 KB of data...
Adding files to archive named "<REDACTED>"...

tar: unrecognized option: posix
BusyBox v1.31.1 () multi-call binary.
Usage: tar c|x|t [-ZzJjahmvokO] [-f TARFILE] [-C DIR] [-T FILE] [-X FILE] [--exclude PATTERN]... [FILE]...
Create, extract, or list files from a tar file
	c	Create
	x	Extract
	t	List
	-f FILE	Name of TARFILE ('-' for stdin/out)
	-C DIR	Change to DIR before operation
	-v	Verbose
	-O	Extract to stdout
	-m	Don't restore mtime
	-o	Don't restore user:group
	-k	Don't replace existing files
	-Z	(De)compress using compress
	-z	(De)compress using gzip
	-J	(De)compress using xz
	-j	(De)compress using bzip2
	-a	(De)compress using lzma
	-h	Follow symlinks
	-T FILE	File with names to include
	-X FILE	File with glob patterns to exclude
	--exclude PATTERN	Glob pattern to exclude

Reverting to v2.4.2 for use in Alpine Linux.

[realtime-neil]

Yikes. Okay, I'm remembering #238 and looking at these...

• https://manpages.debian.org/buster/tar/tar.1.en.html
• https://www.freebsd.org/cgi/man.cgi?query=tar
• https://boxmatrix.info/wiki/Property:tar

...and I'm wondering what The Right Thing to do is.

[realtime-neil]

This is not the first time the limitations of busybox tools have affected makeself. Readers will recall the problem with busybox dd in #161 .

[realtime-neil]

@megastep this can probably be fixed this like you did in #228; i.e., testing that tar supports the --posix flag before attempting to use it. Although, this re-introduces the problem in #238 --- platforms like Alpine will create GNU archives and platforms like FreeBSD will be (apparently) unable to extract them.

Edit: changed ustar to GNU, because that's what my busybox tar is producing.

[ntx-ben]

Upon further investigation, it seems even previous versions are incompatible with busybox' tar command. The r flag is also invalid.

[realtime-neil]

@ntx-ben you are correct. I guess it would be more appropriate to say makeself on Alpine remains broken since d22a2da because busybox tar lacks a r option.

I note that the r "operand" is described here: https://pubs.opengroup.org/onlinepubs/007908799/xcu/tar.html

[realtime-neil]

For those wondering why we append rather than create:

The utility named by utility shall be executed one or more times until the end-of-file is reached or the logical end-of file string is found.

The generated command line length shall be the sum of the size in bytes of the utility name and each argument treated as strings, including a null byte terminator for each of these strings. The xargs utility shall limit the command line length such that when the command line is invoked, the combined argument and environment lists (see the exec family of functions in the System Interfaces volume of POSIX.1-2017) shall not exceed {ARG_MAX}-2048 bytes. Within this constraint, if neither the -n nor the -s option is specified, the default command line length shall be at least {LINE_MAX}.

--- https://pubs.opengroup.org/onlinepubs/9699919799/utilities/xargs.html#tag_20_158_03

tldr; the command length limit is allowed to force multiple xargs tar ... invocations.

[megastep]

If we never really had it working properly on Busybox, I'd be OK with keeping the increased POSIX compatibility instead as it is more important. Also this is mostly for generating the archives - it's not a big ask to create the archives on something else than Busybox, as long as they can be extracted there.

[rbcrwd]

I'm still debugging, but this also appears to have broken extraction on Solaris 10 and 11

[megastep]

I'd be surprised if it broke extraction rather than just the creation of archives? Can you confirm? Solaris support is a must for me.

[rbcrwd]

Testing with the following example from #238 executed in the root of the makeself git repo:

./makeself.sh --notemp . makeself.run "Makeself by Stephane Peter" echo "Makeself has extracted itself"

I see the following result on a vanilla Solaris 11 x86 system:

root@solaris11:/tmp# ./makeself.run
+ ./makeself.run
Verifying archive integrity...  100%   MD5 checksums are OK. All good.
Uncompressing Makeself by Stephane Peter  100%   ... Extraction failed.
Terminated

[megastep]

Was the archive also created on the Solaris system?

[rbcrwd]

No, it's created on an OS X system, 10.15.7:

# tar --version
bsdtar 3.3.2 - libarchive 3.3.2 zlib/1.2.11 liblzma/5.0.5 bz2lib/1.0.6

[rbcrwd]

I'm seeing the same fundamental result when generating the same .run on a Linux VM with GNU tar-1.33.

[megastep]

Mmmh I wonder if some of the other formats might be more compatible, such as ustar, cpio or shar

[rbcrwd]

I've extracted the original tarball (generated on OS X) on Solaris, and have been experimenting with it. It lists just fine, although there are obvious extensions, like the PaxHeader/ directory. However, when extracting the same way you are, I see the following:

# gzip -dc makeself.tar.gz | tar xpvf -
tar: ./.git/PaxHeader/FETCH_HEAD: typeflag 'x' not recognized, converting to regular file
x ./.git/PaxHeader/FETCH_HEAD, 128 bytes, 1 tape blocks
x ./.git/FETCH_HEAD, 211 bytes, 1 tape blocks
tar: ./.git/PaxHeader/HEAD: typeflag 'x' not recognized, converting to regular file
x ./.git/PaxHeader/HEAD, 90 bytes, 1 tape blocks
x ./.git/HEAD, 23 bytes, 1 tape blocks
...

Each file goes on like that, it looks like Solaris doesn't support those type extensions. With the exception of the extra directory the files look fine, but tar's exit code is 1, signifying that it failed. This triggers the failure branch in the UnTAR() function in makeself-header.sh.

[rbcrwd]

I've tested with release-2.4.2 and it doesn't exhibit the same behavior. Although there was a lot of delta between 2.4.2 and 2.4.3, I, like the OP, suspect it was the --posix tar creation flag that, while adding better BSD support, appears to have broken Solaris.

[realtime-neil]

@rbcrwd What's the tar being used on the Solaris host doing the extraction?

[realtime-neil]

@megastep

Mmmh I wonder if some of the other formats might be more compatible, such as ustar, cpio or shar

The last time I went looking, iirc, there's no easy way to get a generic tar utility to select the archive format.

Recent GNU and FreeBSD tar commands certainly sport the --posix flag:

• Mac OS X v11.* and v10.* ship with GNU tar v1.17 (released 2007.06.08) which supports --posix.
• @rbcrwd is using bsdtar from libarchive v3.3.2 (released 2017.07.09) which supports --posix.

The OpenSolaris tar man page mentions neither the --posix flag nor the POSIX.1-2001 "pax" format. This might imply support for the ustar format only.

References:

• https://www.gnu.org/software/tar/manual/tar.html#Option-Summary
• https://www.freebsd.org/cgi/man.cgi?query=tar
• https://www.unix.com/man-page/opensolaris/1/tar
• https://opensource.apple.com/source/gnutar/gnutar-455/
• ftp://ftp.gnu.org/gnu/tar/
• https://github.com/libarchive/libarchive/tags

[rbcrwd]

On the extraction side, my test Solaris VMs are using whatever's first in the default $PATH:

• Solaris 10: SunOS solaris10 5.10 Generic_147148-26 i86pc i386 i86pc: /usr/sbin/tar (no --version support)
• Solaris 11: SunOS solaris11 5.11 11.2 i86pc i386 i86pc: /usr/bin/tar (also no --version support)

[edit]
Both are non-production systems, just installed with the bare minimum of "whatever came from a CD installation".

[rbcrwd]

Any thoughts on how to proceed? I've a modified local version that suffices for now, but am curious what next steps are, and how I can help.

I recognize that the purpose behind the --posix flag was to increase compatibility, but for what platforms?

[megastep]

The goal is to have Makeself produce archives that are as cross-platform as possible. So if POSIX doesn't achieve that goal, we might need to investigate what works better in practice, like maybe ustar - I'm not as concerned with the ability to produce archives on all supported platforms, mostly about their ability to be extracted correctly.

[rbcrwd]

Understood on the goal and compatibility, I'm just curious what OSes were targeted (if any particular ones) with the --posix addition. Specifically, I'm wondering how I might help address those, since it appears to have had a moderately opposite effect.

[realtime-neil]

@rbcrwd the OS in question was FreeBSD --- that story is told here: #238

tldr; tar on some (unnamed) Linux made a ustar archive the FreeBSD tar couldn't extract.

IMHO, busybox has a demonstrated history (ahem, #228) of twiddling commands into POSIX non-compliance. I'm sure there's good reasons for doing so, but the result is fundamentally at odds with the goals of the Makeself project. @megastep please correct me here if I'm overreaching.

[megastep]

Correct; to be fair we can always find some way to work around weird command syntax in the scripts, however it's more of a problem if the raw tar data is not handled by the OS. So that's the limitation we need to work around here so we can produce tar archives that can be extracted on the wide variety of Unix/Linux/BSD systems targeted, ideally without requiring third-party tools to be installed (such as GNU tar)

[rbcrwd]

Thanks for the context.

What about using --format to specify either ustar (more desirable, acknowledging the prior break) or v7 (more limited format)? I see that both GNU Tar and the Mac BSD tar support both of those options, although they're both coy about what their default is, at least in the man-page. That would hopefully keep an overeager GNU tar from defaulting to even more recent formats (e.g., gnutar).

Given that the pax format that --posix flag produce is an extension of the older ustar, it makes sense that the objectively ancient Solaris family doesn't support it. I see this in that I can hand-extract the tarballs, but the extra type flags and metadata files cause the extraction to fail.

Finally, sorry for apparently hijacking a BusyBox ticket. I allowed myself to rush too much.

[realtime-neil]

@rbcrwd how do you feel about a --tar-format flag that populates a TAR_FORMAT_FLAG which defaults to --posix? Makeself would still produce PAX archives by default, but it could be overriden by the creator of a self-extracting archive when targeting recipients lacking a tar with PAX support.

[rbcrwd]

I could work with specifying the tar format, but unless you have outside context, the details in #238 don't seem to support the assertion that a ustar format archive was to blame on BSD (or even where @nunotexbsd generated their tarball that failed on BSD). I'd be quite surprised if ustar was to blame, as it's pretty much universal in all tar implementations I'm aware of (even some hoary old private C++ white-box implementations I work with).

For that matter, since at least 2003 GNU tar has defaulted to the GNU format. That seems to imply that Linux->BSD .run was just silently broken since then, but that's another rabbit-hole I have yet to dig through. Given that and BusyBox's similar default, that seems to stand a much higher likelihood of being the cause of #238.

All that to say: sure, I'm good with a --tar-format flag, but think it should default to the more-compatible ustar rather than pax. I don't think makeself is even exposing the extended attributes that the pax format adds, so am unsure what benefits it would provide.

[realtime-neil]

@rbcrwd point taken; ustar archives are our legacy and it makes sense to choose that format by default. I'm going to start work on a --tar-format flag.

I'm also going to put the test units through my FreeBSD VM --- I'd be similarly surprised if the libarchive tar shipping with such an established unix didn't support ustar.

[megastep]

As a side note, we might be able to run CI tests on FreeBSD via this action: https://github.com/marketplace/actions/freebsd-vm

One of the things I'd like to test for is portability between OSes, so maybe export a test archive as an artifact on Linux and try to extract it on a FreeBSD VM.