Add support for non-standard SquashFS #72

wcampbell0x2a · 2023-01-08T05:27:31Z

unblob supports some non standard squashfs binaries, we should also.

https://github.com/onekey-sec/unblob/tree/main/tests/integration/filesystem/squashfs

They have a fork of sasquash for adding support https://github.com/onekey-sec/sasquatch

qkaiser · 2023-01-16T14:09:48Z

Don't hesitate to reach out if you want some samples or support with this. I shared a table with squashfs v2, v3, v4, and non-standard format in our last talk (slide 35 at https://www.blackalps.ch/ba-22/files/talks/BlackAlps22-Kaiser.pdf).

wcampbell0x2a · 2023-01-16T14:20:12Z

Thanks! I'll see when I get around to this. But looks like a never ending adventure.

wcampbell0x2a · 2023-01-29T20:37:02Z

Some TP-Link gpl code has the following patch to the kernel

$ diff -bur ./fs/squashfs ~/projects/linux/fs/squashfs > patch

--- /fs/squashfs/xz_wrapper.c	2021-11-09 09:57:38.000000000 +0000
+++linux/fs/squashfs/xz_wrapper.c	2023-01-17 04:18:38.293147873 +0000
@@ -39,10 +39,8 @@
 };

 struct comp_opts {
-	__le32 flags;
-	__le16 bit_opts;
-	__le16 fb;
 	__le32 dictionary_size;
+	__le32 flags;
 };

 static void *squashfs_xz_init(struct squashfs_sb_info *msblk, void *buff,

- Add Kind, which stores information that is used as Ctx when Read/Write an image. - Keep regular squashfs working! This currently doesn't work, but can be tested with test files from unblob. ``` wget https://github.com/onekey-sec/unblob/blob/3c7e886e2616413a4d6109ba3d197f91c9596881/tests/integration/filesystem/squashfs/squashfs_v4_be/__input__/squashfs_v4.bin $ RUST_LOG=trace cargo test --release ``` I'm currently having some problems with parsing the lookup table, I must have something wrong. Or this image does something weird. For instance the fragment_table gets parsed as: frag_table: 0x0000ff, Which looks like from the superblock: ┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐ │00000000│ 73 71 73 68 00 00 00 05 ┊ 62 1f 5e 09 00 02 00 00 │sqsh⋄⋄⋄•┊b•^_⋄•⋄⋄│ │00000010│ 00 00 00 01 00 01 00 11 ┊ 00 c0 00 01 00 04 00 00 │⋄⋄⋄•⋄•⋄•┊⋄×⋄•⋄•⋄⋄│ │00000020│ 00 00 00 00 00 00 00 80 ┊ 00 00 00 00 00 00 01 33 │⋄⋄⋄⋄⋄⋄⋄×┊⋄⋄⋄⋄⋄⋄•3│ │00000030│ 00 00 00 00 00 00 01 2b ┊ ff ff ff ff ff ff ff ff │⋄⋄⋄⋄⋄⋄•+┊××××××××│ │00000040│ 00 00 00 00 00 00 00 79 ┊ 00 00 00 00 00 00 00 b5 │⋄⋄⋄⋄⋄⋄⋄y┊⋄⋄⋄⋄⋄⋄⋄×│ │00000050│ 00 00 00 00 00 00 00 ff ┊ 00 00 00 00 00 00 01 1d │⋄⋄⋄⋄⋄⋄⋄×┊⋄⋄⋄⋄⋄⋄••│ ^^^^^^^^^^^^^^^^^^^^^^^ Which when read, is null? 2023-02-28T02:03:14.130010Z INFO backhand::squashfs: Reading Fragments 2023-02-28T02:03:14.130011Z TRACE fragments:lookup_table: backhand::reader: seek: ff 2023-02-28T02:03:14.130015Z TRACE fragments:lookup_table: backhand::reader: [00, 00, 00, 00] 2023-02-28T02:03:14.130016Z TRACE fragments:lookup_table: backhand::reader: ptr: 00 │000000e0│ 01 68 02 4c c4 04 24 02 ┊ 00 0f 1e 10 41 80 10 00 │•h•L×•$•┊⋄•••A×•⋄│ │000000f0│ 00 00 00 00 00 00 60 00 ┊ 00 00 19 00 00 00 00 00 │⋄⋄⋄⋄⋄⋄`⋄┊⋄⋄•⋄⋄⋄⋄⋄│ │00000100│ 00 00 00 00 00 00 ed 00 ┊ 14 78 da 63 60 40 01 0a │⋄⋄⋄⋄⋄⋄×⋄┊•x×c`@•_│ See #72

- Add Kind, which stores information that is used as Ctx when Read/Write an image. - Keep regular squashfs working! This currently doesn't work, but can be tested with test files from unblob. ``` wget https://github.com/onekey-sec/unblob/blob/3c7e886e2616413a4d6109ba3d197f91c9596881/tests/integration/filesystem/squashfs/squashfs_v4_be/__input__/squashfs_v4.bin $ RUST_LOG=trace cargo test --release ``` I'm currently having some problems with parsing the lookup table, I must have something wrong. Or this image does something weird. For instance the fragment_table gets parsed as: frag_table: 0x0000ff, Which looks good from the superblock: ┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐ │00000000│ 73 71 73 68 00 00 00 05 ┊ 62 1f 5e 09 00 02 00 00 │sqsh⋄⋄⋄•┊b•^_⋄•⋄⋄│ │00000010│ 00 00 00 01 00 01 00 11 ┊ 00 c0 00 01 00 04 00 00 │⋄⋄⋄•⋄•⋄•┊⋄×⋄•⋄•⋄⋄│ │00000020│ 00 00 00 00 00 00 00 80 ┊ 00 00 00 00 00 00 01 33 │⋄⋄⋄⋄⋄⋄⋄×┊⋄⋄⋄⋄⋄⋄•3│ │00000030│ 00 00 00 00 00 00 01 2b ┊ ff ff ff ff ff ff ff ff │⋄⋄⋄⋄⋄⋄•+┊××××××××│ │00000040│ 00 00 00 00 00 00 00 79 ┊ 00 00 00 00 00 00 00 b5 │⋄⋄⋄⋄⋄⋄⋄y┊⋄⋄⋄⋄⋄⋄⋄×│ │00000050│ 00 00 00 00 00 00 00 ff ┊ 00 00 00 00 00 00 01 1d │⋄⋄⋄⋄⋄⋄⋄×┊⋄⋄⋄⋄⋄⋄••│ ^^^^^^^^^^^^^^^^^^^^^^^ Which when read, is null? 2023-02-28T02:03:14.130010Z INFO backhand::squashfs: Reading Fragments 2023-02-28T02:03:14.130011Z TRACE fragments:lookup_table: backhand::reader: seek: ff 2023-02-28T02:03:14.130015Z TRACE fragments:lookup_table: backhand::reader: [00, 00, 00, 00] 2023-02-28T02:03:14.130016Z TRACE fragments:lookup_table: backhand::reader: ptr: 00 │000000e0│ 01 68 02 4c c4 04 24 02 ┊ 00 0f 1e 10 41 80 10 00 │•h•L×•$•┊⋄•••A×•⋄│ │000000f0│ 00 00 00 00 00 00 60 00 ┊ 00 00 19 00 00 00 00 00 │⋄⋄⋄⋄⋄⋄`⋄┊⋄⋄•⋄⋄⋄⋄⋄│ │00000100│ 00 00 00 00 00 00 ed 00 ┊ 14 78 da 63 60 40 01 0a │⋄⋄⋄⋄⋄⋄×⋄┊•x×c`@•_│ See #72

- Support non standard images. Read and Write! - Add Kind for including endian and version information - Add Kind: LE_V4_0 for linux kernel and upstream squashfs-tools support - Add Kind: BE_V4_0 for custom vendor firmware - Add Kind: AVM_BE_V4_0 for Fritz!OS firmware support. Added because it's interesting, as they kept some of it still BE. - Change lookup table from u32 to u64. This was working in LE, but is very wrong for LE! See #72

wcampbell0x2a · 2023-05-01T12:15:44Z

@qkaiser starting to look into supporting custom compression algorithms. Do they use the same custom compression for everything, as in all data and metadata blocks?

qkaiser · 2023-05-02T14:30:51Z

@qkaiser starting to look into supporting custom compression algorithms. Do they use the same custom compression for everything, as in all data and metadata blocks?

As far as I remember, yes. The only place we've seen something different is when they encrypt/obfuscate headers. So yeah, same compression applied to both data and metadata blocks.

wcampbell0x2a · 2023-05-10T04:28:48Z

Some TP-Link gpl code has the following patch to the kernel

$ diff -bur ./fs/squashfs ~/projects/linux/fs/squashfs > patch

--- /fs/squashfs/xz_wrapper.c	2021-11-09 09:57:38.000000000 +0000
+++linux/fs/squashfs/xz_wrapper.c	2023-01-17 04:18:38.293147873 +0000
@@ -39,10 +39,8 @@
 };

 struct comp_opts {
-	__le32 flags;
-	__le16 bit_opts;
-	__le16 fb;
 	__le32 dictionary_size;
+	__le32 flags;
 };

 static void *squashfs_xz_init(struct squashfs_sb_info *msblk, void *buff,

This is an openwrt patch: openwrt/openwrt@f97ad87. Looks like that... mistake is now fixed and that patch not active: https://github.com/openwrt/openwrt/tree/master/tools/squashfs4/patches.

qkaiser · 2023-05-10T06:31:38Z

Looks like I need to add OpenWRT to the queue.

Is there a way to differentiate between a filesystem built with the patch and a filesystem built without the patch ? Any heuristics that you can use ?

wcampbell0x2a · 2023-05-10T12:24:34Z

Yep, its a different block length

- Support OpenWrt XZ custom compression options - Enable output of these options during FilesystemWriter See #72

wcampbell0x2a added the C-enhancement New feature or request label Jan 8, 2023

wcampbell0x2a mentioned this issue Feb 28, 2023

WIP: Support non-standard images #174

Merged

wcampbell0x2a added a commit that referenced this issue May 11, 2023

Add support for OpenWrt compression options

319107c

- Support OpenWrt XZ custom compression options - Enable output of these options during FilesystemWriter See #72

wcampbell0x2a mentioned this issue May 11, 2023

Add support for OpenWrt compression options #239

Merged

wcampbell0x2a added a commit that referenced this issue May 11, 2023

Add support for OpenWrt compression options

d5908cc

- Support OpenWrt XZ custom compression options - Enable output of these options during FilesystemWriter See #72

wcampbell0x2a added a commit that referenced this issue May 11, 2023

Add support for OpenWrt compression options

a6ef3a4

- Support OpenWrt XZ custom compression options - Enable output of these options during FilesystemWriter See #72

XVilka mentioned this issue May 24, 2023

Add optional feature to use lzma-rs instead of c bindings #95

Open

wcampbell0x2a added A-support-quashfs-v3 A-support-vendor-specific-format Support vendor specific compression/format labels Nov 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for non-standard SquashFS #72

Add support for non-standard SquashFS #72

wcampbell0x2a commented Jan 8, 2023

qkaiser commented Jan 16, 2023

wcampbell0x2a commented Jan 16, 2023

wcampbell0x2a commented Jan 29, 2023

wcampbell0x2a commented May 1, 2023

qkaiser commented May 2, 2023

wcampbell0x2a commented May 10, 2023

qkaiser commented May 10, 2023

wcampbell0x2a commented May 10, 2023

Add support for non-standard SquashFS #72

Add support for non-standard SquashFS #72

Comments

wcampbell0x2a commented Jan 8, 2023

qkaiser commented Jan 16, 2023

wcampbell0x2a commented Jan 16, 2023

wcampbell0x2a commented Jan 29, 2023

wcampbell0x2a commented May 1, 2023

qkaiser commented May 2, 2023

wcampbell0x2a commented May 10, 2023

qkaiser commented May 10, 2023

wcampbell0x2a commented May 10, 2023