Current limitations of the ModSecurity v3/dev/wasm-experimental branch?

[dspeg]

What are the limitations from the below changes to the ModSecurity V3 branch?
owasp-modsecurity/ModSecurity@v3/master...v3/dev/wasm-experimental

Did we choose the ModSecurity build options mainly to reduce the wasm binary size or to make the build successful?

emconfigure ./configure --without-yajl --without-geoip --without-libxml --without-curl \
    --without-lua --disable-shared --disable-examples --disable-libtool-lock \
    --disable-debug-logs  --disable-mutex-on-pm --without-lmdb --without-maxmind \
    --without-ssdeep --without-pcre2 --with-pcre=./pcre-config \

For examples:
--without-libxml: I tried removing this switch, and this caused build errors in libxml source code.
--without-geoip: Any comments?

[leyao-daily]

Hi, I have tested the commands, it works. Can you attach your steps or errors.

[dspeg]

I removed the "--without-libxml" switch and changed the file https://github.com/intel/modsecurity-wasm-filter/blob/main/wasmplugin/Dockerfile to also install libxml2

FROM ubuntu:bionic
# basics
SHELL ["/bin/bash", "-c"] 
ARG http_proxy 
ARG https_proxy 
RUN apt-get update && apt-get install -y --no-install-recommends apt-utils ca-certificates \ 
    && apt-get autoremove -y \ 
    && apt-get clean \ 
    && apt-get install -y --no-install-recommends \
    software-properties-common apt-transport-https git wget curl pkg-config \
    autoconf autotools-dev automake libtool cmake python zlib1g-dev libpcre3 \
    libpcre3-dev gcc-7 g++-7 cpp-7 gcc python3 python3-distutils libxml2 libxml2-dev \
    && rm -rf /var/lib/apt/lists/* \
    && cd $HOME

Then there are the following output from "docker build" command:

#15 27.24 make[1]: Leaving directory '/root/ModSecurity/others'
#15 27.24 Making all in src
#15 27.25 make[1]: Entering directory '/root/ModSecurity/src'
#15 27.25 make  all-recursive
#15 27.26 make[2]: Entering directory '/root/ModSecurity/src'
#15 27.27 make[3]: Entering directory '/root/ModSecurity/src'
#15 27.27 /bin/bash ../libtool  --tag=CXX   --mode=compile /root/emsdk/upstream/emscripten/em++ -DHAVE_CONFIG_H -I.  -std=c++11 -I.. -g -I../others -fPIC -O3 -I../headers    -DNO_LOGS=1    -I/root/ModSecurity/include     -I/usr/include/libxml2 -DWITH_LIBXML2   -g -O2 -MT libmodsecurity_la-transaction.lo -MD -MP -MF .deps/libmodsecurity_la-transaction.Tpo -c -o libmodsecurity_la-transaction.lo `test -f 'transaction.cc' || echo './'`transaction.cc
#15 27.27 /bin/bash ../libtool  --tag=CXX   --mode=compile /root/emsdk/upstream/emscripten/em++ -DHAVE_CONFIG_H -I.  -std=c++11 -I.. -g -I../others -fPIC -O3 -I../headers    -DNO_LOGS=1    -I/root/ModSecurity/include     -I/usr/include/libxml2 -DWITH_LIBXML2   -g -O2 -MT libmodsecurity_la-anchored_set_variable.lo -MD -MP -MF .deps/libmodsecurity_la-anchored_set_variable.Tpo -c -o libmodsecurity_la-anchored_set_variable.lo `test -f 'anchored_set_variable.cc' || echo './'`anchored_set_variable.cc
#15 27.28 /bin/bash ../libtool  --tag=CXX   --mode=compile /root/emsdk/upstream/emscripten/em++ -DHAVE_CONFIG_H -I.  -std=c++11 -I.. -g -I../others -fPIC -O3 -I../headers    -DNO_LOGS=1    -I/root/ModSecurity/include     -I/usr/include/libxml2 -DWITH_LIBXML2   -g -O2 -MT libmodsecurity_la-anchored_variable.lo -MD -MP -MF .deps/libmodsecurity_la-anchored_variable.Tpo -c -o libmodsecurity_la-anchored_variable.lo `test -f 'anchored_variable.cc' || echo './'`anchored_variable.cc
#15 27.28 /bin/bash ../libtool  --tag=CXX   --mode=compile /root/emsdk/upstream/emscripten/em++ -DHAVE_CONFIG_H -I.  -std=c++11 -I.. -g -I../others -fPIC -O3 -I../headers    -DNO_LOGS=1    -I/root/ModSecurity/include     -I/usr/include/libxml2 -DWITH_LIBXML2   -g -O2 -MT libmodsecurity_la-modsecurity.lo -MD -MP -MF .deps/libmodsecurity_la-modsecurity.Tpo -c -o libmodsecurity_la-modsecurity.lo `test -f 'modsecurity.cc' || echo './'`modsecurity.cc
#15 27.31 libtool: compile:  /root/emsdk/upstream/emscripten/em++ -DHAVE_CONFIG_H -I. -std=c++11 -I.. -g -I../others -fPIC -O3 -I../headers -DNO_LOGS=1 -I/root/ModSecurity/include -I/usr/include/libxml2 -DWITH_LIBXML2 -g -O2 -MT libmodsecurity_la-anchored_set_variable.lo -MD -MP -MF .deps/libmodsecurity_la-anchored_set_variable.Tpo -c anchored_set_variable.cc -o libmodsecurity_la-anchored_set_variable.o
#15 27.31 libtool: compile:  /root/emsdk/upstream/emscripten/em++ -DHAVE_CONFIG_H -I. -std=c++11 -I.. -g -I../others -fPIC -O3 -I../headers -DNO_LOGS=1 -I/root/ModSecurity/include -I/usr/include/libxml2 -DWITH_LIBXML2 -g -O2 -MT libmodsecurity_la-transaction.lo -MD -MP -MF .deps/libmodsecurity_la-transaction.Tpo -c transaction.cc -o libmodsecurity_la-transaction.o
#15 27.31 libtool: compile:  /root/emsdk/upstream/emscripten/em++ -DHAVE_CONFIG_H -I. -std=c++11 -I.. -g -I../others -fPIC -O3 -I../headers -DNO_LOGS=1 -I/root/ModSecurity/include -I/usr/include/libxml2 -DWITH_LIBXML2 -g -O2 -MT libmodsecurity_la-anchored_variable.lo -MD -MP -MF .deps/libmodsecurity_la-anchored_variable.Tpo -c anchored_variable.cc -o libmodsecurity_la-anchored_variable.o
#15 27.32 libtool: compile:  /root/emsdk/upstream/emscripten/em++ -DHAVE_CONFIG_H -I. -std=c++11 -I.. -g -I../others -fPIC -O3 -I../headers -DNO_LOGS=1 -I/root/ModSecurity/include -I/usr/include/libxml2 -DWITH_LIBXML2 -g -O2 -MT libmodsecurity_la-modsecurity.lo -MD -MP -MF .deps/libmodsecurity_la-modsecurity.Tpo -c modsecurity.cc -o libmodsecurity_la-modsecurity.o
#15 28.01 In file included from modsecurity.cc:25:
#15 28.01 In file included from /usr/include/libxml2/libxml/xmlschemas.h:19:
#15 28.01 In file included from /usr/include/libxml2/libxml/tree.h:1307:
#15 28.01 In file included from /usr/include/libxml2/libxml/xmlmemory.h:218:
#15 28.01 In file included from /usr/include/libxml2/libxml/threads.h:35:
#15 28.01 In file included from /usr/include/libxml2/libxml/globals.h:18:
#15 28.01 In file included from /usr/include/libxml2/libxml/parser.h:810:
#15 28.01 /usr/include/libxml2/libxml/encoding.h:31:10: fatal error: 'unicode/ucnv.h' file not found
#15 28.01 #include <unicode/ucnv.h>
#15 28.01          ^~~~~~~~~~~~~~~~
#15 28.04 1 error generated.
#15 28.04 em++: error: '/root/emsdk/upstream/bin/clang++ -target wasm32-unknown-emscripten -DEMSCRIPTEN -D__EMSCRIPTEN_major__=3 -D__EMSCRIPTEN_minor__=1 -D__EMSCRIPTEN_tiny__=16 -fignore-exceptions -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/root/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -DHAVE_CONFIG_H -I. -std=c++11 -I.. -g3 -I../others -fPIC -O3 -I../headers -DNO_LOGS=1 -I/root/ModSecurity/include -I/usr/include/libxml2 -DWITH_LIBXML2 -g3 -O2 -MT libmodsecurity_la-modsecurity.lo -MD -MP -MF .deps/libmodsecurity_la-modsecurity.Tpo -c modsecurity.cc -o libmodsecurity_la-modsecurity.o' failed (returned 1)
#15 28.05 Makefile:1971: recipe for target 'libmodsecurity_la-modsecurity.lo' failed
#15 28.05 make[3]: *** [libmodsecurity_la-modsecurity.lo] Error 1
#15 28.05 make[3]: *** Waiting for unfinished jobs....
#15 28.10 In file included from transaction.cc:40:
#15 28.10 In file included from ../src/request_body_processor/xml.h:17:
#15 28.10 In file included from /usr/include/libxml2/libxml/xmlschemas.h:19:
#15 28.10 In file included from /usr/include/libxml2/libxml/tree.h:1307:
#15 28.10 In file included from /usr/include/libxml2/libxml/xmlmemory.h:218:
#15 28.10 In file included from /usr/include/libxml2/libxml/threads.h:35:
#15 28.10 In file included from /usr/include/libxml2/libxml/globals.h:18:
#15 28.10 In file included from /usr/include/libxml2/libxml/parser.h:810:
#15 28.10 /usr/include/libxml2/libxml/encoding.h:31:10: fatal error: 'unicode/ucnv.h' file not found
#15 28.10 #include <unicode/ucnv.h>
#15 28.10          ^~~~~~~~~~~~~~~~
#15 28.13 1 error generated.
#15 28.14 em++: error: '/root/emsdk/upstream/bin/clang++ -target wasm32-unknown-emscripten -DEMSCRIPTEN -D__EMSCRIPTEN_major__=3 -D__EMSCRIPTEN_minor__=1 -D__EMSCRIPTEN_tiny__=16 -fignore-exceptions -fvisibility=default -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/root/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -DHAVE_CONFIG_H -I. -std=c++11 -I.. -g3 -I../others -fPIC -O3 -I../headers -DNO_LOGS=1 -I/root/ModSecurity/include -I/usr/include/libxml2 -DWITH_LIBXML2 -g3 -O2 -MT libmodsecurity_la-transaction.lo -MD -MP -MF .deps/libmodsecurity_la-transaction.Tpo -c transaction.cc -o libmodsecurity_la-transaction.o' failed (returned 1)
#15 28.15 make[3]: *** [libmodsecurity_la-transaction.lo] Error 1
#15 28.15 Makefile:1915: recipe for target 'libmodsecurity_la-transaction.lo' failed
#15 28.22 mv -f .deps/libmodsecurity_la-anchored_variable.Tpo .deps/libmodsecurity_la-anchored_variable.Plo
#15 28.68 mv -f .deps/libmodsecurity_la-anchored_set_variable.Tpo .deps/libmodsecurity_la-anchored_set_variable.Plo
#15 28.68 make[3]: Leaving directory '/root/ModSecurity/src'
#15 28.68 Makefile:3308: recipe for target 'all-recursive' failed
#15 28.68 make[2]: Leaving directory '/root/ModSecurity/src'
#15 28.68 make[2]: *** [all-recursive] Error 1
#15 28.68 make[1]: *** [all] Error 2
#15 28.68 Makefile:1048: recipe for target 'all' failed
#15 28.68 make[1]: Leaving directory '/root/ModSecurity/src'
#15 28.68 Makefile:1043: recipe for target 'all-recursive' failed
#15 28.68 make: *** [all-recursive] Error 1
#15 28.68 emmake: error: 'make -j 4' failed (returned 2)
------
executor failed running [/bin/bash -c ./build.sh     && git submodule init && git submodule update     && autoreconf -ivf     && cd ~/emsdk     && source ./emsdk_env.sh     && cd ~/ModSecurity     && emconfigure ./configure --without-yajl --without-geoip --without-curl     --without-lua --disable-shared --disable-examples --disable-libtool-lock     --disable-debug-logs  --disable-mutex-on-pm --without-lmdb --without-maxmind     --without-ssdeep --without-pcre2 --with-pcre=./pcre-config     && emmake make -j 4     && emmake make install     && cd ..]: exit code: 1

[leyao-daily]

Hi, is it necessary to use libxml? we remove libxml because we havn't build the WASM LIB for libxml.

[dspeg]

Also, what features or use cases might be impacted by the code changes in the ModSecurity V3 wasm-experimental branch? owasp-modsecurity/ModSecurity@v3/master...v3/dev/wasm-experimental

[dspeg]

I see. Without the libxml, conditions on XML variables won't evaluate properly, I assume.

[M4tteoP]

Hi @dspeg, just like @leyao-daily pointed out, all the --without options are enforced because each library requires to be ported to Wasm in order to be able to consume it. It is a step-by-step work that has to be done in order to extend wasm-ModSecurity functionalities that, currently, are based on a "minimal build" (with just PCRE as a Wasm compiled library). As a matter of example, #26 is the issue related to supporting the PCRE2 library.

[M4tteoP]

Also, what features or use cases might be impacted by the code changes in the ModSecurity V3 wasm-experimental branch? SpiderLabs/ModSecurity@v3/master...v3/dev/wasm-experimental

The changes mostly remove everything related to file system, network and other system calls not yet supported by Wasm such as the generation of random transaction IDs or exception handling. It mainly impacts the possibility of providing rules and .data files via the file system.

There is still also ongoing work that has to be done in order to improve the stability of the filter (E.g. owasp-modsecurity/ModSecurity#2783) and reduce unexpected behaviours that the users may face (Currently, if the filter stops working, the Envoy filter chain breaks, and all the requests return a 5xx response).

[dspeg]

Thanks @leyao-daily and @M4tteoP for the detailed info!

[dspeg]

"The changes mostly remove everything related to file system ..." Then the Mod Security operator "pmFromFile" won't work?

Are there additional workarounds besides changing "pmFromFile" to "pm" to make it "inline"?
Is it possible to bundle additional data files into the .wasm plugin file when generating the .wasm file and make these data files available to ModSecurity?

[M4tteoP]

To the best of my knowledge:

Then the Mod Security operator "pmFromFile" won't work?

It will not, because it is looking for its related .data file and envoy side we are not exposing files to wasm extensions. This is the open issue to make it happen: envoyproxy/envoy#22557

Are there additional workarounds besides changing "pmFromFile" to "pm" to make it "inline"?

Currently, I don't think so. Speaking about pmFromFile -> pm tweak, it is something that can work but to have all the .data entries fully working would be needed to use a custom separator instead of the default (otherwise you are going to be able to match only individual words). I have an open PR about it if you wish to give it a go.

Is it possible to bundle additional data files into the .wasm plugin file when generating the .wasm file and make these data files available to ModSecurity?

Would be really nice. it is something that coraza-proxy-wasm is doing with one line thanks to Golang, but C++ side I'm not aware of a solution like this one.

[dspeg]

Thank you @M4tteoP for the in-depth explanation.
The custom separator is a very useful enhancement for the "pm" operator that should be added to the ModSecurity master branch.

[M4tteoP]

You are welcome @dspeg! Feel free to provide any feedback about the custom separator, even just for showing interest from the community :)