Compiling Ok in AlmaLinux 8.10, but not working

[decioaalves]

I managed to compile it ok on AlmaLinux 8.10, a dozen times, in different ways, with 7 different kernels, on three different hardware, and in none of them did I have any problems building the new version of this connector. However, in all of them, the result is broken pages, the server only serves a partial page. In my tests, the result was that when the web page is close to 500Kb, the result is only part of the page, about 70% of the real page.

But compiling the previous version of this connector, it works normally.

Is this problem happening only to me, or has anyone else noticed this problem?

[decioaalves]

In all times, I have recompiled libmodsecurity.so and nginx before compile ngx_http_modsecurity_module, and in all times the result is only head of the pages are supressed. No problems are shown in the logs.

[airween]

Hi @decioaalves,

... and in none of them did I have any problems building the new version of this connector.

Sorry, what is "this" version? Could you explain it exactly?

However, in all of them, the result is broken pages, the server only serves a partial page. In my tests, the result was that when the web page is close to 500Kb, the result is only part of the page, about 70% of the real page.

But compiling the previous version of this connector, it works normally.

Here too: what are the "previous" and "this" versions?

[decioaalves]

Hi @decioaalves,
Sorry, what is "this" version? Could you explain it exactly?

I mean the current version of the 'ModSecurity-nginx' module, as you get when you clone with git. After compile, it generates a 'ngx_http_modsecurity_module.so' library normally, with no warnings.

However, in all of them, the result is broken pages, the server only serves a partial page. In my tests, the result was that when the web page is close to 500Kb, the result is only part of the page, about 70% of the real page.
But compiling the previous version of this connector, it works normally.

Here too: what are the "previous" and "this" versions?

The 'this' version, is the current version. The connector was updated two months ago. The 'previous' version, was a version earlier version, from June, 2024, I was running.

I could to make a debug trace, and I can post it, if you want..

Thank you.

[airween]

The connector was updated two months ago.

You mean about PR #334? That was committed in 2 months ago (19th of Nov, 2024), but was merged few days ago.

To clarify and reproduce this issue we need:

• commit id of "this" version
• commit id of "previous" version
• relevant configs (both Nginx and ModSecurity)
• a curl example

We need to reproduce the issue.

[decioaalves]

Commit id of 'ModSecurity-nginx' module:

'Older version' (working good version):

git rev-parse HEAD
ef64996
v1.0.3-24-gef64996

'current version' (NOT working good version):

git rev-parse HEAD
fb678c5
v1.0.3-27-gfb678c5

System: AlmaLinux 8.10 (Cerulean Leopard)

Kernels tested:

/boot/kernel-4.18.0-553.34.1.el8_10.x86_64
/boot/kernel-4.18.0-553.33.1.el8_10.x86_64
/boot/kernel-4.18.0-553.32.1.el8_10.x86_64
/boot/kernel-4.18.0-553.30.1.el8_10.x86_64
/boot/kernel-4.18.0-553.27.1.el8_10.x86_64
/boot/kernel-4.18.0-553.el8_10.x86_64

ModSecurity installed pre-requisites

$ sudo dnf -y install pcre pcre-devel libxml2 libxml2-devel curl curl-devel openssl openssl-devel yajl yajl-devel
$ sudo dnf -y install gcc-c++ flex bison GeoIP-devel doxygen zlib-devel
$ sudo dnf -y install lmdb lmdb-devel ssdeep ssdeep-devel lua lua-devel

ModSecurity3 commit id installed in hardware #1:

git rev-parse HEAD
9e685bf86d45b21d1f9f6c4145c3979cc4da0196
v3.0.12-296-g9e685bf

ModSecurity3 commit id installed in machine #2 and #3:

git rev-parse HEAD
41fd21b0fbdbdaf202f830f700027767fbd6fba7
v3.0.12-277-g41fd21b

ModSecurity3 installation

$ sudo git clone --depth 100 -b v3/master --single-branch https://github.com/SpiderLabs/ModSecurity
$ cd ModSecurity
$ sudo git submodule init
$ sudo git submodule update

$ sudo libtoolize && aclocal && autoheader && autoconf && automake --add-missing
The last command is due to 'ltmain.sh no found' error, step recommended from: https://stackoverflow.com/questions/22603163/automake-error-ltmain-sh-not-found
$ sudo sh build.sh
$ sudo ./configure
$ sudo make -j /usr/bin/nproc
$ sudo make install
$ sudo ldconfig

Nginx must be customized, due to the need for compatibility with the CWP WebPanel that is installed
Nginx dependencies checked and installed from "Installing NGINX Dependencies" section from https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source/#installing-nginx-dependencies
Compilation process based in the tips "CentOS 7 Minimal (dynamic)" sequence steps from https://github.com/owasp-modsecurity/ModSecurity/wiki/Compilation-recipes-for-v3.x :

$ sudo dnf groupinstall 'Development Tools' -y
$ sudo dnf install gcc-c++ flex bison yajl yajl-devel curl-devel curl GeoIP-devel doxygen zlib-devel
$ sudo dnf install lmdb lmdb-devel libxml2 libxml2-devel ssdeep ssdeep-devel lua lua-devel
(changed) $ sudo git clone --depth 100 -b v3/master --single-branch https://github.com/SpiderLabs/ModSecurity
$ cd ModSecurity
$ sudo dnf submodule init
$ sudo dnf submodule update
(due to 'ltmain.sh no found' error, step recommended from: https://stackoverflow.com/questions/22603163/automake-error-ltmain-sh-not-found, added:)
(added) # libtoolize && aclocal && autoheader && autoconf && automake --add-missing
$ sudo ./build.sh
(changed) $ sudo ./configure --prefix=/usr/local/cwpsrv --sbin-path=/usr/local/cwpsrv/bin/cwpsrv --conf-path=/usr/local/cwpsrv/conf/cwpsrv.conf --pid-path=/usr/local/cwpsrv/var/run/nginx.pid --error-log-path=/usr/local/cwpsrv/logs/error_log --http-log-path=/usr/local/cwpsrv/logs/access_log --user=root --group=root --with-http_ssl_module --with-http_gzip_static_module --with-file-aio --with-http_secure_link_module --with-http_ssl_module --add-module=src/misc/ngx_http_auth_pam_module --with-compat --add-dynamic-module=../ModSecurity-nginx
$ sudo make
$ sudo make install
(changed) $ sudo git clone https://github.com/SpiderLabs/ModSecurity-nginx
(changed) $ sudo wget http://nginx.org/download/nginx-1.15.10.tar.gz
(changed) $ sudo tar zxvf nginx-1.15.10.tar.gz
(changed) # cd nginx-1.15.10
(changed) $ sudo ./configure --with-compat --add-dynamic-module=../ModSecurity-nginx
$ sudo make modules
(changed) $ sudo cp objs/ngx_http_modsecurity_module.so /usr/local/cwpsrv/modules
(changed) $ sudo mkdir /usr/local/cwpsrv/conf/security
(changed) $ sudo cp ~/ModSecurity/unicode.mapping /usr/local/cwpsrv/conf/security/

Installed with fresh OWASP CRS 4.3.0, original steps, unchanged
Installation steps based in https://github.com/coreruleset/coreruleset/blob/main/INSTALL.md, just adapted base directoy, instead /etc/crs4/ to /usr/local/cwpsrv/conf/security/crs4/
Added step to install pam-devel to satisfy 'ngx_http_auth_pam_module':

$ sudo dnf install pam-devel

nginx -V

nginx version: nginx/1.15.10
built by gcc 8.5.0 20210514 (Red Hat 8.5.0-22) (GCC)
built with OpenSSL 1.1.1k FIPS 25 Mar 2021
TLS SNI support enabled
configure arguments: --prefix=/usr/local/cwpsrv --sbin-path=/usr/local/cwpsrv/bin/cwpsrv --conf-path=/usr/local/cwpsrv/conf/cwpsrv.conf --pid-path=/usr/local/cwpsrv/var/run/nginx.pid --error-log-path=/usr/local/cwpsrv/logs/error_log --http-log-path=/usr/local/cwpsrv/logs/access_log --user=root --group=root --with-http_ssl_module --with-http_gzip_static_module --with-file-aio --with-http_secure_link_module --with-http_ssl_module --add-module=src/misc/ngx_http_auth_pam_module --with-compat --add-dynamic-module=../ModSecurity-nginx

cwpsrv -V (original binary built from CWP Web Panel, that was tested too, and didn't work):

cwpsrv version: cwpsrv/1.15.10
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC)
built with OpenSSL 1.0.2k-fips 26 Jan 2017 (running with OpenSSL 1.0.2o-fips 27 Mar 2018)
TLS SNI support enabled
configure arguments: --prefix=/usr/local/cwpsrv --sbin-path=/usr/local/cwpsrv/bin/cwpsrv --conf-path=/usr/local/cwpsrv/conf/cwpsrv.conf --pid-path=/usr/local/cwpsrv/var/run/nginx.pid --error-log-path=/usr/local/cwpsrv/logs/error_log --http-log-path=/usr/local/cwpsrv/logs/access_log --user=root --group=root --with-http_ssl_module --with-http_gzip_static_module --with-file-aio --with-http_secure_link_module --with-http_ssl_module --add-module=src/misc/ngx_http_auth_pam_module

changes to main configuration file from nginx/cwp: /usr/local/cwpsrv/conf/cwpsrv.conf :

worker_processes 2;
+ include /usr/local/cwpsrv/conf/mod-http-modsecurity.conf;
events {
(...)
+ modsecurity on;
+ modsecurity_rules_file /usr/local/cwpsrv/conf/security/conf/modsec_includes.conf;
location / {
(...)

Content from file /usr/local/cwpsrv/conf/mod-http-modsecurity.conf:

load_module modules/ngx_http_modsecurity_module.so;

Original installation from OWASP CRS 4.3.0:
Content from file /usr/local/cwpsrv/conf/security/conf/modsec_includes.conf:

Include /usr/local/cwpsrv/conf/security/crs4/modsecurity.conf (recommended original)
Include /usr/local/cwpsrv/conf/security/crs4/crs-setup.conf
Include /usr/local/cwpsrv/conf/security/crs4/plugins/-config.conf
Include /usr/local/cwpsrv/conf/security/crs4/plugins/-before.conf
Include /usr/local/cwpsrv/conf/security/crs4/rules/.conf
Include /usr/local/cwpsrv/conf/security/crs4/plugins/-after.conf

Synthoms:

-ModSecurity-nginx compiling and installing normally in version v1.0.3-27-gfb678c5, no errors nor warnings
-Enabling ModSecurity-nginx version v1.0.3-27-gfb678c5 does not work as expected, almost always gets partial pages, broken pages, rendered with strange characters, like binary characters. The problem with these broken pages is always at the beginning of the pages, about 30% a 40% of the beginning of the pages is suppressed, completely damaging their rendering.
-Disabling ModSecurity in configuration in nginx configuration (just setting "modsecurity off";" and/or commenting both "modsecurity" and "modsecurity_rules_file" diretives), the partial/broken pages error persist.
-Tested with originally built nginx version from CWP (cwpsrv) as dynamic module, but tested too with compiled sources from nginx original sources, version 1.15.10, but continues not working as expected.
-Almost always get partial/broken web pages, from php pages, with ModSecurity-nginx version v1.0.3-27-gfb678c5
-Note: PHP pages are from closed source control webpanel CWP, I have no access to sources.

Solution:

-When disabled ModSecurity-nginx, the pages come back to normal, working as expected (when is removed completely the directive "load_module modules/ngx_http_modsecurity_module.so;" from nginx configuration file /usr/local/cwpsrv/conf/mod-http-modsecurity.conf) unloading it from nginx.
-Another solution was downgrade to ModSecurity-nginx version v1.0.3-24-gef64996

(I'm having trouble making the request with curl, with a password protected page via POST method, as soon as I can, I'll post it here.)

[airween]

Hi @decioaalves,

thanks, I have these ones:

~/src/ModSecurity-nginx$ git describe
v1.0.3-27-gfb678c5
~/src/ModSecurity$ git describe
v3.0.12-296-g9e685bf8

so the newest commits from both repository.

I also installed Nginx, but during some tests yesterday I didn't get any problem. The curl request would be important, what's the request - and if you turn off the module, then please show us the response from the backend. If it contains any sensitive info, please send that to [email protected].

Nginx must be customized, due to the need for compatibility with the CWP WebPanel that is installed

I see, but I need to know the configuration to reproduce the issue. There was a PR (#334) which probably can occur this issue, but without details I can't do anything. Under the mentioned PR you can see some comments, a customer who uses heavily this module confirms that this patch works as well. But of course, we need to find a solution - if we see what's the problem.

[DaveeHorvath]

Hi,
An issue similar to this one happened for me in a docker container based from debian:bullseye, nginx version 1.18 (installed from apt) using the configuration directly from https://github.com/coreruleset/coreruleset.git with the latest commit of Modsecurity. The requests for a fairly big file (~39kb) fails, as it made the body of the response scrambled as in similar size but a lot of parts moved over one another. The content size matches as well. I would happy to share the request, response and the original file, if that could help? The repo where the issue occured is https://github.com/DaveeHorvath/ascension, and the failing file was main.js. Any other information that would be needed?

[airween]

@DaveeHorvath,

thanks - do you also use the connector with last commit (fb678c5) with Bullseye?

The content size matches as well. I would happy to share the request, response and the original file, if that could help?

Yes, definitely.

The repo where the issue occured is ...

Sorry, I don't use Docker just very rarely, it would be huge help if you could provide the request (with payload) and an example response (that I can set as content).

And many thanks for these!

[decioaalves]

I was unable to create an example using curl, but I was able to generate two *.har files using Chrome, one with the correct page returned, and the other broken.
The sequence shown in these files represents the call to two pages, the first a simple call to the main page of the CWP system. After logging into this page, the following page represents the page in question being analyzed. For simplicity, I will call it the "second page".
The first file, 'request-good.har' represents these two calls working normally, with version v1.0.3-24-gef64996 of ModSecurit-nginx.
The second correct page is returned on line 956 ("text": ...).

So I then upgraded to version v1.0.3-27-gfb678c5, and generated the second file, 'request-broken.har', which shows these same HTTP calls, but results in the second page being broken, being returned at line 954.

Strangely, after doing a scan through the modsecurity audit logs, it shows the second page as if it had been returned to me in its entirety, rather than broken in half. But I had already tested it in two different browsers, Chrome, Firefox and links(text only), and the result is always the same.

request-good.har.zip
request-broken.har.zip

[airween]

@decioaalves,

thanks - I hope this will help me to investigate the issue. Let me check that in next few days.

Cc @Dr-Lazarus-V2 - if you able to check, please help me in this.

[promiteyka]

Have the same problem, this appeared after the commit fb678c5.
tried to work to work with nginx-1.26.2-1.el7.x86_64(EL) and nginx/stable,now 1.26.0-1 amd64(DEB)
Files more than 32kb are cut off at the front and become invalid

[makar112233]

Hi all. I breaked my brains with debugging same issue.
Please, help with solution.

[DaveeHorvath]

a temporary solution is to set use an older commit of the connector mentioned by @decioaalves, the link ef64996 should be a working version without this issue.

[airween]

Files more than 32kb are cut off at the front and become invalid

You mean if the response is bigger than 32 kB, then it will be cut off?

[promiteyka]

Files more than 32kb are cut off at the front and become invalid

You mean if the response is bigger than 32 kB, then it will be cut off?

We came to this through experiments. We have some static on the site, files larger than 32 kb open on the client without the beginning of the file. Although the size is shown the same as on the server.

[airween]

We have some static on the site, files larger than 32 kb

Could you try it with curl? And show the full output (eg use curl -v http://your.site/path/to/static - maybe you should pass --output static.bin)

Now I tried on my environment (with same versions) but everything was fine:

$ curl -v waf.test:8080/outfile.bin --output temp.bin 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Host waf.test:8080 was resolved.
* IPv6: (none)
* IPv4: 127.0.0.1
*   Trying 127.0.0.1:8080...
* Connected to waf.test (127.0.0.1) port 8080
* using HTTP/1.x
> GET /outfile.bin HTTP/1.1
> Host: waf.test:8080
> User-Agent: curl/8.11.1
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 200 OK
< Server: nginx/1.26.0
< Date: Thu, 16 Jan 2025 13:07:05 GMT
< Content-Type: application/octet-stream
< Last-Modified: Thu, 16 Jan 2025 13:06:26 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< ETag: "67890452-19000"
< 
{ [65296 bytes data]
100  100k    0  100k    0     0  1610k      0 --:--:-- --:--:-- --:--:-- 1639k
* Connection #0 to host waf.test left intact

the file:

$ ls -la temp.bin 
-rw-rw-r-- 1 airween airween 102400 Jan 16 14:07 temp.bin

Note that there is no proxy, Nginx serves the files directly.

[promiteyka]

We have some static on the site, files larger than 32 kb

Could you try it with curl? And show the full output (eg use curl -v http://your.site/path/to/static - maybe you should pass --output static.bin)

Now I tried on my environment (with same versions) but everything was fine:

$ curl -v waf.test:8080/outfile.bin --output temp.bin 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Host waf.test:8080 was resolved.
* IPv6: (none)
* IPv4: 127.0.0.1
*   Trying 127.0.0.1:8080...
* Connected to waf.test (127.0.0.1) port 8080
* using HTTP/1.x
> GET /outfile.bin HTTP/1.1
> Host: waf.test:8080
> User-Agent: curl/8.11.1
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 200 OK
< Server: nginx/1.26.0
< Date: Thu, 16 Jan 2025 13:07:05 GMT
< Content-Type: application/octet-stream
< Last-Modified: Thu, 16 Jan 2025 13:06:26 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< ETag: "67890452-19000"
< 
{ [65296 bytes data]
100  100k    0  100k    0     0  1610k      0 --:--:-- --:--:-- --:--:-- 1639k
* Connection #0 to host waf.test left intact

the file:

$ ls -la temp.bin 
-rw-rw-r-- 1 airween airween 102400 Jan 16 14:07 temp.bin

Note that there is no proxy, Nginx serves the files directly.

Yes, I tried both via curl and wget, the result is the same. Can you check the contents of the file? The size is also fine for me when downloading.

This is an example of a downloaded file without mod_sec

And this is the same file but with mod_sec enabled

As you can see, the beginning of the file was cut off and some information was added at the end.

[airween]

Oh, now I see. I tested with a 46kB length JS file, and I got exactly the same result as you (I mean the file is cut off).

Thanks.

@liudongmiao, do you have any idea what would cause this problem? - edit: sorry, this false info.

[airween]

Hi @decioaalves, @DaveeHorvath, @promiteyka and @makar112233,

I can confirm that ef64996 is the last commit where the connector works as we expected.

But I'm afraid the broken commit is 62639fa (part of #326), not fb678c5 .

Could you confirm guys this? It's easy to reproduce: if you check out the broken commit, the file cut effect is still reproducible as with last commit.

@g00g1 ping, it seems like your PR broke the module's normal behavior.

[promiteyka]

@airween Yes, you are right, the problem is not 62639fa. The last successful build on our CI farm was January 9th, now the last working commit is ef64996

[airween]

To @decioaalves, @DaveeHorvath, @promiteyka and @makar112233:

I think I found the cause of the mentioned problem: in the current code commit @62639fa2 added this line:

            chain->buf->pos = chain->buf->last;

(in line 180).

There is a quick fix but before I merge that I want to review the whole patch again, and I need some help to test that it works as well.

So please if anyone who's able to test this, please help me. A quick help:

git clone --branch issue336fix https://github.com/airween/ModSecurity-nginx.git
cd /path/to/nginx
...configure and make...

Please try the module with this patch with:

• huge files (JSON, HTML or XML responses)
  • after the download please make a diff with the original files
  • eg: curl your.server.com/something.json --output something.json && diff something.json /var/www/html/something.json or similar
• also please make tests against issue header already sent while sending response to client  #41 (which was solved with the wrong patch)
• and also make tests against Restore method name for error_page in audit log #270 and Fix phase 2 and 3 audit logging in case of internal redirect #255 (which was fixed in recovery context after internal redirect #273)

@Dr-Lazarus-V2 could you help us in this (run tests)?

@g00g1 could you help me to explain why is that value giving in the mentioned line?

[zero0cool0]

@airween - Thanks for taking care of this issue.

I've tested the above mentioned issue336fix as part of a custom docker build and it compiles, but any request to a modsec enabled nginx server results in a client-side timeout after a long wait period. I suspect the process is stuck.
Nothing in the logs.
Disabling modsec (i.e. comment out modsecurity on gets everything working again.
Not a lot of help I'm afraid, but wanted to report back.

[airween]

@zero0cool0,

I've tested the above mentioned issue336fix as part of a custom docker build and it compiles, but any request to a modsec enabled nginx server results in a client-side timeout after a long wait period. I suspect the process is stuck. Nothing in the logs. Disabling modsec (i.e. comment out modsecurity on gets everything working again. Not a lot of help I'm afraid, but wanted to report back.

Many thanks for this reply.

It's very interesting, this is what I definitely can't reproduce.

Note, that I don't use Docker (very rarely) - the problem is I don't know how is it possible to debug inside of Docker.

[airween]

@zero0cool0,

I've tested the above mentioned issue336fix as part of a custom docker build and it compiles, but any request to a modsec enabled nginx server results in a client-side timeout after a long wait period.

Could you check with the current master branch and the mentioned "last worked" commit?

[zero0cool0]

@airween
I certainly don't intend to make this any weirder, but when I use
https://github.com/owasp-modsecurity/ModSecurity-nginx/archive/refs/heads/master.zip as a source reference to build_module.sh as part of my custom docker build, I run into the issue above.

However, when I clone the master branch via git clone https://github.com/airween/ModSecurity-nginx.git and use this folder as source in build_module.sh I get a well working modsec.

Downloading and unzipping the master.zip file confirms that the content is different to the cloned repo.

Shouldn't the zip archive be the same as the current master?

[airween]

@zero0cool0,

here:

I certainly don't intend to make this any weirder, but when I use
https://github.com/owasp-modsecurity/ModSecurity-nginx/archive/refs/heads/master.zip as a source

you mentioned the repository source as owasp-modsecurity/ModSecurity-nginx. The organization (owner) is the owasp-modsecurity.

But here:

However, when I clone the master branch via git clone https://github.com/airween/ModSecurity-nginx.git and use this folder as source in build_module.sh I get a well working modsec.

you downloaded from my GH repository.

If you take a look at my repository, you can see:

my master branch is behind from the official repository's master.

as part of my custom docker build, I run into the issue above.

You mean the issue that explained in this ticket (namely the downloaded files which bigger than 32 kB cut off)? That's the "expected" now.

Shouldn't the zip archive be the same as the current master?

It must be the same - if you use the same source. Now I downloaded the zip and cloned the git, and got the same content (after I removed the .git directory from the git clone'd directory):

$ diff -rN ModSecurity-nginx-zip ModSecurity-nginx-git
<EMTPY>

Could you try to change to the mentioned branch after you cloned my repository?

git checkout -b issue336fix
git pull origin issue336fix

then run build_module.sh. Then you get the "stacked" Nginx with enabled ModSecurity module?

[zero0cool0]

Ok, apologies for the confusion I caused.

I've just tested again three things:

1. Cloning the master from https://github.com/owasp-modsecurity/ModSecurity-nginx results in the issue explained in this ticket (32kB cut off) and renders practically every service I run unusable.

2. Cloning the master from your GH repo https://github.com/airween/ModSecurity-nginx.git results in a working modsec for me although I'm not able to run all tests.

3. Finally, with the proposed fix

git checkout -b issue336fix
git pull origin issue336fix

(repo is fast-forwarded)

22:14 $ git status
On branch issue336fix
nothing to commit, working tree clean

... I did some more testing. Not every request results in a timeout.
I can request for example my photo gallery's main URL without any issue. Browser developer tools confirm that content such as various JS files are loaded that are much bigger than 32KB and nothing is cut off. However, as soon as I try to do other things (for example log in), I get the timeout described above. Hard to pinpoint exactly what triggers the timeout.

Similarly with a Nextcloud instance. Some URLs work, a simple curl works as well, some absolutely don't.

By the way, I'm running a modsec disabled nginx instance in parallel with otherwise the same config to rule out other issues.

I'll perform some more tests tomorrow to see if I can pinpoint the issue further if that helps.

[decioaalves]

The 62639fa is broken for me, with *.html, *.json and *.xml above ~32Kb it breaks.

However, with this version 'issue336.fix', I was able to test *.json, *.html files, up to ~64Kb Mb, it worked, and the file is served immediately. I cloned with:

git clone --branch issue336fix https://github.com/airween/ModSecurity-nginx.git

However, for files larger than ~64Kb, the files are not immediately served, and curl gives up after 60 seconds.

The server access log (nothing in the error log) records:

123.213.231.132 - - [17/Jan/2025:23:22:04 -0300] "GET /design/js/libs/teste.json HTTP/1.1" 408 0 "-" "curl/7.61.1"

However, with Chrome navigator, although the 408 (Server Timeout) error is recorded in the server access logs after 60 seconds, seems Chrome request the file, and sometimes the file is served (as recorded in modsec_audit.log file) the first 65535 bytes of the file after more 60 seconds, even though and then, after another 180 seconds, the keep-alive connection is closed in this browser (possibly browser configuration). Sometimes, in the error logs, is recorded only:

2025/01/17 20:27:43 [info] 2361865#0: *27 client 127.0.0.1 closed keepalive connection

[promiteyka]

To @decioaalves, @DaveeHorvath, @promiteyka and @makar112233:

I think I found the cause of the mentioned problem: in the current code commit @62639fa2 added this line:

        chain->buf->pos = chain->buf->last;

(in line 180).

There is a quick fix but before I merge that I want to review the whole patch again, and I need some help to test that it works as well.

So please if anyone who's able to test this, please help me. A quick help:

git clone --branch issue336fix https://github.com/airween/ModSecurity-nginx.git
cd /path/to/nginx
...configure and make...

Please try the module with this patch with:

• huge files (JSON, HTML or XML responses)

  • after the download please make a diff with the original files
  • eg: curl your.server.com/something.json --output something.json && diff something.json /var/www/html/something.json or similar

• also please make tests against issue header already sent while sending response to client  #41 (which was solved with the wrong patch)

• and also make tests against Restore method name for error_page in audit log #270 and Fix phase 2 and 3 audit logging in case of internal redirect #255 (which was fixed in recovery context after internal redirect #273)

@Dr-Lazarus-V2 could you help us in this (run tests)?

@g00g1 could you help me to explain why is that value giving in the mentioned line?

I tried to build with
repo: "https://github.com/airween/ModSecurity-nginx.git"
version: "f4ae9c690ffc9d22957bb7af61b70b4246c30245"
the problem is not reproduced, web-sites open correctly.

[dbc-ca]

Just for your info, I tried the issue336fix branch and got

wget https://waf-update.*******************.com/lczw.poster.png
--2025-01-18 08:47:38-- https://waf-update.*******************.com/lczw.poster.png
Résolution de waf-update..com (waf-update..com)… 10.10.10.10
Connexion à waf-update..com (waf-update..com)|10.10.10.10|:443… connecté.
requête HTTP transmise, en attente de la réponse… Aucune donnée reçue.
Nouvel essai.

--2025-01-18 08:48:39-- (essai : 2) https://waf-update.*******************.com/lczw.poster.png
Connexion à waf-update..com (waf-update..com)|10.10.10.10|:443… connecté.
requête HTTP transmise, en attente de la réponse… Aucune donnée reçue.
Nouvel essai.

--2025-01-18 08:49:41-- (essai : 3) https://waf-update.*******************.com/lczw.poster.png
Connexion à waf-update..com (waf-update..com)|10.10.10.10|:443… connecté.
requête HTTP transmise, en attente de la réponse… Aucune donnée reçue.
Nouvel essai.

no data coming it seems

[decioaalves]

I tried to build with repo: "https://github.com/airween/ModSecurity-nginx.git" version: "f4ae9c690ffc9d22957bb7af61b70b4246c30245" the problem is not reproduced, web-sites open correctly.

Yes, I tryed too at this moment, and I can confirm, this version it's funcional and working.