Downloading a large file from VQL fails

[misje]

I've been trying to download a 108 MB file using either FetchBinary or a upload-type parameter. The download never succeeds. The following various behaviours are observed:

• The client instantly downloads the file, which is obviously incomplete
• The client starts downloading (a temporary file is created and starts to grow), then at some point it just stops downloading and hangs
• The client claims that it's downloading (log says "fetching" in the case of upload-type parameters), but no temporary file is created. The client just hangs.

A smaller file works just fine. Tested with both client v. 0.73.3 and 0.7.1. Server is 0.73.3.

[scudette]

When you say a temporary file what do you mean? It should not need to create any temporary file to upload an existing file.

Do you have anything in the query logs?

[misje]

Let me clarify that "upload" here refers to artifact parameter type called "upload". Here is an example artifact I am using to consistently reproduce the issues:

name: Issue4008
description: |
   Print size of downloaded file

type: CLIENT

parameters:
   - name: File
     type: upload

sources:
  - precondition:
      SELECT OS From info() where OS = 'linux'

    query: |
       LET Result = SELECT stat(filename=tempfile(data=File)) AS Stat
         FROM scope()
       SELECT *, humanize(bytes=Stat.Size) AS Size
       FROM Result

I've done more testing, and smaller files work just fine. Large files (100+ MB) never work, and either "complete" after about 4 MB or time out. The server log contains nothing, and the client logs not much either. I've tried to rule out wss as a culprit, but nothing changed. This is what I deem as the only relevant log row from the client (this uses wss, but using http-only server URIs doesn't seem to make a difference):

[INFO] 2025-01-10T09:21:08Z Starting query execution for $bc5fc8d27188ecc1f04b2124b0431f2c. 
[INFO] 2025-01-10T09:21:08Z Fetching https://<REDACTED>/public/temp/31810558abbab9e187e255d7454723d72a5f1595067e4f1e7ed7363e8094347f/LargeFile 
[INFO] 2025-01-10T09:21:09Z Receiver C.c0cdc69dbd0e4efa: Connected to wss://<REDACTED>/reader after waiting for limiter for 7.429µs 
[INFO] 2025-01-10T09:21:10Z File Ring Buffer: Enqueue {"header":"{\"ReadPointer\":50,\"WritePointer\":113,\"MaxSize\":1073741874,\"AvailableBytes\":55,\"LeasedBytes\":0}","leased_pointer":50}
[INFO] 2025-01-10T09:21:10Z File Ring Buffer: Enqueue {"header":"{\"ReadPointer\":50,\"WritePointer\":483,\"MaxSize\":1073741874,\"AvailableBytes\":417,\"LeasedBytes\":0}","leased_pointer":50}
[INFO] 2025-01-10T09:21:11Z Sender: Connected to wss://<REDACTED>/control after waiting for limiter for 9.134µs 
[INFO] 2025-01-10T09:21:11Z Sender: sent 979 bytes, response with status: 200 after 24.598903ms, waiting for server messages 
[INFO] 2025-01-10T09:21:11Z Sender: received 0 bytes in 24.679945ms

[predictiple]

I get the same result.

There seems to be a fixed limit where only 4194304B = 4096KB = 4.2MB gets delivered to the client. It completes immediately without errors (i.e. no timeout issue)

The large file is correctly uploaded to /public on the server and can be fetched from the download link in the collection log, and it delivers the full file.

[scudette]

This is working as intended. The vql compiler generates the following vql for an upload parameter

https://github.com/Velocidex/velociraptor/blob/master/services%2Flauncher%2Fcompiler.go#L75

This calls http client to retrieve a file into the vql scope. Because the chunk size is not specified the default is 4mb. We don't generally want to allow http client to download massive files into memory so there is a reasonable limit.

If you want to go larger you should use http client manually with a temp file to overcome the limit or specify a larger chunk size ( not recommended).