Queries without watchdog restrictions

[ar-mi]

1) What version of Fleet are you using (fleet version --full)?
fleet - version 3.5.1
branch: master
revision: 55a2aa2
build date: 2020-12-15T02:50:05Z
build user: zwass
go version: go1.15

What operating system are you using?
NAME="CentOS Linux"
VERSION="8"

What did you do?
Opened Fleet GUI, Queries page. Created and pushed scheduled query. But after some period of time I've seen what this query is in denylist in some amount of hosts, although interactive query works fine.

What did you expect to happen?
I expected to have stable results from osquery from every enrolled in Fleet hosts because of we have alerts in SIEM based on these data.

What happened instead?
Blacklisted query for 24 hours

Enhancement details
At the moment, we are trying to optimize our query to reduce its load on CPU, and we are also trying to increase the parameters of the watchdog.
Nevertheless, it would be great to have an opportunity to launch some queries with an analogue of denylist: false option from the description https://osquery.readthedocs.io/en/stable/deployment/configuration/#schedule from Fleet (in particular, in scheduled queries).
This would allow us to force the execution of queries that we are confident in the correctness and get more stable results.

[ar-mi]

Discussed this moment with @zwass in Slack.
If I understood correctly, this is possible.

[zwass]

This is currently possible to configure by using fleetctl to edit the pack yaml and set the denylist parameter as false. Shall we make this a feature request to expose the same capability in the UI?

Setting denylist to false does not actually remove watchdog restrictions from the query. See osquery/osquery#6922.

[zwass]

Issue #202 is now resolved, so the denylist parameter can be set via fleetctl and the Fleet API starting with Fleet 3.7.0 (coming next week).

We've discussed internally and decided that we will not actually expose this configuration via the UI. We believe that it's confusing, as setting the denylist parameter will not prevent the Watchdog from killing the query (see osquery/osquery#6922). We will make the option available as we believe that Fleet should support all osquery options, but we'd prefer not to expose such a confusing option in the UI.

[dachin11]

Hi, I'm a little confused on this as I was just reading the updated docs (via osquery/osquery#6922) - https://github.com/zwass/osquery/blob/denylist-doc/docs/wiki/deployment/configuration.md.

There was a note added:
Note that queries are still constrained by the Watchdog when the denylistkey is set to false. This means that settingdenylist to false is _not_ sufficient to ensure a query will be run without resource constraints. Queries stopped by the Watchdog should be addressed by modifying the query SQL and/or Watchdog configuration until the limits are not exceeded.

If this note above is correct, then what exactly does adding the denylist: false parameter via fleetctl actually do?

[zwass]

The denylist parameter in osquery prevents the query from being added to the denylist. I'm skeptical of the value of it overall because it does us little benefit if the query is going to be killed by the watchdog anyway. I suppose there may be some small set of queries that complete without the watchdog killing the worker sometimes and that could motivate turning the denylist off for that query.

[dachin11]

Ah, I get it now.

The denylist parameter in osquery prevents the query from being added to the denylist. I'm skeptical of the value of it overall because it does us little benefit if the query is going to be killed by the watchdog anyway. I suppose there may be some small set of queries that complete without the watchdog killing the worker sometimes and that could motivate turning the denylist off for that query.

Yes, I don't understand the value of that at all. I wasn't even aware that it was possible for a query to be denylisted without getting killed by the watchdog.

[zwass]

@dachin11 you still don't have it quite right. A query cannot be denylisted without getting killed by the watchdog.

You can't control the watchdog on a per-query basis. What you can control is whether the query is added to the denylist after it is killed by the watchdog. Typically you want a query to be denylisted if it's killed by the watchdog because it will most likely be killed each time it executes. If it's not denylisted then you just use more resources attempting to run the query. This would only have value if you believe that the query might sometimes complete without triggering the watchdog.

[dachin11]

Thank you for the clarification.