This patch speeds up startup and adds the --unique-map option

[notnyt]

Barnyard2 takes a long time to start due to poorly optimized list code. For every SID added to the list, the entire list is iterated to get to the tail of the list. Adding and maintaining a tail pointer prevents this,
since we can just add entries directly at the end.

The --unique-map option is also added, this prevents each SID from the files being checked against what's in memory. Most tools generate unique files, making this check unnecessary and wasteful.

[binf]

The code should not assume certain tools usage if input is corrupted output
will also be.

On Mon, Feb 8, 2016 at 9:01 AM, Rob Mosher [email protected] wrote:

Barnyard2 takes a long time to start due to poorly optimized list code.
For every SID added to the list, the entire list is iterated to get to the
tail of the list. Adding and maintaining a tail pointer prevents this,
since we can just add entries directly at the end.

The --unique-map option is also added, this prevents each SID from the
files being checked against what's in memory. Most tools generate unique

files, making this check unnecessary and wasteful.

You can view, comment on, or merge this pull request online at:

#181
Commit Summary

• This patch speeds up startup and adds the --unique-map option

File Changes

• M etc/barnyard2.conf
  https://github.com/firnsy/barnyard2/pull/181/files#diff-0 (4)
• M src/barnyard2.c
  https://github.com/firnsy/barnyard2/pull/181/files#diff-1 (9)
• M src/barnyard2.h
  https://github.com/firnsy/barnyard2/pull/181/files#diff-2 (13)
• M src/map.c
  https://github.com/firnsy/barnyard2/pull/181/files#diff-3 (29)
• M src/map.h
  https://github.com/firnsy/barnyard2/pull/181/files#diff-4 (2)
• M src/output-plugins/spo_database_cache.c
  https://github.com/firnsy/barnyard2/pull/181/files#diff-5 (2)
• M src/parser.c
  https://github.com/firnsy/barnyard2/pull/181/files#diff-6 (8)
• M src/parser.h
  https://github.com/firnsy/barnyard2/pull/181/files#diff-7 (2)

Patch Links:

• https://github.com/firnsy/barnyard2/pull/181.patch
• https://github.com/firnsy/barnyard2/pull/181.diff

—
Reply to this email directly or view it on GitHub
#181.

[notnyt]

Well, it's added as an option if you have known good input. The linked list optimization is unrelated, you can merge whichever parts you want.

[binf]

I agree that using tail for that can speed up startup but its not the in
memory linked list parsed from sid-msg.map or gen-msg.map file that are
slow.
Its the sanity check of what is in the database.

On Tue, Mar 22, 2016 at 8:57 PM, Rob Mosher [email protected]
wrote:

Well, it's added as an option if you have known good input. The linked
list optimization is unrelated, you can merge whichever parts you want.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#181 (comment)

[notnyt]

It still shaved a few minutes of time off during startup using a long list.

[binf]

#define long_list

On Tue, Mar 22, 2016 at 9:13 PM, Rob Mosher [email protected]
wrote:

It still shaved a few minutes of time off during startup using a long list.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#181 (comment)

[notnyt]

10-20k

[binf]

And on what type of hardware?

Minutes you see are from here:
if( BcUniqueMap() ||
(cacheSignatureLookup(&lookupNode,iMasterCache->cacheSignatureHead)
== 0) )
Comment that line in your patch and you shouldn't see minutes anywhere.

On Tue, Mar 22, 2016 at 9:19 PM, Rob Mosher [email protected]
wrote:

10-20k

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#181 (comment)

[notnyt]

hardware is AMD GX-412TC SOC, quad core 1ghz

With the --unique-map option, BcUniqueMap will return 1, and prevent cacheSignatureLookup from running on the line you referenced.

It still takes quite some time to load the rules, even with that in place.

[binf]

Sorry i replied quickly with something that should be read as follow:

Initialization of 10-20k if it takes minutes to read the sid-msg.map is
probably more hardware setup dependant.

Where is agree is tail adding should be good enough, on the other
hand using head rather than using tail would have the same effect.
Where i do not agree is on integrity bypass.
Will queue that in a form or an other in the pipe.

On Tue, Mar 22, 2016 at 9:22 PM, beenph [email protected] wrote:

And on what type of hardware?

Minutes you see are from here:
if( BcUniqueMap() || (cacheSignatureLookup(&lookupNode,iMasterCache->cacheSignatureHead)
== 0) )
Comment that line in your patch and you shouldn't see minutes anywhere.

On Tue, Mar 22, 2016 at 9:19 PM, Rob Mosher [email protected]
wrote:

10-20k

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#181 (comment)

[binf]

If you use a database, do a select count(*) from signature;
im curious.

On Tue, Mar 22, 2016 at 9:35 PM, beenph [email protected] wrote:

Sorry i replied quickly with something that should be read as follow:

Initialization of 10-20k if it takes minutes to read the sid-msg.map is
probably more hardware setup dependant.

Where is agree is tail adding should be good enough, on the other
hand using head rather than using tail would have the same effect.
Where i do not agree is on integrity bypass.
Will queue that in a form or an other in the pipe.

On Tue, Mar 22, 2016 at 9:22 PM, beenph [email protected] wrote:

And on what type of hardware?

Minutes you see are from here:
if( BcUniqueMap() || (cacheSignatureLookup(&lookupNode,iMasterCache->cacheSignatureHead)
== 0) )
Comment that line in your patch and you shouldn't see minutes anywhere.

On Tue, Mar 22, 2016 at 9:19 PM, Rob Mosher [email protected]
wrote:

10-20k

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#181 (comment)

[notnyt]

50k in there at the moment. it's not really bypassing integrity, worst case is the same rule can be added multiple times. If other tools are known to produce unique output, it helps somewhat.