Overhaul of the Logging Infrastructure #1596
Conversation
…ace to satisfy ADL requirements.
- LogLevel is now a top level enum class. - Added a new log level Off to disable output from a specific module. - Logger::LogLevel is a deprecated alias to LogLevel. - Logger::Info, Debug, Error are deprecated aliases to LogLevel::... - Removed public "internal*" functions from Logger.Logger - Added LogSource struct to encapsulate source information. - Added shouldLog method to check if a log should be emitted for a given level and module. - Removed nonfunctional artifacts "m_LogStream" and "Logger::operator<<" - Added templated "log" functions that are friends to Logger. - Reworked PCPP_LOG macros to no longer utilize the now removed internal functions. - Added PCPP_LOG_INFO macro level. - Changed PCPP_LOG_ERROR to now check if the log should be emitted. - Fixed NetworkUtils log module name overlapping with NetworkUtils class. - Fixed missing enum value for PacketLogModuleSll2Layer.
Dimi1010
force-pushed
the
refactor/log-templates
branch
from
bd53c74 to
f21f85d
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## dev #1596 +/- ##
========================================
Coverage 83.16% 83.16%
========================================
Files 277 279 +2
Lines 48201 48403 +202
Branches 9932 10284 +352
========================================
+ Hits 40086 40255 +169
+ Misses 7266 7041 -225
- Partials 849 1107 +258
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Dimi1010
added
breaking change
Common++/header/Logger.h
Outdated
| std::ostringstream sstream; \ | ||
| sstream << message; \ |
There was a problem hiding this comment.
Can you please check if this change affects the PcapPlusPlus binary sizes?
I made a change in this PR a couple of years ago to significantly reduce the binary sizes: #748
There was a problem hiding this comment.
This is a breakdown on the install directory for the x64 release w/npcap.
Folder1 is the new size. Folder2 is the old size.
File Folder1Size Folder2Size AbsoluteDelta RelativeDelta
---- ----------- ----------- ------------- -------------
bin\Arping.exe 714240 639488 74752 10.47%
bin\ArpSpoofing.exe 703488 631296 72192 10.26%
bin\benchmark.exe 599552 562688 36864 6.15%
bin\DisplayLiveDevices.exe 30208 30208 0 0.00%
bin\DNSResolver.exe 748032 664064 83968 11.23%
bin\DnsSpoofing.exe 747520 671232 76288 10.21%
bin\HttpAnalyzer.exe 832512 743936 88576 10.64%
bin\IcmpFileTransfer-catcher.exe 730624 658432 72192 9.88%
bin\IcmpFileTransfer-pitcher.exe 742400 666112 76288 10.28%
bin\IPDefragUtil.exe 711680 655360 56320 7.91%
bin\IPFragUtil.exe 693248 637952 55296 7.98%
bin\PcapPrinter.exe 658432 612352 46080 7.00%
bin\PcapSearch.exe 661504 618496 43008 6.50%
bin\PcapSplitter.exe 720384 667648 52736 7.32%
bin\SSLAnalyzer.exe 822272 733184 89088 10.83%
bin\TcpReassembly.exe 799232 710144 89088 11.15%
bin\TLSFingerprinting.exe 796672 712704 83968 10.54%
lib\Common++.lib 5045102 4887832 157270 3.12%
lib\Packet++.lib 19849740 18454316 1395424 7.03%
lib\Pcap++.lib 3620658 2896688 723970 20.00%
Full breakdown: npcap-x64-release.txt
There was a problem hiding this comment.
Did you test it with MSVC or MinGW? Also, did you have a chance to test on Linux also?
I don't remember in which platform the impact was the highest, but I think it was Linux 🤔
There was a problem hiding this comment.
Msvc, I dont think I have mingw setup. I should be able to test the sizes on Unix via WSL.
There was a problem hiding this comment.
@Dimi1010 I see you're still working on it. Please let me know when the changes are done and when you want me to review the PR again.
It'd be good to make sure binary sizes don't increase. I'd test at least on Windows and Linux
There was a problem hiding this comment.
@seladb Ok, ended up keeping the optimization.
As bc82caf: Release builds ended up being:
- Windows
- Static Libraries
- Common++.lib - 4.7MB -> 5MB
- Packet++.lib - 17.6MB -> 19.7MB
- Pcap++.lib - 2.8MB -> 3.3MB
- Examples
- Same binary size or <=10KB increase.
- Static Libraries
- Linux (WSL)
- Static Libraries
- libCommon++.a - 408KB -> 444KB
- libPakcet++.a - 3.1MB -> 3.3MB
- libPcap++.a - 676KB -> 748KB
- Examples
- On average ~0.1MB increase on each binary.
- Static Libraries
This reverts commit be95ab8.
- Added optional compile time elimination of log calls below set level.
… allowed pool size.
- Changed infinite pool size to be maximum value of size_t instead of 0, to fix an issue if max size is set to 0. - Added size getter to the pool. - Marked the pool mutex as mutable.
…alse positive memory leak.
# Conflicts: # Common++/CMakeLists.txt
Dimi1010
force-pushed
the
refactor/log-templates
branch
from
d9df72b to
421a419
Compare
- Changed `pcpp::Logger::getInstance()` to use a cached `logger` variable. - Removed the need for fully qualified names in the logger test. - C-style casts to Cpp casts.
…stream from Logger.h
| { | ||
| // We don't need the lock anymore, so release it. | ||
| lock.unlock(); | ||
| return new T(); |
There was a problem hiding this comment.
If the pool is empty it allows acquiring any number of objects. Meaning, if the pool is initialized with preallocate = 0 which is the default, the user can call acquireObject() an infinite number of times before they call releaseObject(). This is a bit confusing assuming the pool has a pre-configured size (maxPoolSize)
There was a problem hiding this comment.
The pool has a max size to the objects it will store. Not the objects it will generate. The user should not concern himself if the pool is empty before asking for a new object because the pool will always provide him an object. Whether the object is brand new or reused is an implementation detail of the pool.
PS: That was the reason why originally there wasn't a way to check the current size of the pool.
There was a problem hiding this comment.
This is somewhat counter-intuitive because pools are usually used to pre-allocate objects and offer them to the user instead of just-in-time memory allocation. This pool is unique because it offers pre-allocated objects if it has any, otherwise it allocates new one, and the user doesn't know / cannot control whether memory is allocated or not 🤔
There was a problem hiding this comment.
I suppose it can seem so. Although I have seen many pools that do just-in-time allocations when needed. For example EFCore's DbContextPool implementation.
The options for a pool when a new entity is required and it is empty are three:
- Return an error.
- JIT a new instance of the object.
- Wait for another thread to release an object.
I preferred option 2, because the pool designed for a drop in replacement for somewhere where the objects would already be JIT-ed. It just provides optimization for reuse if it can, and just works if it can't.
When we don't want to cancel the operation Option 1 is just Option 2 or Option 3 with extra steps because the user has to do the checks manually at every call site. There is also the case for release, to mirror the acquire function, it should also return an error when the pool is already full, which is again Option 2 with extra steps where the object needs to be manually released.
Option 3 isn't desirable because it would essentially block the operation for an unknown amount of time.
There was a problem hiding this comment.
In object-pools objects are usually pre-allocated so an empty pool essentially means "out of memory", thus Option 2 suggests increasing the total memory beyond what was allocated initially. In our case since pre-allocation is optional we don't know if an empty pool means "out of memory" or not. The ways to solve it are:
- Force pre-allocating of all objects when the pool is created
- Give the option to avoid pre-allocation, but count the number of objects that were allocated
If we choose one of these options we can also let the user decide what to do when the pool is empty via a flag in the c'tor (return an error or JIT a new instance and increase maxPoolSize).
I agree Option 3 isn't desirable.
There was a problem hiding this comment.
Yes, they are more common, but dynamic object pools have been used where the the demands can fluctuate between common low number of active objects and occasional spikes, which I believe is the current use case for LogContext.
Demands for LogContext will usually be in the range of 1~3 objects based on the number of threads that are simultaneously using the pcpp logger. There is however the option for the user to spawn many capture threads and the logs to become congested. This is my rationale for using a dynamic pool with 2 initial size and 10 soft cap (maxPoolSize) where objects released to the pool after there are 10 objects in the pool won't be stored, as the likelihood of them being needed again is low. (that would require 10+ threads logging something at the exact same moment)
Also the pool is technically not supposed to be a user facing class but an implementation detail made for the logger, so I think it might be beneficial to shift it to the internals namespace.
There was a problem hiding this comment.
Since you already implemented an object pool we can consider using it in other places as well (although I can't think of a specific example right now). If you have a strong opinion about making it a dynamic object pool, maybe we can rename the class to DynamicObjectPool and document what it does, so in the future we can also implement a more "standard" object pool?
I agree we can move it to the internal namespace so it's not exposed to library users
|
|
||
| #include "ObjectPool.h" | ||
|
|
||
| PTF_TEST_CASE(TestObjectPool) |
There was a problem hiding this comment.
We can add another tests that runs clear()
| /// @brief Logs a message with the given source, level, and message. | ||
| /// @param message The log message. | ||
| void log(std::unique_ptr<internal::LogContext> message); |
There was a problem hiding this comment.
Do we use this overload of log() anywhere?
There was a problem hiding this comment.
Not atm. I initially thought it might be used in the macros, but they ended up using emit directly as they already checked the log level.
Overhaul of the Logging Infrastructure
Changes
LogLevelhas been updated to anenum classand moved to thepcppnamespace.Loggerretains aliases to the new enum.LogModulevalues in theLoggingModuleenum.LogSource: Encapsulates file, function, line number and module information.LogContext: Encapsulates the context of a single log message.Loggerclass into public methods with documentation.internalCreateLogStreamwithcreateLogContext.internalPrintLogMessagewithemit.log*functions toLoggerclass to allow logging, through runtime methods.LogContextto reduce allocation overhead.ObjectPool<T>class to facilitate pooling for reusable objects.useContextPooling(false)on theLoggerinstance.PCPP_LOGmacros at compile time viaPCPP_ACTIVE_LOG_LEVELpreprocessor variable.std::mutexlocking indefaultLogPrinterto eliminate possible data races during log message emission.ostream& operator<<overloads forIPAddressandMacAddressinsidepcppto satisfy ADL (Argument-Dependent Lookup).LogContext Pooling Optimization
When enabled, the pooling optimization introduces the following behavior:
Allocation on
createLogContext:createLogContextis called, the pool is queried for an availableLogContext.LogContextis found, a new one is created immediately.Releasing on
emit:emitmethod with aunique_ptr<LogContext>parameter is called, theLogContextis returned to the pool after the operation completes.LogContextis stored for reuse.LogContextis deleted.Optimization Benchmarks
Performance benchmarks highlight the impact of enabling the pooling optimization:
Windows 10 - MSVC x64 (Release Build)
WSL (Ubuntu 22.04) - GCC x64 (Release Build)
Notes
LogLeveland other enums may introduce minor compatibility issues, they align with modern C++ best practices.