Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat] Add observability to OutOfMemoryErrors when OutOfMemoryPolicy is FallbackToHeap #4534

Open
lhotari opened this issue Dec 5, 2024 · 1 comment

Comments

@lhotari
Copy link
Member

lhotari commented Dec 5, 2024

FEATURE REQUEST

Currently when direct memory allocation fails with OutOfMemoryError, the default OutOfMemoryPolicy is FallbackToHeap and the exception is catched silently:

} catch (OutOfMemoryError e) {
if (canFallbackToHeap && outOfMemoryPolicy == OutOfMemoryPolicy.FallbackToHeap) {
try {
return unpooledAllocator.heapBuffer(initialCapacity, maxCapacity);
} catch (OutOfMemoryError e2) {
consumeOOMError(e2);
throw e2;
}
} else {
// ThrowException
consumeOOMError(e);
throw e;
}
}

It would be great to be able to observe this condition. Pulsar uses ByteBufAllocatorImpl also for Pulsar client and it's hard to detect that Netty direct buffer direct access isn't properly configured unless there's a way to observe the condition.
Netty direct memory buffer access is explained in Pulsar client's "Java client Performance considerations", https://pulsar.apache.org/docs/4.0.x/client-libraries-java-setup/#java-client-performance .

After there's a counter in ByteBufAllocatorImpl, it would be possible to implement a metric in Pulsar client.

@lhotari lhotari changed the title [feat] Add observability to [feat] Add observability to OutOfMemoryErrors when OutOfMemoryPolicy is FallbackToHeap Dec 5, 2024
@lhotari
Copy link
Member Author

lhotari commented Dec 5, 2024

I guess having a metric for heapBuffer allocations would also be a way to notice the rate of heapBuffer allocations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant