Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thread-safety and cancellation handling issues in PublishDelayAsync with delay time < 1 minute #1637

Open
amimelia opened this issue Jan 14, 2025 · 0 comments · May be fixed by #1638
Open

Comments

@amimelia
Copy link

Version: 8.3.2

Problem Summary

The ICapPublisher.PublishDelayAsync method in the .NET Core CAP library exhibits thread-safety and cancellation handling issues when the delay time is less than 1 minute. These issues can lead to silent failures and message accumulation in the database.


Detailed Problems

  • Thread-safety Issue:

    • The Dispatcher.EnqueueToScheduler method uses a non-thread-safe PriorityQueue to manage delayed messages.
    • Under race conditions, the PriorityQueue can return null messages.
    • This causes the fire-and-forget task (Dispatcher: line 80) to fail silently, with no error logging.
    • As a result, delayed message processing halts, and future scheduled messages are not sent until the application restarts.
  • Cancellation Handling Issue:

    • The _delayCts cancellation token is initialized only once.
    • Once _delayCts is cancelled, subsequent calls to await Task.Delay(new TimeSpan(delayTime), _delayCts.Token) throw an OperationCanceledException.
    • The exception is ignored in the catch block, leading to a while (TryPeek) loop that spins continuously until the delay time expires.
  • Consequences of Silent Failures:

    • Delayed messages continue to accumulate in the database without being processed.
    • Upon application restart, querying delayed messages without a limit causes the database to timeout.
    • Manual intervention is required to delete unprocessed messages and restore application functionality.

Steps to Reproduce

  1. Call ICapPublisher.PublishDelayAsync concurrently with delay times less than 1 minute.
  2. Observe that:
    • Some messages are not processed.
    • Future scheduled messages are not sent.

I have fixed the above problems while maintaining existing functionality as closely as possible. I will open a pull request with tests and code changes to address the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant