Calling external services (Rest apis)/database CRUD operations within grain

[DhananjayKushwaha]

Hello team,

I am using Orleans framework in one of our projects. This is the pattern of calls from our grains.

Domain Service ->
Calls CustomerGrain.Save() ->
Call1 from CustomerGrain - CustomerService.Save (external service) -> Saves customer information in sql server
Call2 from CustomerGrain - ProfileService.Save (external service) -> Saves customer recent activity in sql server
Call3 from CustomerGrain - Repo.Save ->
Call1 from Repo.Save - Calls Stored proc 1 to save in sql server
Call2 from Repo.Save - Calls Stored proc 2 to save in sql server

1. All the calls are using asynchronous code.
2. Usually external service (using HttpClient) or db operation calls takes <1s for all the operations but a few calls may take longer (<30s) during load.

Would this call pattern cause any issues during high load?

Thanks
Dhananjay

[ReubenBond]

This pattern is fine: you can call external services from an Orleans grain without concern. As you identified, blocking code can cause a problem with the .NET Thread Pool with or without Orleans being involved. In other words, the same code would cause a problem in an ASP.NET application or any other .NET application. Orleans schedules all operations using .NET's ThreadPool.

Blocking tasks using methods such as task.Wait(), task.Result, & Task.WaitAll(tasks...) will cause your grain to deadlock because grains are single-threaded, and it will also quickly lead to ThreadPool starvation, so it's something which must be avoided.

Messaging timeouts will occur if your HTTP calls exceed the configured maximum response timeout for your grain calls.

It is highly recommended to profile any code intended for production under load to determine if there are blocking code paths or other inefficiencies. If you would like help analyzing CPU profiling traces, we can help.

[DhananjayKushwaha]

Thanks @ReubenBond for your response! We do have multiple tasks created for different external service calls + db calls in same method. Tasks are waiting with Task.WhenAll(tasks...) something like this:

CustomerGrain.Save()
Task1 = SaveService1()
Task2 = SaveService2()
Task3 = SaveToDB()
Task.WhenAll(task1, task2, task3);

I hope this should be OK.

Note - As you mentioned, we had faced issues with Tasks.WaitAll(tasks...) and corrected it

[ReubenBond]

Task.WhenAll is the right choice, just make sure you await the call

[DhananjayKushwaha]

Thanks @ReubenBond!