runq flagman is a long-living, supervised process which only monitors the system runq and ensures actions are taken when conditions are met.
By design, the process runs as high priority and avoids communicating with other processes.
runq flagman can also take its own scheduling roundtrip time into a factor (F5) to speed up the detection of system overload. This is done by taking timestamps when it is off and on the scheduler to check if there is a time leap. If the gap is large enough that is a sign that VM has very limited resources on scheduling.
| Term | Definition |
|---|---|
| Runq | The length of beam system run_queue, checked periodly, see T1 and T2. |
| Scheduler | Number of Online schedulers. |
| Credits | Gain credits when runq is under limit, lose credits when runq is over limit. |
| Priority Group | Process could join/leave prioriy groups. Load contol takes different actions when conditions are meet. |
| Overloaded | State when credits runs out. System is overloaded and action must be taken. |
| Warm | State when system is losing/gaining credits, a middle state preventing system is shifting between cold and overloaded |
| Cold | State when system is full of credits |
| Flag | Regisered process name when system is overloaded |
| Actions | Actions to take to cool down the system. Action is taken by either flagman or API caller |
Actions to take:
Control the flag
Brutal kill processes in low (<= F3) priority group in either of any conditions listed below
Caller get scheduled out for a limited amount of time if the system is overloaded
see load_ctl:maydelay
Caller is back on scheduler when flag is cleared.
TODO Caller could be scheduled out when the system is warm as well?
| parm name | type | default | comments |
|---|---|---|---|
| F0 | boolean() | true | Set it to false to disable the load control function. To reenable, you should call load_ctl:restart_runq_flagman/0. |
| F1 | integer() | 8 | scheduler multipler, runq is overlimit when runq > scheduler * F1 |
| F2 | float() | 0.8 | scheduler utilization limit for increasing credits |
| F3 | integer() | 2 | Priority threshold for killing processes in priority group forcefully. |
| F4 | integer() < 100 | 50 | credits left percentage. |
| F5 | integer() | 0 | Scheduler turnaround time threshold (in ms) for extra credits loss. 0 is off |
| T1 | integer() | 3000 (ms) | Regular runq check interval, see also T2 |
| T2 | integer() | 1000 (ms) | runq check interval when it is overlimit. See F1 |
| C2 | integer() | 3 | Cooldown credits. |
Similar to Runq Flagman, Memory Flagman monitors the system memory usage every T1 from
- /proc/meminfo
- sys/fs/cgroup
Once it is over threshold F1, it rasies the flag and alarm.
The main reason I drop this idea is that there seems no any benefit to make it a gen_statm because it should not handle any calls from other processes to minimize any undesired workload and let it be standalone.
fun is_overload()-> boolean()
