Constant reconnect fills akka DLA

[cddude229]

Old note from Main.scala - the issue is that our spin loop is too aggressive and isn't handling some failure messages (as far as I can tell):

// FIXME: When connecting, there are DLA issues at times.  Here's some logs to debug with:
//2018-09-18 17:11:59,102 -0700 [default-akka.actor.default-dispatcher-6] INFO  com.sumologic.hackathon.Main$ - Connecting to KQ cabinet
//2018-09-18 17:11:59,103 -0700 [default-akka.actor.default-dispatcher-6] INFO  com.sumologic.hackathon.Main$ - Attempt to start connection to cabinet done.  Waiting.
//2018-09-18 17:11:59,103 -0700 [default-akka.actor.default-dispatcher-6] INFO  com.sumologic.hackathon.Main$ - Closed connection to cabinet.  Retrying connection
//[INFO] [09/18/2018 17:11:59.103] [default-akka.actor.default-dispatcher-4] [akka://default/system/StreamSupervisor-0/flow-7-1-actorRefSource] Message [akka.http.scaladsl.model.ws.TextMessage$Strict] without sender to Actor[akka://default/system/StreamSupervisor-0/flow-7-1-actorRefSource#1227979493] was not delivered. [6] dead letters encountered. If this is not an expected behavior, then [Actor[akka://default/system/StreamSupervisor-0/flow-7-1-actorRefSource#1227979493]] may have terminated unexpectedly, This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
//2018-09-18 17:11:59,114 -0700 [default-akka.actor.default-dispatcher-6] DEBUG com.sumologic.hackathon.state.StateMachine - Reset state machine
//2018-09-18 17:11:59,161 -0700 [default-akka.actor.default-dispatcher-6] INFO  com.sumologic.hackathon.Main$ - Connecting to KQ cabinet
//2018-09-18 17:11:59,161 -0700 [default-akka.actor.default-dispatcher-6] INFO  com.sumologic.hackathon.Main$ - Attempt to start connection to cabinet done.  Waiting.
//2018-09-18 17:11:59,161 -0700 [default-akka.actor.default-dispatcher-6] INFO  com.sumologic.hackathon.Main$ - Closed connection to cabinet.  Retrying connection
//[INFO] [09/18/2018 17:11:59.161] [default-akka.actor.default-dispatcher-3] [akka://default/system/StreamSupervisor-0/flow-8-1-actorRefSource] Message [akka.http.scaladsl.model.ws.TextMessage$Strict] without sender to Actor[akka://default/system/StreamSupervisor-0/flow-8-1-actorRefSource#1701609517] was not delivered. [7] dead letters encountered. If this is not an expected behavior, then [Actor[akka://default/system/StreamSupervisor-0/flow-8-1-actorRefSource#1701609517]] may have terminated unexpectedly, This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
//2018-09-18 17:11:59,180 -0700 [default-akka.actor.default-dispatcher-6] DEBUG com.sumologic.hackathon.state.StateMachine - Reset state machine
//2018-09-18 17:11:59,274 -0700 [default-akka.actor.default-dispatcher-6] INFO  com.sumologic.hackathon.Main$ - Connecting to KQ cabinet
//2018-09-18 17:11:59,274 -0700 [default-akka.actor.default-dispatcher-6] INFO  com.sumologic.hackathon.Main$ - Attempt to start connection to cabinet done.  Waiting.
//2018-09-18 17:11:59,274 -0700 [default-akka.actor.default-dispatcher-6] INFO  com.sumologic.hackathon.Main$ - Closed connection to cabinet.  Retrying connection
//[INFO] [09/18/2018 17:11:59.274] [default-akka.actor.default-dispatcher-12] [akka://default/system/StreamSupervisor-0/flow-9-1-actorRefSource] Message [akka.http.scaladsl.model.ws.TextMessage$Strict] without sender to Actor[akka://default/system/StreamSupervisor-0/flow-9-1-actorRefSource#2007154941] was not delivered. [8] dead letters encountered. If this is not an expected behavior, then [Actor[akka://default/system/StreamSupervisor-0/flow-9-1-actorRefSource#2007154941]] may have terminated unexpectedly, This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
//2018-09-18 17:11:59,282 -0700 [default-akka.actor.default-dispatcher-6] DEBUG com.sumologic.hackathon.state.StateMachine - Reset state machine
//2018-09-18 17:11:59,283 -0700 [default-akka.actor.default-dispatcher-6] INFO  com.sumologic.hackathon.Main$ - Connecting to KQ cabinet
//2018-09-18 17:11:59,283 -0700 [default-akka.actor.default-dispatcher-6] INFO  com.sumologic.hackathon.Main$ - Attempt to start connection to cabinet done.  Waiting.
//2018-09-18 17:11:59,283 -0700 [default-akka.actor.default-dispatcher-12] INFO  com.sumologic.hackathon.Main$ - Closed connection to cabinet.  Retrying connection
//[INFO] [09/18/2018 17:11:59.284] [default-akka.actor.default-dispatcher-13] [akka://default/system/StreamSupervisor-0/flow-10-1-actorRefSource] Message [akka.http.scaladsl.model.ws.TextMessage$Strict] without sender to Actor[akka://default/system/StreamSupervisor-0/flow-10-1-actorRefSource#-1540068731] was not delivered. [9] dead letters encountered. If this is not an expected behavior, then [Actor[akka://default/system/StreamSupervisor-0/flow-10-1-actorRefSource#-1540068731]] may have terminated unexpectedly, This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
//2018-09-18 17:11:59,318 -0700 [default-akka.actor.default-dispatcher-12] DEBUG com.sumologic.hackathon.state.StateMachine - Reset state machine
//2018-09-18 17:11:59,319 -0700 [default-akka.actor.default-dispatcher-12] INFO  com.sumologic.hackathon.Main$ - Connecting to KQ cabinet
//2018-09-18 17:11:59,320 -0700 [default-akka.actor.default-dispatcher-12] INFO  com.sumologic.hackathon.Main$ - Attempt to start connection to cabinet done.  Waiting.
//2018-09-18 17:11:59,320 -0700 [default-akka.actor.default-dispatcher-13] INFO  com.sumologic.hackathon.Main$ - Closed connection to cabinet.  Retrying connection
//[INFO] [09/18/2018 17:11:59.320] [default-akka.actor.default-dispatcher-3] [akka://default/system/StreamSupervisor-0/flow-11-1-actorRefSource] Message [akka.http.scaladsl.model.ws.TextMessage$Strict] without sender to Actor[akka://default/system/StreamSupervisor-0/flow-11-1-actorRefSource#1949080542] was not delivered. [10] dead letters encountered, no more dead letters will be logged. If this is not an expected behavior, then [Actor[akka://default/system/StreamSupervisor-0/flow-11-1-actorRefSource#1949080542]] may have terminated unexpectedly, This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.