Crash due to market data connection broken

[emretezel]

Hi Everyone,

Just wanted to check whether any of you frequently experience in live trading stack handler crashes due to IB market data farm connection getting broken. Happens to me quite a lot.

2022-12-07 15:49:16 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 376, 'strategy_name': 'prod', 'instrument_code': 'CAD_micro', 'contract_order_id': 129, 'instrument_order_id': 153} Sending order (Order ID:129) Type best for prod/CAD_micro/20221200, qty [-1], fill [0]@ price, None Parent:153 Children:no_children to algo sysexecution.algos.algo_original_best.algoOriginalBest
2022-12-07 15:49:16 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 376, 'component': 'ibFuturesContractData'} Reqid -1: 2119 Market data farm is connecting:usfuture
2022-12-07 15:49:16 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 376, 'component': 'ibContractPositionData'} Reqid -1: 2119 Market data farm is connecting:usfuture
2022-12-07 15:49:16 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 376, 'component': 'ibFuturesContractPriceData'} Reqid -1: 2119 Market data farm is connecting:usfuture
2022-12-07 15:49:16 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 376, 'component': 'ibFuturesContractPriceData'} Reqid -1: 2119 Market data farm is connecting:usfuture
2022-12-07 15:49:17 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 376, 'component': 'ibFuturesContractData'} Reqid -1: 2103 Market data farm connection is broken:usfuture
2022-12-07 15:49:17 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 376, 'component': 'ibContractPositionData'} Reqid -1: 2103 Market data farm connection is broken:usfuture
2022-12-07 15:49:17 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 376, 'component': 'ibFuturesContractPriceData'} Reqid -1: 2103 Market data farm connection is broken:usfuture
2022-12-07 15:49:17 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 376, 'component': 'ibFuturesContractPriceData'} Reqid -1: 2103 Market data farm connection is broken:usfuture
2022-12-07 15:49:23 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 376, 'component': 'ibFuturesContractData'} Reqid -1: 2103 Market data farm connection is broken:usfuture
2022-12-07 15:49:23 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 376, 'component': 'ibContractPositionData'} Reqid -1: 2103 Market data farm connection is broken:usfuture
2022-12-07 15:49:23 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 376, 'component': 'ibFuturesContractPriceData'} Reqid -1: 2103 Market data farm connection is broken:usfuture
2022-12-07 15:49:23 {'type': 'stack_handler', 'broker': 'IB', 'clientid': 376, 'component': 'ibFuturesContractPriceData'} Reqid -1: 2103 Market data farm connection is broken:usfuture
Traceback (most recent call last):
File "/home/pysystemtrade/opt/pysystemtrade/sysproduction/linux/scripts/run.py", line 65, in
func(*args, **kwargs)
File "/home/pysystemtrade/opt/pysystemtrade/sysproduction/run_stack_handler.py", line 11, in run_stack_handler
price_process.run_process()
File "/home/pysystemtrade/opt/pysystemtrade/syscontrol/run_process.py", line 103, in run_process
self._main_loop_over_methods()
File "/home/pysystemtrade/opt/pysystemtrade/syscontrol/run_process.py", line 126, in _main_loop_over_methods
timer_class.check_and_run()
File "/home/pysystemtrade/opt/pysystemtrade/syscontrol/timer_functions.py", line 124, in check_and_run
self.run_function()
File "/home/pysystemtrade/opt/pysystemtrade/syscontrol/timer_functions.py", line 299, in run_function
self._function()
File "/home/pysystemtrade/opt/pysystemtrade/sysexecution/stack_handler/create_broker_orders_from_contract_orders.py", line 43, in create_broker_orders_from_contract_orders
self.create_broker_order_for_contract_order(contract_order_id)
File "/home/pysystemtrade/opt/pysystemtrade/sysexecution/stack_handler/create_broker_orders_from_contract_orders.py", line 59, in create_broker_order_for_contract_order
algo_instance_and_placed_broker_order_with_controls = self.send_to_algo(
File "/home/pysystemtrade/opt/pysystemtrade/sysexecution/stack_handler/create_broker_orders_from_contract_orders.py", line 254, in send_to_algo
placed_broker_order_with_controls = algo_instance.submit_trade()
File "/home/pysystemtrade/opt/pysystemtrade/sysexecution/algos/algo_original_best.py", line 58, in submit_trade
placed_broker_order_with_controls = self.prepare_and_submit_trade()
File "/home/pysystemtrade/opt/pysystemtrade/sysexecution/algos/algo_original_best.py", line 106, in prepare_and_submit_trade
okay_to_do_limit_trade = limit_trade_viable(ticker_object = ticker_object,
File "/home/pysystemtrade/opt/pysystemtrade/sysexecution/algos/algo_original_best.py", line 220, in limit_trade_viable
if adverse_size_issue(ticker_object, wait_for_valid_tick=True, log=log):
File "/home/pysystemtrade/opt/pysystemtrade/sysexecution/algos/algo_original_best.py", line 340, in adverse_size_issue
latest_imbalance_ratio_exceeded = _is_imbalance_ratio_exceeded(
File "/home/pysystemtrade/opt/pysystemtrade/sysexecution/algos/algo_original_best.py", line 358, in _is_imbalance_ratio_exceeded
latest_imbalance_ratio = current_tick_analysis.imbalance_ratio
AttributeError: 'named_object' object has no attribute 'imbalance_ratio'

[emretezel]

Usually works again when I restart the IB Gateway and stack handler.

Do you guys usually update always to the latest version of IB Gateway?

[robcarver17]

I had this problem but restarting the stack handler did the trick (I didn't have to restart the gateway).

More worrying, this shouldn't crash - it should fail gracefully and carry on running, since psystemtrade is designed to recover from eg short internet outages. I wonder if the error being thrown is due to the refactoring done by @tgibson11

[emretezel]

I am pretty much at the head of master, except for today's check-ins. I agree with Rob that ideally the stack handler should wait until market data farm connection restores and don't crash. I really don't think this is an internet connection issue.

Let's see if others on running latest master also experience this.

BTW, sometimes I do log in via browser to the IB site to check capital, etc, but I do not login with trading or market data permissions, not sure whether this could be related.

Thanks all,

[tgibson11]

Just logging in to the IB website should not close a TWS or IB Gateway connection. And if you're trying to do something that will close it, they will warn you first (at least in my experience that has always been the case).

[robcarver17]

Just to say I don't think this is code that @tgibson11 has touched - I will fix it now.

[tgibson11]

ideally the stack handler should wait until market data farm connection restores and don't crash. I really don't think this is an internet connection issue.

Perhaps. But when I was having my problems, the IB gateway did not automatically reconnect even when internet connectivity was restored.

[tgibson11]

That's why it took me so long to figure out...from what I could see my internet connection always looked fine. It just looked like the Gateway was losing it's connection to IB and not doing anything about it. Restarting the Gateway (and stack handler) always solved the problem.

[tgibson11]

This may not be your issue (unless you use a USB wifi adapter), but I'll tell you my experience anyway.

I had been losing the connection several times per week. After trying different things, I eventually tried buying a new USB Wifi adapter, and that solved the problem. I haven't had a single lost connection for months now.

This site led me to my solution: https://github.com/morrownr/USB-WiFi

TL;DR, "there are only 2 companies supplying chipsets for USB Wifi adapters - Mediatek and Realtek." And Realtek does not make standards-compliant drivers that can be supported directly in the linux kernel.

Compounding the problem is that the 2 most popular brands of USB wifi adapter will switch chipsets without changing the model number, so you don't know what chipset you'll get before buying one of their products.

[tgibson11]

So, the error is ultimately coming from tick_data.py, line 245 (analyse_for_tick).

This returns missing_data (not missing_contract, which is the only thing I've touched so far.). So I think I'm exonerated.

This is exactly the type of error I'm trying to eliminate with my refactoring.

[robcarver17]

Behaviour now is (i) before an order is sent, if no prices detected, cancel the order. System will try and resubmit.
(ii) during order management, switch to a market order.

I appreciate the behaviour for (ii) perhaps isn't ideal, but it will require quite a bit of work to safely modify the code so it does something else without introducing more unexpected behaviour. I will add this to the issue stack, but it isn't a five minute job.

Alternatively you can locally patch to reproduce the previous behaviour and just throw an exception. That will leave the limit order in the ether until you manually fix the problem.

[ak2k]

Presumably not now, but whenever someone gets into this, to what degree could using a SNAP MID order type (including with an offset) be worth considering (even if as a stage) in the process? https://www.interactivebrokers.com/en/index.php?f=5934

I don't have particular experience with it, but it would seem on the surface attractive when compared with a market order.

[emretezel]

Do you think upgrading ib-insync package to the latest version (current required version seems to be fairly behind) would resolve this issue, or break pysystemtrade?

[tgibson11]

ib-insync seems to be pretty backward compatible. I've never had an issue upgrading to newer versions.

As to whether it will resolve your issue, I think it's unlikely, but it doesn't hurt to try.

Upgrading the IB Gateway would seem more likely to help, if you haven't done that yet. As to your much earlier question, I usually only upgrade the IB Gateway if I'm having problems (or forced to). I can't recall that updating it has ever caused a new problem, but it has occasionally made some go away.

[emretezel]

Just an update, upgraded IB Gateway version from 1012 to 1019, and in the last few days and have not observed any crashes (ps: I did not pulled the above mentioned code changed yet so still the old behaviour).

[oldlore]

I've had many crashes of IB gateway over the past 4 months. It subjectively seems to occur more often when logged into paper trading but happens with live trading as well. Most of the time this seems to happen during the overnight NA reset 23:45 - 00:45 ET. I'll wake up in the morning and find the dialog box open asking me if I want to reconnect. I click reconnect and it usually does so immediately.
run_stack_handler will usually have crashed. capital_update usually handles the crash fine for a short while but will also crash if it lasts too long. Here is an example of the stack_handler crash trace. This one happened just before the processes started so maybe crashing is normal behavior then? I can't remember the last time IB gateway crashed during the day but if one occurs I can post that.

Traceback (most recent call last):
  File "/home/pysystemtrade/sysdata/data_blob.py", line 126, in _add_ib_class
    resolved_instance = class_object(self.ib_conn, log=log)
  File "/home/pysystemtrade/sysdata/data_blob.py", line 274, in ib_conn
    ib_conn = self._get_new_ib_connection()
  File "/home/pysystemtrade/sysdata/data_blob.py", line 297, in _get_new_ib_connection
    raise e
  File "/home/pysystemtrade/sysdata/data_blob.py", line 286, in _get_new_ib_connection
    ib_conn = connectionIB(client_id, log=self.log)
  File "/home/pysystemtrade/sysbrokers/IB/ib_connection.py", line 72, in __init__
    ib.connect(ipaddress, port, clientId=client_id, account=account)
  File "/usr/local/lib/python3.8/dist-packages/ib_insync-0.9.70-py3.8.egg/ib_insync/ib.py", line 269, in connect
    return self._run(self.connectAsync(
  File "/usr/local/lib/python3.8/dist-packages/ib_insync-0.9.70-py3.8.egg/ib_insync/ib.py", line 308, in _run
    return util.run(*awaitables, timeout=self.RequestTimeout)
  File "/usr/local/lib/python3.8/dist-packages/ib_insync-0.9.70-py3.8.egg/ib_insync/util.py", line 332, in run
    result = loop.run_until_complete(task)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/dist-packages/ib_insync-0.9.70-py3.8.egg/ib_insync/ib.py", line 1658, in connectAsync
    await asyncio.wait_for(self.reqExecutionsAsync(), timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pysystemtrade/sysproduction/linux/scripts/run.py", line 65, in <module>
    func(*args, **kwargs)
  File "/home/pysystemtrade/sysproduction/run_stack_handler.py", line 11, in run_stack_handler
    price_process.run_process()
  File "/home/pysystemtrade/syscontrol/run_process.py", line 103, in run_process
    self._main_loop_over_methods()
  File "/home/pysystemtrade/syscontrol/run_process.py", line 126, in _main_loop_over_methods
    timer_class.check_and_run()
  File "/home/pysystemtrade/syscontrol/timer_functions.py", line 124, in check_and_run
    self.run_function()
  File "/home/pysystemtrade/syscontrol/timer_functions.py", line 299, in run_function
    self._function()
  File "/home/pysystemtrade/sysexecution/stack_handler/checks.py", line 51, in check_external_position_break
    data_broker = dataBroker(self.data)
  File "/home/pysystemtrade/sysproduction/data/generic_production_data.py", line 10, in __init__
    data = self._add_required_classes_to_data(data)
  File "/home/pysystemtrade/sysproduction/data/broker.py", line 54, in _add_required_classes_to_data
    data.add_class_list(broker_class_list)
  File "/home/pysystemtrade/sysdata/data_blob.py", line 83, in add_class_list
    self.add_class_object(class_object)
  File "/home/pysystemtrade/sysdata/data_blob.py", line 89, in add_class_object
    resolved_instance = self._get_resolved_instance_of_class(class_object)
  File "/home/pysystemtrade/sysdata/data_blob.py", line 94, in _get_resolved_instance_of_class
    resolved_instance = class_adding_method(class_object)
  File "/home/pysystemtrade/sysdata/data_blob.py", line 134, in _add_ib_class
    self._raise_and_log_error(msg)
  File "/home/pysystemtrade/sysdata/data_blob.py", line 330, in _raise_and_log_error
    raise Exception(error_msg)
Exception: Error  couldn't evaluate ibFxPricesData(self.ib_conn, log = self.log.setup(component = ibFxPricesData)) This might be because (a) IB gateway not running, or (
b) import is missing                         or (c) arguments don't follow pattern

So I run interactive_control, stop both processes, then run them again by either pasting the crontab code line in a terminal or changing the cron time to a few minutes from present then changing it back later. Since most of the crashes seem to happen overnight for me there are usually no problems with orders since the markets are mostly not in permitted trading hours.

[robcarver17]

I had a crash last night. I lost the processes that were running (update prices and reports) but when stack handler and capital update started this morning they were fine, so I didn't need to do anything manually.

But it looks like it was the normal auto log off (11:45pm). All the new markets I've added recently mean that everything is just taking longer to run. So for me it looks like pushing the daily prices run earlier in the day is the solution.

[emretezel]

The crash I reported at the beginning of the thread occurs when the time comes around 3:30p UK time and stack handlers starts sending broker orders.