Backtest oddity #796

rorymac · 2022-10-05T14:40:20Z

rorymac
Oct 5, 2022

TLDR:
Doing backtests I get an odd result:
The daily returns from the backtest "overflow" the last date of the data.
The current data I have ends on the 2022-10-04, the daily return output continues to the 2022-12-12 with some zeros and occasional returns.
There are no error messages and the code does not crash.
The same DB is used for production and that is working without errors.

Problem Statement:
I am using sha 39a9df3
When running a backtest either DO or static, the daily returns output overflows the last date of the data available.

2022-10-03	-1.89485500160467
2022-10-04	-1.103998077567
2022-10-05	0
2022-10-06	0
2022-10-07	0
2022-10-10	0
2022-10-11	0
2022-10-12	0
2022-10-13	-0.002371468498625
2022-10-14	-0.001022972487589
2022-10-17	0
2022-10-18	0
2022-10-19	0
2022-10-20	0
2022-10-21	0
2022-10-24	0
2022-10-25	0
2022-10-26	0
2022-10-27	0
2022-10-28	0
2022-10-31	0
2022-11-01	0
2022-11-02	0
2022-11-03	0
2022-11-04	0
2022-11-07	0
2022-11-08	0
2022-11-09	0
2022-11-10	0
2022-11-11	0
2022-11-14	0
2022-11-15	-0.017393304988872
2022-11-16	0
2022-11-17	0
2022-11-18	0
2022-11-21	0
2022-11-22	0
2022-11-23	0
2022-11-24	0
2022-11-25	-0.001073529411765
2022-11-28	0
2022-11-29	0
2022-11-30	0
2022-12-01	0
2022-12-02	0
2022-12-05	0
2022-12-06	0
2022-12-07	0
2022-12-08	0
2022-12-09	0
2022-12-12	0

Code used to get the daily returns and put them into a csv is as follows:

###test
from private.test_backtest_static.estimatedsystem import futures_system
from systems.diagoutput import systemDiag
import pandas as pd
system=futures_system()
system.set_logging_level("on")
stats = system.accounts.portfolio().percent.stats() ## see some statistics
print(stats)
sysdiag = systemDiag(system)
#send the daily returns to a file
acc_curve=system.accounts.portfolio()
pnl_series = acc_curve.net.daily.percent.as_ts
pnl_series.to_csv('daily_pnl_percent.csv')

The same code snippet is used for a DO backtest.

What have I tried?

My first thoughts were that my DB was corrupted.
I went and inspected all the adjusted prices and the multiple prices to check there were no extra dates floating around.
That was fine.

Has anyone else experienced this?

Answered by MarkSurfNZ

Oct 11, 2022

The following relates to the original question regarding returns flowing past the last data date, not the Mongo/artic discussion:

Pysys ver 1.47
I have found the issue; and, yes, it does relate to costs.
For reference, I am running a back test using one instrument, BUXL, with the momentum16 rule, using data from 2000-01-01 to 2022-09-19.
When the back test gets to the point where it shows the message, “Calculating buffered subsystem positions” the problem occurs in the _pseudo_fills_for_year method of the pandlCalculationWithCashCostsAndFills class, at line 114 of the pandl_cash_costs.py module. This method creates a date list for each year by generating the same number of equally-spaced …

View full answer

robcarver17 · 2022-10-06T08:24:48Z

robcarver17
Oct 6, 2022
Maintainer

Best bet to debug this is to work backwards, or forwards, and try and work out where the extra data is creeping in.

I'd set up a system with one trading rule and one instrument, and then step forward through the various stages.

4 replies

rorymac Oct 6, 2022
Author

Thanks Rob, I will give that a go, unfortunately in the transitions (1.45 & 1.47) I must have blown the DB up somehow and awkwardly have only the old versions in backup.

robcarver17 Oct 6, 2022
Maintainer

You wouldn't need the old version to do this?

rorymac Oct 6, 2022
Author

By old versions I meant old versions of the mongo dump prior to 1.47.
<<You wouldn't need the old version to do this?>>
True, but at worst I may have to reconstitute the DB from the old version, roll back to prior to 1.45 and go through the process again. OR
try and reconstitute the DB from the csv files which look clean (I have checked all of them adjusted, multiple and the latest contract for date errors and they appear fine)

I have got as far as doing a run with only BOBL and it has thrown an odd date, rather randomly 2022-11-15. The next instrument did the same.
I need to spend a bit more time working through the code and pulling out intermediate results.

MarkSurfNZ Oct 9, 2022

I am seeing the same issue. I use my own data, which I convert to csv pysys format. We are seeing the same thing with different data, which indicates it may not be a data issue. My gut and initial research tells me it is costs (rather than returns) somehow propagating forward. I am working on this and will report if/when I come up with anything.

rorymac · 2022-10-08T17:37:05Z

rorymac
Oct 8, 2022
Author

This is getting interesting.
I am running mongoDB version 5.0.12 on one server and 4.4.17 on the backup server.
I notice that in Rob's quickstart doc, that arctic is said not to run with MongoDB >= 2.4.x <= 4.2.8 (via the link)
Is anyone else running a later version of Mongo?
The recommended version is <=4.2.8 and on Mongo, only 4.2.23 is available.
That however fails to install on Mint20 as it lacks a dependency that is not available.

So anyone using mongoDB ver 5.0.12 or 4.4.17 without issues?

0 replies

tgibson11 · 2022-10-08T20:33:36Z

tgibson11
Oct 8, 2022

I've been using 4.4.17 without issues for some time (over a year, I think). However...I'm also using pymongo 3.11.3, and arctic 1.79.4, which are both later versions than Rob recommends. I did these upgrades to resolve one or more issues (not just on a whim). I remember that the pymongo upgrade was specifically related to poor performance. The arctic upgrade might have been related, or something entirely different, but I did do them both around the same time (March 2021).

…

On Sat, Oct 8, 2022 at 10:37 AM Rory Mackay ***@***.***> wrote: This is getting interesting. I am running mongoDB version 5.0.12 on one server and 4.4.17 on the backup server. I notice that in Rob's quickstart doc, that arctic is said not to run with MongoDB >= 2.4.x <= 4.2.8 (via the link) Is anyone else running a later version of Mongo? The recommended version 4.2.8 is not available from Mongo, only 4.2.23 That however fails to install on Mint20 as it lacks a dependency that is not available. So anyone using mongoDB ver 5.0.12 or 4.4.17 without issues? — Reply to this email directly, view it on GitHub <#796 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA3JWKV2UHGB2DGSNMUEZ2LWCGWM3ANCNFSM6AAAAAAQ5UTKLU> . You are receiving this because you are subscribed to this thread.Message ID: <robcarver17/pysystemtrade/repo-discussions/796/comments/3829905@ github.com>

1 reply

rorymac Oct 8, 2022
Author

Thanks for that, yes I have had no problems with the later versions.
I have somehow screwed my DB (see above) and Thought that I should set up a virtual machine and try to reinstall the db,
firstly from the csv files and secondly if that does not work from scratch.
Hence the current adventure ...

MarkSurfNZ · 2022-10-11T02:35:27Z

MarkSurfNZ
Oct 11, 2022

The following relates to the original question regarding returns flowing past the last data date, not the Mongo/artic discussion:

Pysys ver 1.47
I have found the issue; and, yes, it does relate to costs.
For reference, I am running a back test using one instrument, BUXL, with the momentum16 rule, using data from 2000-01-01 to 2022-09-19.
When the back test gets to the point where it shows the message, “Calculating buffered subsystem positions” the problem occurs in the _pseudo_fills_for_year method of the pandlCalculationWithCashCostsAndFills class, at line 114 of the pandl_cash_costs.py module. This method creates a date list for each year by generating the same number of equally-spaced dates within the year as there are rolls per year. So, for BUXL it creates 4 dates, 3 months apart, with the first date 1.5 months after the start of the calendar year, and the last date 1.5 months prior to the finish. The problem is that it when it does this for the last year in the data it does not consider where the data ends. Therefore, the last date it creates in the list is 2022-11-15. The method then combines these dates with the “average holding quantity by period” to get the return list, which shows dates and pseudo fills. Because it is showing a pseudo-fill for 2022-11-15 it causes the daily returns (costs actually) to continue through to that date rather than end on the last data date. This end date will vary for different instruments, depending the number of rolls per year.
I don’t have a simple fix for this and am curious as to what Rob thinks.
I am a Github Noob, so not sure what to do next. Do I open an issue? Separately, I have found a different issue where Pysys creates NaN FDMs early in an instrument’s data history. I do have a (hacky) simple solution for that one. Should I raise that as an issue also? I appreciate any advice.

2 replies

robcarver17 Oct 11, 2022
Maintainer

I will fix the roll costs problem, but if you can raise a seperate issue for the NAN FDM that would be good. Since your fix is 'hacky' would be better as an issue rather than a pull request, but if you can include the fix that would help. A well founded, non hacky, fix could be a PR.

MarkSurfNZ Oct 11, 2022

My fix is OK, but there may well be a better one, so will raise an issue.
Love your work BTW.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backtest oddity #796

{{title}}

Replies: 4 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Backtest oddity #796

rorymac Oct 5, 2022

Replies: 4 comments · 7 replies

robcarver17 Oct 6, 2022 Maintainer

rorymac Oct 6, 2022 Author

robcarver17 Oct 6, 2022 Maintainer

rorymac Oct 6, 2022 Author

MarkSurfNZ Oct 9, 2022

rorymac Oct 8, 2022 Author

tgibson11 Oct 8, 2022

rorymac Oct 8, 2022 Author

MarkSurfNZ Oct 11, 2022

robcarver17 Oct 11, 2022 Maintainer

MarkSurfNZ Oct 11, 2022

rorymac
Oct 5, 2022

Replies: 4 comments 7 replies

robcarver17
Oct 6, 2022
Maintainer

rorymac Oct 6, 2022
Author

robcarver17 Oct 6, 2022
Maintainer

rorymac Oct 6, 2022
Author

rorymac
Oct 8, 2022
Author

tgibson11
Oct 8, 2022

rorymac Oct 8, 2022
Author

MarkSurfNZ
Oct 11, 2022

robcarver17 Oct 11, 2022
Maintainer