P1: Can't do sample plot of term structure with today's date (after close) where date of last day market open works. #18

AQuant206 · 2023-03-27T23:57:12Z

I got sample_plot_term_structure.py working over weekend and it seems to work with yesterday's date (e.g. 2023-03-24 on Sat 3-25). However it doesn't work with today's date well after 16:00 Central/CBOE time (e.g. 4:50PM Pacific), when Yahoo finance has CBOE values for for 1st and 2nd month forward contracts after today's close.

You should be able to repro this by editing src/examples/sample_plot_term_structure.py - changing the day_of_interest value to today's date (after market close). E.g.

    #plot the term structure for Mon Mar 27, 2023
    day_of_interest = '2023-03-27'
    df_day_of_interest =wide_with_continuous_futures.loc[[day_of_interest]]

This is the output from running the sample with above change. It works if I set day_of_interest to '2023-03-24' and Yahoo finance has CBOE values for market close for 1st and 2nd month forward contracts.

********************************************************************************Running python file:
z:\Downloads\Source-Master\vix_utils\examples\aq_sample_plot_today_term_structure.py

WARNING:root:
Duplicates detected
Trade Date Weekly Tenor_Monthly Tenor_Days Tenor_Trade_Days Expiry Open High Low Close Settle Change Total Volume EFP Open Interest Year MonthOfYear Futures
File Expired
37824 2020-02-27 00:00:00-05:00 True 2.0 26 19.0 2020-03-25 00:00:00-04:00 24.15 25.35 22.15 24.25 25.025 3.000 48 0 63 2020 3 H (Mar 2020) 2020-03-25.w_.CFE_VX_2020.csv True
37825 2020-02-27 00:00:00-05:00 True 2.0 26 19.0 2020-03-25 00:00:00-04:00 24.15 25.35 22.15 24.25 25.025 3.000 48 0 63 2020 3 H (Mar 2020) 2020-03-25.w_.CFE_VX_2020.csv True
37839 2020-02-28 00:00:00-05:00 True 2.0 25 18.0 2020-03-25 00:00:00-04:00 26.50 27.50 25.65 26.50 26.100 1.075 112 0 139 2020 3 H (Mar 2020) 2020-03-25.w_.CFE_VX_2020.csv True
37840 2020-02-28 00:00:00-05:00 True 2.0 25 18.0 2020-03-25 00:00:00-04:00 26.50 27.50 25.65 26.50 26.100 1.075 112 0 139 2020 3 H (Mar 2020) 2020-03-25.w_.CFE_VX_2020.csv True, cleaning them out
Traceback (most recent call last):
File "z:\Downloads\Source-Master\vix_utils\examples\aq_sample_plot_today_term_structure.py", line 71, in
main()
File "z:\Downloads\Source-Master\vix_utils\examples\aq_sample_plot_today_term_structure.py", line 51, in main
df_day_of_interest =wide_with_continuous_futures.loc[[day_of_interest]]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "C:\Users\R Dev\anaconda3\envs\VixUtils\Lib\site-packages\pandas\core\indexing.py", line 1073, in getitem
return self._getitem_axis(maybe_callable, axis=axis)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\R Dev\anaconda3\envs\VixUtils\Lib\site-packages\pandas\core\indexing.py", line 1301, in _getitem_axis
return self._getitem_iterable(key, axis=axis)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\R Dev\anaconda3\envs\VixUtils\Lib\site-packages\pandas\core\indexing.py", line 1239, in _getitem_iterable
keyarr, indexer = self._get_listlike_indexer(key, axis)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\R Dev\anaconda3\envs\VixUtils\Lib\site-packages\pandas\core\indexing.py", line 1432, in _get_listlike_indexer
keyarr, indexer = ax._get_indexer_strict(key, axis_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\R Dev\anaconda3\envs\VixUtils\Lib\site-packages\pandas\core\indexes\base.py", line 6070, in _get_indexer_strict
File "C:\Users\R Dev\anaconda3\envs\VixUtils\Lib\site-packages\pandas\core\indexes\base.py", line 6070, in _get_indexer_strict
self._raise_if_missing(keyarr, indexer, axis_name)
File "C:\Users\R Dev\anaconda3\envs\VixUtils\Lib\site-packages\pandas\core\indexes\base.py", line 6070, in _get_indexer_strict
File "C:\Users\R Dev\anaconda3\envs\VixUtils\Lib\site-packages\pandas\core\indexes\base.py", line 6070, in _get_indexer_strict
self._raise_if_missing(keyarr, indexer, axis_name)
File "C:\Users\R Dev\anaconda3\envs\VixUtils\Lib\site-packages\pandas\core\indexes\base.py", line 6130, in _raise_if_missing
raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['2023-03-27'], dtype='object', name='Trade Date')] are in the [index]"

The text was updated successfully, but these errors were encountered:

dougransom · 2023-03-28T17:32:52Z

Ok, if you feel like looking into it please assign to yourself, otherwise I'll look at it eventually.

dougransom · 2023-07-05T18:08:28Z

Finally looking at this. Could be the files downloaded from https://www.cboe.com/us/futures/market_statistics/historical_data/ are not updated until a certain time of day, or it could be some problem with vix_utils.

dougransom · 2023-07-05T20:03:21Z

Reproduced this at 1:02PM pacific time July 5, 2023.
Problem persists at 3:17PM.

dougransom · 2023-07-05T22:59:31Z

As it appears to be a problem with the CBOE data source, it is not easy to fix.
Same problem with the other Vol indexes (VIX9D, VIX, VIX3M etc.).
It is theoretically possible to get the VX futures from
https://www.cboe.com/delayed_quotes/vix/future_quotes
or
https://www.cboe.com/tradable_products/vix/vix_futures/
or
https://www.cboe.com/tradable_products/vix/term_structure/

These pages work

https://www.cboe.com/delayed_quotes/vix9d
https://www.cboe.com/delayed_quotes/vix
https://www.cboe.com/delayed_quotes/shortvol

The data are created with javascript, try scraping with requests-html

dougransom · 2023-07-05T23:05:40Z

Maybe someone has some ideas about how to scrape the links above or another source of delayed quotes we can use?

dougransom · 2023-07-05T23:10:08Z

https://www.bing.com/videos/search?q=yahoo+finance+api&docid=603500247230070630&mid=A75A5497AB532B641AF5A75A5497AB532B641AF5&view=detail&FORM=VIRE

We should be able to get real time index values, but not futures:
https://pypi.org/project/yfinance/

vectorbt can load data from a variety of sources.

AQuant206 · 2023-07-06T08:34:04Z

I’ve been pulling daily Vix1m, etc from Yahoo finance after market closes as a workaround. If Yahoo has the data 30m after close, doesn’t CBOE?

…

On Thu, Jul 6, 2023 at 12:10 AM Doug Ransom ***@***.***> wrote: https://www.bing.com/videos/search?q=yahoo+finance+api&docid=603500247230070630&mid=A75A5497AB532B641AF5A75A5497AB532B641AF5&view=detail&FORM=VIRE https://pypi.org/project/yfinance/ — Reply to this email directly, view it on GitHub <#18 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3454GULNEYYRWWE2NQJAI3XOXX5VANCNFSM6AAAAAAWJZYFLE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

dougransom · 2023-07-06T16:15:57Z

Can you please post some code snippets, maybe in this thread? I don't know how to read the vix futures from yahoo or even access them through the web.

I don't know when CBOE updates their historical data online, not timely obviously.

AQuant206 · 2023-07-08T08:17:23Z

Can do next week (box w/ my code is 6 timezones away).

…

On Thu, Jul 6, 2023 at 5:16 PM Doug Ransom ***@***.***> wrote: Can you please post some code snippets, maybe in this thread? I don't know how to read the vix futures from yahoo or even access them through the web. — Reply to this email directly, view it on GitHub <#18 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3454GRT6VHNZ5KYA46E5CDXO3QERANCNFSM6AAAAAAWJZYFLE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

dougransom · 2024-06-15T20:32:17Z

@AQuant206 snippets?

dougransom · 2024-06-15T20:35:30Z

It is theoretically possible to download from CBOE web site at end of day:

`import json
import logging
import os
import subprocess
import sys
import time
import urllib
from logging import getLogger
from time import sleep
from dotenv import load_dotenv
from playwright.async_api import async_playwright
import asyncio as aio
from icecream import ic
from parsel import Selector

setup basic logging for our project which will display the time, log level & log message

logger = getLogger("webscapper.py")
logging.basicConfig(
stream=sys.stdout, # uncomment this line to redirect output to console
format="%(message)s",
level=logging.DEBUG,
)

capabilities = {
"browserName": "MicrosoftEdge", # Browsers allowed: Chrome, MicrosoftEdge, pw-chromium, pw-firefox and pw-webkit
"browserVersion": "latest",
"LT:Options": {
"platform": "Windows 10",
"build": "E Commerce Scrape Build",
"name": "Scrape Lambda Software Product",
"user": os.getenv("LT_USERNAME"),
"accessKey": os.getenv("LT_ACCESS_KEY"),
"network": False,
"video": True,
"console": True,
"tunnel": False, # Add tunnel configuration if testing locally hosted webpage
"tunnelName": "", # Optional
"geoLocation": "", # country code can be fetched from https://www.lambdatest.com/capabilities-generator/
},
}

vx_futures_expected_div_text="VX - Cboe Volatility Index (VIX) Futures"

async def main():
async with async_playwright() as pw:

    xpath_to_title=f"//div[text()=\"Futures\"]"
    ic(xpath_to_title)

    browser=await pw.chromium.launch()
    page=await browser.new_page()
    await page.goto("https://www.cboe.com/delayed_quotes/vix/")
    await page.wait_for_selector(xpath_to_title)
    sleep(3)

    page_html=await page.content()


    with open("sample_futures.pw.html","w") as f2:
        f2.write(page_html)

    
    sel = Selector(page_html)


    xpath_to_table=f"{xpath_to_title}/following-sibling::div/table"
    ic(xpath_to_table)

    div_title=sel.xpath(xpath_to_title)
    ic(div_title)
    table=sel.xpath(xpath_to_table)
    ic(table)

if name == "main":
aio.run(main())`

and to parse out

`import asyncio as aio
from icecream import ic
from parsel import Selector

def toString(a):
return f"{a}"

def main():
ic.configureOutput(argToStringFunction=toString)
with open("sample_futures.pw.html","r") as i:
page_html=i.read()

    sel = Selector(page_html)
    xpath_to_title=f"//div[text()=\"Futures\"]"
    ic(xpath_to_title)

    xpath_to_table=f"{xpath_to_title}/following-sibling::div/table"
    ic(xpath_to_table)

    title=sel.xpath(xpath_to_title)
    ic(title)

    table=sel.xpath(xpath_to_table)
    ic(table)

    cols=table.xpath("//th/text()").getall()
    ic(cols)      
    rows=table.xpath("//tbody/tr")
    ic(rows)
    for row in rows:
        cels=row.xpath("td//text()").getall()
        ic(cels)
        for col,cel in zip(cols,cels):
            #\n\tValue {cel}
            print(f"Column: \t{col}  \n\tText {cel}")

    all_cels=sel.xpath("//td")
    #ic(all_cels,)

if name == "main":
main()
`
I am not intending to add any such functionality to vix_utils though.

dougransom · 2024-08-06T20:36:47Z

CBOE has a list of approved data vendors. Probably none of them are free, even for delayed data. There should be no exchange fees though.

AQuant206 changed the title ~~P1: Can't do sample plot of term structure with today's date (after market close)~~ P1: Can't do sample plot of term structure with today's date (after close) where date of last day market open works. Mar 28, 2023

dougransom added the bug Something isn't working label Mar 28, 2023

dougransom self-assigned this Mar 28, 2023

dougransom added the help wanted Extra attention is needed label Jul 5, 2023

dougransom removed their assignment Jul 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

P1: Can't do sample plot of term structure with today's date (after close) where date of last day market open works. #18

P1: Can't do sample plot of term structure with today's date (after close) where date of last day market open works. #18

AQuant206 commented Mar 27, 2023 •

edited

Loading

dougransom commented Mar 28, 2023

dougransom commented Jul 5, 2023

dougransom commented Jul 5, 2023 •

edited

Loading

dougransom commented Jul 5, 2023 •

edited

Loading

dougransom commented Jul 5, 2023

dougransom commented Jul 5, 2023 •

edited

Loading

AQuant206 commented Jul 6, 2023 via email

dougransom commented Jul 6, 2023 •

edited

Loading

AQuant206 commented Jul 8, 2023 via email

dougransom commented Jun 15, 2024

dougransom commented Jun 15, 2024

dougransom commented Aug 6, 2024

P1: Can't do sample plot of term structure with today's date (after close) where date of last day market open works. #18

P1: Can't do sample plot of term structure with today's date (after close) where date of last day market open works. #18

Comments

AQuant206 commented Mar 27, 2023 • edited Loading

dougransom commented Mar 28, 2023

dougransom commented Jul 5, 2023

dougransom commented Jul 5, 2023 • edited Loading

dougransom commented Jul 5, 2023 • edited Loading

dougransom commented Jul 5, 2023

dougransom commented Jul 5, 2023 • edited Loading

AQuant206 commented Jul 6, 2023 via email

dougransom commented Jul 6, 2023 • edited Loading

AQuant206 commented Jul 8, 2023 via email

dougransom commented Jun 15, 2024

dougransom commented Jun 15, 2024

setup basic logging for our project which will display the time, log level & log message

dougransom commented Aug 6, 2024

AQuant206 commented Mar 27, 2023 •

edited

Loading

dougransom commented Jul 5, 2023 •

edited

Loading

dougransom commented Jul 5, 2023 •

edited

Loading

dougransom commented Jul 5, 2023 •

edited

Loading

dougransom commented Jul 6, 2023 •

edited

Loading