Too many fields problem

[hinder-asset]

When downloading more then 12 fields at once for one ticker with bdh, always the second, thrid and fourth field will not retrieve any data, no matter which fields they are (get NA values). The problem is linked with setting an appropriate start date: if the start date is close to today it works, if it lies in the very past it doesn't. Is there a download limitation or something?

Example code: (returns NA in second, third and fourth column):

ticker="MXUS INDEX"
fields=c("PX_LAST","PX_TO_CASH_FLOW","GROSS_AGGTE_DVD_YLD","PROF_MARGIN", "RETURN_ON_ASSET","BEST_PE_RATIO","INDX_GENERAL_EARN","INDX_GENERAL_EST_EARN", "IDX_EST_EARNINGS_NXT_YR GEN_EST_EARNINGS_FY3_AGGTE", "PCT_MEMB_ABOVE_MOV_AVG_200D", "NUM_DAILY_ADV_MINUS_DECL", COUNT_INDEX_MEMBERS","TOT_RETURN_INDEX_NET_DVDS","PE_RATIO PX_TO_BOOK_RATIO","PX_TO_EBITDA","PX_TO_SALES_RATIO","MOV_AVG_50D", "MOV_AVG_200D"
)
bdh(ticker, fields, start.date=as.Date("1970-01-01"),...)

[johnlaing]

The code you pasted doesn't quite run as is. But it's close enough that I was able to reproduce the bug.

library(Rblpapi)
ticker <- "MXUS INDEX"
fields <- c("PX_LAST", "PX_TO_CASH_FLOW", "GROSS_AGGTE_DVD_YLD", "PROF_MARGIN", "RETURN_ON_ASSET", "BEST_PE_RATIO",
            "INDX_GENERAL_EARN", "INDX_GENERAL_EST_EARN", "IDX_EST_EARNINGS_NXT_YR", "GEN_EST_EARNINGS_FY3_AGGTE",
            "PCT_MEMB_ABOVE_MOV_AVG_200D", "NUM_DAILY_ADV_MINUS_DECL", "COUNT_INDEX_MEMBERS", "TOT_RETURN_INDEX_NET_DVDS",
            "PE_RATIO", "PX_TO_BOOK_RATIO", "PX_TO_EBITDA", "PX_TO_SALES_RATIO", "MOV_AVG_50D", "MOV_AVG_200D")

## first pass, see where the NAs are
bb1 <- bdh(ticker, fields, start.date=as.Date("1970-01-01"))
which(sapply(bb1, function(x) all(is.na(x))))
#   PX_TO_CASH_FLOW GROSS_AGGTE_DVD_YLD
#                 3                   4

## move the offending fields to the end and try again
fields <- c(fields[-(2:3)], fields[2:3])
bb1 <- bdh(ticker, fields, start.date=as.Date("1970-01-01"))
which(sapply(bb1, function(x) all(is.na(x))))
#   PROF_MARGIN RETURN_ON_ASSET   BEST_PE_RATIO
#             3               4               5

So it is the case that the NAs are not due to the fields but instead seem related to the ordering. This is a strange bug, we'll look into it.

[johnlaing]

Update: this appears to be an issue with the response we are getting from Bloomberg.

I wrote 2 R scripts that differ only by a single day in the start date:

jlaing@lwvm-jlaing:~/tmp$ cat bb1.R
library(Rblpapi)
ticker <- "MXUS INDEX"
fields <- c("PX_LAST", "PX_TO_CASH_FLOW", "GROSS_AGGTE_DVD_YLD", "PROF_MARGIN", "RETURN_ON_ASSET", "BEST_PE_RATIO",
            "INDX_GENERAL_EARN", "INDX_GENERAL_EST_EARN", "IDX_EST_EARNINGS_NXT_YR", "GEN_EST_EARNINGS_FY3_AGGTE",
            "PCT_MEMB_ABOVE_MOV_AVG_200D", "NUM_DAILY_ADV_MINUS_DECL", "COUNT_INDEX_MEMBERS", "TOT_RETURN_INDEX_NET_DVDS",
            "PE_RATIO", "PX_TO_BOOK_RATIO", "PX_TO_EBITDA", "PX_TO_SALES_RATIO", "MOV_AVG_50D", "MOV_AVG_200D")

bb <- bdh(ticker, fields, start.date=as.Date("1975-03-19"), verbose=TRUE)
#which(sapply(bb, function(x) all(is.na(x))))
jlaing@lwvm-jlaing:~/tmp$ diff bb1.R bb2.R
8c8
< bb <- bdh(ticker, fields, start.date=as.Date("1975-03-19"), verbose=TRUE)
---
> bb <- bdh(ticker, fields, start.date=as.Date("1975-03-20"), verbose=TRUE)

I run them and compare results via verbose printing of the BB responses:

jlaing@lwvm-jlaing:~/tmp$ Rscript bb1.R > bb1.out 2>&1
jlaing@lwvm-jlaing:~/tmp$ Rscript bb2.R > bb2.out 2>&1
jlaing@lwvm-jlaing:~/tmp$ grep -c PX_TO_CASH_FLOW bb[12].out
bb1.out:0
bb2.out:5871

For one script (later date) we get thousands of data points for this field, while for the other we get none. Which explains why that field is blank in our response.

This will have to be a question for the helpdesk. They don't support R so I'll have to code it up in C++ for them.