Missing Associations when associations exceeds 100

[reinthal]

dlt version

1.4.0

Source name

hubspot

Describe the problem

Background & Problem

When loading deals from hubspot we noticed that associations were missing. When the table companies__deals was loaded and the deals exceeded 100 deals for a company, then data would be missing.

Expected behavior

All deals would be present for a given company

Steps to reproduce

Load data from a hubspot repo with

dlt init hubspot duckdb

followed by

pyhon hubspot_pipeline.py

On a hubspot instance for which one company has more than 100 deals. Then from your duckdb

  select 
`_dlt_parent_id`, count(*) 
from dev_bronze.hubspot_dataset.companies__deals
 group by `_dlt_parent_id` order by count(*) desc

would yield a result like his

_dlt_parent_id	count(*)
0jPQhdDNPvSIjA	100
k6UqDpeeEc3FGQ	100
mz8PfMM1XUPcLg	100
XCTmACDNKqSaoQ	100
A+2i/EXFpt5+oQ	100
zgjZJIoj9f+5/g	100
wOg/f8/0G7Xgdw	100

How you are using the source?

I run this source in production.

Operating system

Linux

Runtime environment

Other

Python version

3.12

dlt destination

duckdb

Additional information

No response