AVNI connector is slow for Arghyam Data

[siddhant3030]

I'm trying to test AVNI connector for last few days and I noticed that its taking lot of time for such a small dataset. We need to test this with incremental as well. My best guess is its because of the API's but lets look into this.

[fatchat]

2024-02-01 06:50:14 source > Marking stream encounters as RUNNING
2024-02-01 06:57:56 destination > INFO i.a.i.d.b.BufferedStreamConsumer(periodicBufferFlush):220 Periodic buffer flush started
2024-02-01 06:57:56 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAll):85 Flushing encounters: 1287 records (8 MB)
2024-02-01 06:57:57 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAll):91 Flushing completed for encounters
2024-02-01 06:57:57 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAll):85 Flushing subjects: 1204 records (8 MB)
2024-02-01 06:57:58 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAll):91 Flushing completed for subjects
2024-02-01 07:13:38 destination > INFO i.a.i.d.b.BufferedStreamConsumer(periodicBufferFlush):220 Periodic buffer flush started
2024-02-01 07:13:38 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAll):85 Flushing encounters: 2507 records (7 MB)
2024-02-01 07:13:38 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAll):91 Flushing completed for encounters
2024-02-01 07:28:51 destination > INFO i.a.i.d.b.BufferedStreamConsumer(periodicBufferFlush):220 Periodic buffer flush started
2024-02-01 07:28:51 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAll):85 Flushing encounters: 1889 records (5 MB)
2024-02-01 07:28:51 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAll):91 Flushing completed for encounters
2024-02-01 07:44:00 destination > INFO i.a.i.d.b.BufferedStreamConsumer(periodicBufferFlush):220 Periodic buffer flush started
2024-02-01 07:44:00 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAll):85 Flushing encounters: 2214 records (6 MB)
2024-02-01 07:44:00 destination > INFO o.a.s.c.s.h.AbstractSession(requestNewKeysExchange):2271 requestNewKeysExchange(ClientSessionImpl[<user>@/<ip-address>:22]) Initiating key re-exchange
2024-02-01 07:44:01 destination > WARN o.a.s.c.k.StaticServerKeyVerifier(handleAcceptance):59 Server at /<ip-address>:22 presented unverified EC key: SHA256:<....>/<....>/w
2024-02-01 07:44:01 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAll):91 Flushing completed for encounters
2024-02-01 07:50:54 source > Read 9188 records from encounters stream

[avirajsingh7]

Hello @siddhant3030, we can increase the page size here but I am not sure what is max value for that due to small number of records ( there is no mention of max page size on api docs) . Can you try hit and trail to find max page size in postman and change it to that.
It can be 10X faster if max page size is 1000