You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
cursor := reader.Select(readColumns...)
defer cursor.Close()
err = cursor.PrepareStripe(sr.StripeObject.StripeIndex)
for cursor.Next() {
row0 := cursor.Row()
row := make([]driver.Value, len(dataBlock.Columns.Columns))
for i := range row0 {
row[i] = driver.Value(row0[i])
}
for i, j := len(row0), 0; j < len(sr.StripeObject.ExtraValues); i, j = i+1, j+1 {
row[i] = driver.Value(sr.StripeObject.ExtraValues[j])
}
dataBlock.Payload = append(dataBlock.Payload, row
I try to build a dataBlock with 20000 rows and then insert this dataBlock to ClickHouse, However this process consume too much memory. So is there any way which can be used to decrease memory usage when read orc file to build dataBlock?
The text was updated successfully, but these errors were encountered:
I read an ORC file using code much like this:
I try to build a dataBlock with 20000 rows and then insert this dataBlock to ClickHouse, However this process consume too much memory. So is there any way which can be used to decrease memory usage when read orc file to build dataBlock?
The text was updated successfully, but these errors were encountered: