Storage position for entity 1000000000000029026 exceeds the technical int value limit of 2147483647

[Bios-Marcel]

I am a little bit confused by this. This just happened in a transaction that would have pushed the database size to roughly 3 Gigabytes. I've already had databases that were over 100GiB.

Either this is a bug or a very confusing exception message.

29.10.2024 - 12:33:21.089 [Eclipse Serializer-StorageChannel-0] ERROR org.eclipse.store.storage.types.StorageChannel - StorageChannel#0 encountered disrupting exception
org.eclipse.store.storage.exceptions.StorageException: Storage position for entity 1000000000000029026 exceeds the technical int value limit of 2147483647. This happens when a single store grows too big. This limitation will be removed in a future version.
	at org.eclipse.store.storage.types.StorageEntityCache$Default.validateStoragePosition(StorageEntityCache.java:1047) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageEntityCache$Default.internalPutEntities(StorageEntityCache.java:1074) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageEntityCache$Default.postStorePutEntities(StorageEntityCache.java:1138) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageChannel$Default.postStoreUpdateEntityCache(StorageChannel.java:600) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageRequestTaskStoreEntities$Default.postCompletionSuccess(StorageRequestTaskStoreEntities.java:137) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageRequestTaskStoreEntities$Default.postCompletionSuccess(StorageRequestTaskStoreEntities.java:40) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageChannelSynchronizingTask$AbstractCompletingTask.complete(StorageChannelSynchronizingTask.java:131) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageChannelTask$Abstract.processBy(StorageChannelTask.java:260) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageChannel$Default.work(StorageChannel.java:453) ~[storage-2.0.0.jar:?]
	at org.eclipse.store.storage.types.StorageChannel$Default.run(StorageChannel.java:536) ~[storage-2.0.0.jar:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

The problem is, that I can't even start the server now. The data is effectively unsalvagable as far as I am concerned. (Asides from backups).

I am not really sure what exactly I can do here OR how I couild have avoided this in the first place.

[Bios-Marcel]

Given microstream-one/microstream#611 is still open, I assume this is the same bug?

[hg-ms]

Did some initial investigations, the error was easy to reproduce. Basically, I just need to do a single store that produces more than 2GB data in one channel, e.g. by storing a list of byte arrays.

I found two different behaviors:

1. If the store consists of many “small” objects I get a StorageException “Storage position for entity 1000000000004052884 exceeds the technical int value limit of 2147483647.”
   after the data was written but before the store call returns.
2. If the store consists of only a few large objects no exception was thrown, the store call returns without any indication of an error.

In both cases a StorageExceptionIoReading “Storage file size exceeds Java technical IO limitations” is thrown at the next storage startup, the storage can’t be loaded any more.

I’m not sure if this is the same as the linked Microstream issue microstream-one/microstream#611 .
Issue #320 is caused by the problematic store operation described in #319

This is a serious bug that needs to be fixed soon.