Add test for tserver.wal.max.referenced #5250
Conversation
| int total = 0; | ||
| for (Entry<TServerInstance,List<UUID>> entry : wals.getAllMarkers().entrySet()) { | ||
| total += entry.getValue().size(); | ||
| } | ||
| return total; |
There was a problem hiding this comment.
Write ahead logs can have different states. For this test do not care about the one in the open state (every tserver will always have 2 open), but want to count the non open ones.
| int total = 0; | |
| for (Entry<TServerInstance,List<UUID>> entry : wals.getAllMarkers().entrySet()) { | |
| total += entry.getValue().size(); | |
| } | |
| return total; | |
| return wals.getAllState().values().stream().filter(walState -> walState != WalStateManager.WalState.OPEN).count(); |
|
|
||
| private void writeData(AccumuloClient client, String table, int startRow, int endRow) | ||
| throws Exception { | ||
| try (BatchWriter bw = client.createBatchWriter(table, new BatchWriterConfig())) { |
There was a problem hiding this comment.
Could modify this method to write 2XTSERV_WAL_MAX_SIZE data to a single tablet by generating rows that fall within a target tablet and using the Mutation.estimatedMemoryUsed() method.
while(totalWritten < 2*hdfsMinBlockSize){
Mutation m = ...
bw.addMutation(m);
totalWritten+=m.estimateMemoryUsed();
}Could use the same row, it does not matter for the walog it will keep adding it even if it is the same row.
|
|
||
| log.info("Created table {} with splits. Now writing data.", tableName); | ||
|
|
||
| // Write data multiple times until we see the WAL count exceed WAL_MAX_REFERENCED |
There was a problem hiding this comment.
The way this writes data it seems out of sync w/ the tablets. Would be good to push a lot of data to single tablet until a new non-open log has been produced and then move to the next tablet.
| cfg.setProperty(Property.TSERV_WAL_MAX_SIZE, hdfsMinBlockSize); | ||
| // Set the max number of WALs that can be referenced | ||
| cfg.setProperty(Property.TSERV_WAL_MAX_REFERENCED, Integer.toString(WAL_MAX_REFERENCED)); | ||
| cfg.setNumTservers(1); |
There was a problem hiding this comment.
May want to increase the amount of memory that tservers can use to hold new data in memory. If this is too low then tservers will minor compaction because memory is low and not because of walogs.
cfg.setProperty(Property.TSERV_MAXMEM,"256M");
Co-authored-by: Keith Turner <[email protected]>
| for (int i = 1; i <= 4; i++) { | ||
| splits.add(new Text(Integer.toString(i))); | ||
| } |
There was a problem hiding this comment.
Adding more splits would be good, could then keep writing to a different tablet.
| for (int i = 1; i <= 4; i++) { | |
| splits.add(new Text(Integer.toString(i))); | |
| } | |
| for (int i = 1; i <= 100; i++) { | |
| splits.add(new Text(String.format("%03d", i)); | |
| } |
| Wait.waitFor(() -> { | ||
|
|
||
| // Write data that should fill or partially fill the WAL | ||
| writeData(client, tableName); |
There was a problem hiding this comment.
In earlier comment I meant we could use the same row per tablet, but did not state that. If we add the 100 tablet then this code can keep advancing tablets it will write too. The writeData can just write everything to the same row passed in.
| writeData(client, tableName); | |
| var rowToWrite = String.format("%03d", iteration.get()); | |
| writeData(client, tableName, rowToWrite); |
Co-authored-by: Keith Turner <[email protected]>
Thanks for the suggestions. They should be incorporated as of 7170142. |
Fixes #5200
Adds a a test to make sure that
tserver.wal.max.referencedworks.