SDL_IO* Buffering Mode?

[MikuAuahDark]

This is more like of a question rather than a bug report.

Since I don't think this is specified and I can't find anything about this in the documentation, what is the default buffering mode for SDL_IOStreams? Does all writes are committed directly or is there an underlying buffering mode of it? If so, how much the buffer size is?

[slouken]

There's no default, it depends on what's backing the IOStream.

[slime73]

Would it make sense for users to have a way to set the buffer size for IOStreams created from files (or in general when the interface supports that)? We're porting some code to use IOStream from a file, and it's missing that ability compared to our old code.

C#'s FileStream has that ability as part of the stream's constructor, for example. It also explicitly documents the default buffer size for file streams.

[slouken]

What's the use case?

[slime73]

The code we're porting is for a general purpose file IO abstraction (e.g. it also uses PhysFS including PHYSFS_setBuffer in a different backend for the same abstraction), so our users have more use cases than I do probably.

Our API is primarily for a higher level language where people don't have a simple way to make their own byte buffer, so they have to rely on APIs for it if they'd like to tune file read or write chunk sizes to fit their needs, for example. We could add our own chunk buffer implementation on top of IOStreams, but if the IOStream also does this under the hood it would sort of conflict. Another thing would be automatic file flushing when a newline is detected, e.g. for logging. If we don't know how big the file's write buffer is or if there is one at all, then it's hard to make it work consistently.

Maybe we can use SDL_PROP_IOSTREAM_WINDOWS_HANDLE_POINTER and friends to try and configure the underlying platform file's buffer, as a workaround? I'm unsure how safe or reliable that would be though.

[slouken]

I guess my question is, what would the application do differently if the underlying filesystem buffering was 1K or 4K? Why would it set one or the other?

@icculus, do you have thoughts here?

[slime73]

I'd guess the difference in read/write performance between 1K and 4K won't matter too much most of the time. But the difference between no buffering versus 4K could be very dramatic especially when functions like SDL_WriteU32LE(stream, value) are repeatedly called. And 1K versus 16K can be significant depending on the type of content being read/written.

Overall as a user it just seems hard to understand behaviour and performance with file IOStreams right now, since buffering is unspecified and not settable. Knowing also helps inform when to call FlushIO.

[slouken]

Since the IO stream is just an interface and most of the implementations do no buffering, you can consider them to be unbuffered at the API level, however at the OS level there is always some level of buffering due to disk cache and so forth. In general you should not call FlushIO(), the only time you would do that is when closing a file and being concerned about data loss, e.g. on consoles.

I'm happy to be shown wrong here, but I think this is an overthinking of the interface. Just write as needed and close when done. If you need buffering for some reason (and I think you don't) then you should buffer above this API so you get consistent behavior across platforms and implementations.

[slime73]

From a user perspective, IOStream is the generic interface (where I wouldn't expect buffer APIs) and SDL_IOFromFile() is for a concrete implementation with specific considerations like possible buffering. I think I just expected IOFromFile to be at about the same level as the equivalent C / C# file stream APIs, but it's not quite, so reimplementing those parts makes sense for us. Thanks!

[icculus]

@icculus, do you have thoughts here?

We'd have to build a generic buffering layer into IOStream, which would be superfluous for SDL_IOFromFile and unnecessary for SDL_IOFrom*Mem. But potentially useful for external implementations that wouldn't have to build their own!

But the specific concern, file buffering...it's not useful here. At this moment I'd say people should mess with the low-level handles they can get through the SDL_Properties...sort of a pain, but probably the right move for something like LÖVE's codebase.