Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CreateAttachment::to_base64 causes slowdowns with big files #3091

Open
Vrtgs opened this issue Jan 10, 2025 · 0 comments
Open

CreateAttachment::to_base64 causes slowdowns with big files #3091

Vrtgs opened this issue Jan 10, 2025 · 0 comments

Comments

@Vrtgs
Copy link

Vrtgs commented Jan 10, 2025

I'm questionably trying to make discord store some big files, by splitting them and sending in chunks, and it seems there are slowdowns esspecially on debug builds, because sernity first turns the bytes into base64, and then inserts the string, which causes a realloc and a memcopy of some pretty large sections of memory (and this already happens a lot, because I have to split files, and then pass them to create attachment, which is another copy...... yeah this runs really slowly with big files on my old laptop)

and yes, I have profiled the code, this just seemed like an odd hotspot and an easy fix

anyways this function just seems easier and less of a breaking change than turning CreateAttachment

    pub fn to_base64(&self) -> String {
        let mut encoded = {
            use base64::Engine;
            base64::prelude::BASE64_STANDARD.encode(&self.data)
        };
        encoded.insert_str(0, "data:image/png;base64,");
        encoded
    }

I suggest an adimtedly much less simple implementation more like:

    pub fn to_base64(&self) -> String {
        use base64::Engine;

        const PREFIX: &str = "data:image/png;base64,";
        
        let engine = base64::prelude::BASE64_STANDARD;
        let encoded_len = base64::encoded_len(
            self.data.len(), 
            engine.config().encode_padding()
        )
            .and_then(|len| len.checked_add(PREFIX.len()))
            // we can't even allocate this much
            .expect("capacity overflow");
        
        let mut encoded = String::with_capacity(encoded_len);
        encoded.push_str(PREFIX);
        engine.encode_string(&self.data, &mut encoded);
        encoded
    }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant