How to programmatically edit plugin state (to convert existing patches to Dexed) ?

[probonopd]

Dexed saves plugin state in this format:

00000000   56 43 32 21  22 18 00 00  3C 3F 78 6D  6C 20 76 65  72 73 69 6F  6E 3D 22 31  VC2!"...<?xml version="1
00000018   2E 30 22 20  65 6E 63 6F  64 69 6E 67  3D 22 55 54  46 2D 38 22  3F 3E 20 3C  .0" encoding="UTF-8"?> <
00000030   64 65 78 65  64 53 74 61  74 65 20 63  75 74 6F 66  66 3D 22 31  2E 30 22 20  dexedState cutoff="1.0"
00000048   72 65 73 6F  3D 22 30 2E  30 22 20 67  61 69 6E 3D  22 31 2E 30  22 20 63 75  reso="0.0" gain="1.0" cu
00000060   72 72 65 6E  74 50 72 6F  67 72 61 6D  3D 22 30 22  20 6D 6F 6E  6F 4D 6F 64  rrentProgram="0" monoMod
00000078   65 3D 22 30  22 20 65 6E  67 69 6E 65  54 79 70 65  3D 22 30 22  20 6D 61 73  e="0" engineType="0" mas
00000090   74 65 72 54  75 6E 65 3D  22 30 22 20  6F 70 53 77  69 74 63 68  3D 22 31 31  terTune="0" opSwitch="11
000000A8   31 31 31 31  22 20 74 72  61 6E 73 70  6F 73 65 31  32 41 73 53  63 61 6C 65  1111" transpose12AsScale
000000C0   3D 22 31 22  20 6D 70 65  45 6E 61 62  6C 65 64 3D  22 31 22 20  6D 70 65 50  ="1" mpeEnabled="1" mpeP
000000D8   69 74 63 68  42 65 6E 64  52 61 6E 67  65 3D 22 32  34 22 20 77  68 65 65 6C  itchBendRange="24" wheel
000000F0   4D 6F 64 3D  22 30 20 30  20 30 20 30  22 20 66 6F  6F 74 4D 6F  64 3D 22 30  Mod="0 0 0 0" footMod="0
00000108   20 30 20 30  20 30 22 20  62 72 65 61  74 68 4D 6F  64 3D 22 30  20 30 20 30   0 0 0" breathMod="0 0 0
00000120   20 30 22 20  61 66 74 65  72 74 6F 75  63 68 4D 6F  64 3D 22 30  20 30 20 30   0" aftertouchMod="0 0 0
00000138   20 30 22 20  61 63 74 69  76 65 46 69  6C 65 43 61  72 74 72 69  64 67 65 3D   0" activeFileCartridge=
00000150   22 2F 68 6F  6D 65 2F 75  73 65 72 2F  2E 6C 6F 63  61 6C 2F 73  68 61 72 65  "/home/user/.local/share
00000168   2F 44 69 67  69 74 61 6C  53 75 62 75  72 62 61 6E  2F 44 65 78  65 64 2F 43  /DigitalSuburban/Dexed/C
00000180   61 72 74 72  69 64 67 65  73 2F 53 79  6E 70 72 65  7A 46 4D 2F  53 79 6E 70  artridges/SynprezFM/Synp
00000198   72 65 7A 46  4D 5F 30 31  2E 73 79 78  22 3E 3C 64  65 78 65 64  42 6C 6F 62  rezFM_01.syx"><dexedBlob
000001B0   20 62 61 73  65 36 34 3A  73 79 73 65  78 3D 22 34  31 30 34 2E  76 4F 44 2E   base64:sysex="4104.vOD.
000001C8   49 2E 42 2E  50 6B 7A 43  4A 4C 31 44  2E 2E 50 4D  2E 2E 2E 43  79 51 41 55  I.B.PkzCJL1D..PM...CyQAU
000001E0   2E 2E 66 58  54 58 66 2E  61 6F 45 2E  2E 6A 42 2E  61 4C 66 58  44 62 6B 2E  ..fXTXf.aoE..jB.aLfXDbk.
000001F8   2E 6E 45 50  62 7A 78 58  67 41 2E 2E  74 2E 2E 2E  2E 4C 47 42  65 49 2E 2E  .nEPbzxXgA..t....LGBeI..
(...)

Is this format documented somewhere?

Specifically, the contents of <dexedBlob base64:sysex="..." don't realy look like base64 to me. What is the secret to decoding them?

[probonopd]

The respective code seems to be https://github.com/asb2m10/dexed/blob/b6cf2e9190e216387ad96f977284ebbef765b447/Source/PluginData.cpp#L361C1-L376C43

    NamedValueSet blobSet;
    blobSet.set("sysex", var((void *) currentCart.getVoiceSysex(), 4104));
    blobSet.set("program", var((void *) &data, 161));
    
    blobSet.copyToXmlAttributes(*dexedBlob);
    
    XmlElement *midiCC = dexedState.createNewChildElement("midiCC");
    HashMap<int, Ctrl *>::Iterator i(mappedMidiCC);
    while(i.next()) {
        XmlElement *ccMapping = midiCC->createNewChildElement("mapping");
        ccMapping->setAttribute("cc", i.getKey());
        Ctrl *ctrl = i.getValue();
        ccMapping->setAttribute("target", ctrl->label);
    }
    
    copyXmlToBinary(dexedState, destData);

So it looks like a standard 32-voice bulk dump sys-ex format (4104 bytes long) is copied into <dexedBlob base64:sysex="..."> but in some form of encoding which I am having trouble interpreting.

It is probably no coincidence that the content of <dexedBlob base64:sysex="4104..."> starts with the string 4104.

https://docs.juce.com/master/classMemoryBlock.html vaguely refers to

JUCE-specific text encoding that represents the binary contents of this block

I think this is the code that does the encoding/decoding:

https://github.com/juce-framework/JUCE/blob/master/modules/juce_core/memory/juce_MemoryBlock.cpp#L353C1-L411C2

[probonopd]

https://forum.juce.com/t/base64-encoding-algorithm-in-memoryblock-is-wrong/2613

@julianstorer wrote:

Well no, it’s not a standard algorithm, but it is a base-64 encoding, so it’s not a misnomer either! I originally wrote it just as a quick way to store things internally, rather than for working with standard formats, but I agree it’d be handy to have a proper one as well.

Now, if only there was a working Python implementation of this!

[probonopd]

Here is a Python version of the en/decoder that seems to work for me:

#!/usr/bin/env python3

# This script implements the non-standard JUCE implementation of Base64 encoding/decoding
# https://github.com/juce-framework/JUCE/blob/master/modules/juce_core/memory/juce_MemoryBlock.cpp#L353C1-L411C2

encodingTable = ".ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+"

def _setBitRange(size, data, bitRangeStart, numBits, bitsToSet):
    bite = bitRangeStart >> 3
    offsetInByte = int(bitRangeStart & 7)
    mask = ~(((0xffffffff) << (32 - numBits)) >> (32 - numBits))
    while numBits > 0 and bite < size:
        bitsThisTime = min(numBits, 8 - offsetInByte)
        tempMask = (mask << offsetInByte) | ~(((0xffffffff) >> offsetInByte) << offsetInByte)
        tempBits = bitsToSet << offsetInByte
        data[bite] = (data[bite] & tempMask) | tempBits
        bite += 1
        numBits -= bitsThisTime
        bitsToSet >>= bitsThisTime
        mask >>= bitsThisTime
        offsetInByte = 0

def fromJuceBase64Encoding(s):
    """
    Decodes a string encoded in JUCE Base64 format.

    Args:
        s (str): The JUCE Base64 encoded string.

    Returns:
        bytes: The decoded data.

    """
    size = int(s[:s.index(".")])
    b64str = s[s.index(".") + 1:]
    pos = 0
    data = [0] * size
    for i in range(len(b64str)):
        c = b64str[i]
        for j in range(64):
            if encodingTable[j] == c:
                _setBitRange(size, data, pos, 6, j)
                pos += 6
                break
    return bytes(i & 0xff for i in data)

def _getBitRange(size, data, bitRangeStart, numBits):
    res = 0
    byte = bitRangeStart >> 3
    offsetInByte = bitRangeStart & 7
    bitsSoFar = 0
    while numBits > 0 and byte < size:
        bitsThisTime = min(numBits, 8 - offsetInByte)
        mask = (0xff >> (8 - bitsThisTime)) << offsetInByte
        res |= (((data[byte] & mask) >> offsetInByte) << bitsSoFar)
        bitsSoFar += bitsThisTime
        numBits -= bitsThisTime
        byte += 1
        offsetInByte = 0
    return res

def toJuceBase64Encoding(bytes):
    """
    Converts a byte array to JUCE Base64 encoding.

    Args:
        bytes (bytes): The byte array to be encoded.

    Returns:
        str: The JUCE Base64 encoded string.
    """
    if not bytes:
        return None
    numChars = ((len(bytes) << 3) + 5) // 6
    s = str(len(bytes)) + '.'
    for i in range(numChars):
        s += encodingTable[_getBitRange(len(bytes), bytes, i * 6, 6)]
    return s