Question about preloading: "exception: set: Map instance is read-only!"

[linfan68]

Here is my project:

{
  "creation": { },
  "modules": {
    "*": "./*"
  },
  "preload": [ ]
}

// main.js
import { _foo } from './ClassFoo'
trace('Try call _foo.addNumber\n')
_foo.addNumber('key', 123)
trace('testing preloading DONE\n')

export class ClassFoo {
  _map = new Map()

  addNumber(key, value) {
    this._map.set(key, value)
  }
}

export const _foo = new ClassFoo()

It works fine, and the output:

Try call _foo.addNumber
testing preloading DONE

But if "ClassFoo" is added into the "preload" list:

"preload": [ "ClassFoo" ]

Then it failed with this output:

Try call _foo.addNumber
/workspaces/dk-hwjs/src/targets/linux/prototypes/test_preload/ClassFoo.js:5: exception: Map.prototype.set: Map instance is read-only!
### TypeError: Map.prototype.set: Map instance is read-only

I (kinda of) understand the "preloading" works as the compiler runs the code during compile time and try to take a snapshot of the result memory into ROM, then it can be used during runtime without taking RAM space. And as the document mentioned preload, the build-in Map can be stored (and read-only I assume) in ROM during preloading.

Some more context: I was trying to port a project running on ESP32-S3 (with 8MB SPIRAM) into ESP32-C6 (with 512K RAM). At first the program took more that 1MB RAM to run (without preloading), and after make most modules preloaded, the RAM usage get down to (unbelieveable) few KB (and saved my day :-D, until I found this Map issue...)

I think I need some help about how to design a "preloadable" system. I think I am jumping between "preload nothing, and use lots of RAM" and "preload everything, but everything are fixed.", and I need to find a way to land some where in the middle between them.

Some thoughts:

1. Is there a trick to make SOME objects writable after preloading?
2. Maybe I should create the _foo during runtime? Which will end up using lots of memory since I'm using that pattern a lot...
3. If one module cannot be preloaded, then I guess all modules depends on it cannot be preloaded.

[phoddie]

Glad to hear that you are enjoying preload. It is close to magic, so it is easy to overlook the details.

Everything you describe is exactly as expected. There's one key thing to understand. Anything that is preloaded is, of course, stored in immutable ROM (flash memory) and so cannot be modified. But, XS has an aliasing mechanism for certain preloaded objects which are not explicitly frozen. XS always freezes functions (and consequently classes, since they are "just" functions) and instances of most built-ins. But, it does not automatically freeze plain JavaScript objects. Of course, are placed into ROM too, but XS also creates an entry for the object in its alias table. When a script attempts to modify those objects (setting an existing property, adding a new property, deleting a property), XS automatically copies the object to RAM. It is a shallow copy, so any strings and objects referenced remain in ROM.

Therefore, if this code is executed during preload...

const x = {a: 1, b: 2};
const y = Object.freeze({c: 3});

...this code will still work after preload (e.g. on the device):

delete x.a;
x.b = 3;
x.c = "c";

And this will throw, as expected:

delete y.c;

This is, more or less, copy-on-write behavior. There is only a RAM cost when you modify it.

We can take advantage of the alias mechanism to create a preload-aware Map class that also does a copy-on-write. Here's a quick example of that along with some test code – put it all in main.js with main.js preloaded:

class MutableMap {
	#state = {};

	constructor(...args) {
		if (Object.isFrozen(MutableMap))
			return new Map(...args);					// post-lockdown/preload – use Map instance directly
		this.#state.map = new Map(...args);
	}
	#getMap(mutable) {
		let map = this.#state.map;
		if (mutable && Object.isFrozen(map))
			map = this.#state.map = new Map(map);		// clone to mutable
		return map;
	}
	set(key, value) {
		return this.#getMap(true).set(key, value);
	}
	get(key) {
		return this.#getMap(false).get(key);
	}
	has(key) {
		return this.#getMap(false).has(key);
	}
	delete(key) {
		return this.#getMap(true).delete(key);
	}
	clear() {
		return this.#getMap(true).clear();
	}
	[Symbol.iterator]() {
		return this.#getMap(false)[Symbol.iterator]();
	}
	//@@ other methods here as needed
}

const pre = new MutableMap;
pre.set(1, "one");
pre.set(2, "two");
pre.set("four", 4);

export default function() {
	let result = pre.get(2);
	pre.set(3, "three");
	pre.set(1, "ONE");
	pre.delete(2);

	const post = new MutableMap(pre);
	for (const item of post)
	  trace(item, "\n");
}

Note that the alias is on the value of the private instance variable #state. The value of the #state property cannot be set, but the contents of the object it references can be, because of the alias.

As an optimization, after preload, this function reverts to returning Map instances directly.

This is just a proof-of-concept example to show the concept. Depending on the specifics of your maps, you might take a different approach.

[linfan68]

@phoddie Thank you Peter!!!
Based on what you suggested, here is my final solution (and how it works in my project)

I created a "PreloadableMap" class, and use it as a drop-in replacement of the build-in Map.
To make things simpler, when a mutable map is needed, I'll just make a new copy of the frozen one.

export class PreloadableMap<K, V> {
  private __map: Map<K, V> = new Map()
  public constructor(args: [K, V][] = []) {
    this.__map = new Map(args)
  }

  private getMap(mutable: boolean = false) {
    if (mutable && Object.isFrozen(this.__map)) {
      this.__map = new Map(this.__map)
    }
    return this.__map
  }

  public get(key: K): V | undefined {
    return this.getMap().get(key)
  }

  public set(key: K, value: V): this {
    this.getMap(true).set(key, value)
    return this
  }

  public has(key: K): boolean {
    return this.getMap().has(key)
  }

  public delete(key: K): boolean {
    return this.getMap(true).delete(key)
  }

  public clear(): void {
    this.getMap(true).clear()
  }

  public keys(): IterableIterator<K> {
    return this.getMap().keys()
  }

  public values(): IterableIterator<V> {
    return this.getMap().values()
  }

  public entries(): IterableIterator<[K, V]> {
    return this.getMap().entries()
  }

  [Symbol.iterator](): IterableIterator<[K, V]> {
    return this.getMap()[Symbol.iterator]()
  }

  public get size(): number {
    return this.getMap().size
  }
}

export class ClassFoo {
  _map = new PreloadableMap()

  addNumber(key, value) {
    this._map.set(key, value)
  }
}

Surprisingly, this is the ONLY change I need to do to fixed the entire project for preloading.
You're right, it's magic....

[phoddie]

@linfan68 – glad the approach works for you. Thanks for catching my oversight on set, I forget about chaining. Your name, PrelodableMap is better than mine too. .)

Maybe TypeScript does something unexpected for private variables, but I don't think your example will work when set is or delete is called, because the private variable should be immutable. That's why I used one level of indirection on #state – to get the XS aliasing behavior.

Since this might be useful for others, I updated my version based on your improvements. That's here.

class PreloadableMap {
	#state = {};

	constructor(...args) {
		if (Object.isFrozen(PreloadableMap))
			return new Map(...args);					// post-lockdown/preload – use Map instance directly
		this.#state.map = new Map(...args);
	}
	#getMap(mutable = false) {
		let map = this.#state.map;
		if (mutable && Object.isFrozen(map))
			map = this.#state.map = new Map(map);		// clone to RAM
		return map;
	}
	set(key, value) {
		this.#getMap(true).set(key, value);
		return this;
	}
	get(key) {
		return this.#getMap().get(key);
	}
	has(key) {
		return this.#getMap().has(key);
	}
	delete(key) {
		return this.#getMap(true).delete(key);
	}
	clear() {
		return this.#getMap(true).clear();
	}
	keys() {
		return this.#getMap().keys();
	}
	values() {
		return this.#getMap().values();
	}
	entries() {
		return this.#getMap().entries();
	}
	[Symbol.iterator]() {
		return this.#getMap()[Symbol.iterator]();
	}
	get size() {
		return this.#getMap().size;
	}
}

Surprisingly, this is the ONLY change I need to do to fixed the entire project for preloading.
You're right, it's magic....

...or at least close.