Allow passing exact file path to loam.initialize()

[attilaolah]

Because the file might be named differently, or because the file might be on a remote origin, in which case Chrome won't allow creating a web worker using a script that is on the remote origin.

The latter can be worked around by fetch()ing the file and creating an object URL, like so:

const loamWorkerJS = await fetch('https://unpkg.com/[email protected]/lib/loam-worker.js')
  .then(res => res.blob())
  .then(blob => URL.createObjectURL(blob)) + "#";

Not the trailing #, which is really a hack so that when loam.initialize() appends the filename (loam-worker.js), it would end up being ignored. But that just so happens to work with object URLs.

I'd suggest adding a second parameter for backwards compatibility, for the filename, i.e. something like:

function initialize(pathPrefix, fileName) {
  if (fileName === undefined) {
    fileName = 'loam-worker.js';
  }

  …;
}

I'll send a PR later on, just wanted to note this down first.

[ddohler]

Hey @attilaolah -- thanks for opening this! I think I'd like to try to understand a bit more about your use-case. Are you trying to use Loam via unpkg.com or was that an example to demonstrate the problem?

[attilaolah]

That's exactly what I'm doing! Here is an example code pen.

What I'm doing is the following:

1. Load loam.js from unpkg.com via an anonymous cross-origin script.
2. Call loam.initialize(), but passing in unpkg.com as the prefix would try to create a cross-origin worker which will fail.
3. So instead, I do an anonymous fetch() to get the contents of loam-worker.js to create a blob.
4. Create a blob URL, and append # so that the suffix that loab.js adds would be ignored. This is the hacky part.

But there is more tricks to do: loam-worker.js will try to call importScript('gdal.js'), which will also fail since it will try to resolve gdal.js using a relative URL. So instead I do a search & replace in the contents of loam-worker.js, to replace the importScript('gdal.js') with importScript('https://unpkg.com/…/gdal.js'), which will also load gdal.js from unpkg.com.

But then, gdal.js will try to load the gdal.data and gdal.wasm files. To provide the right file paths for this, I implement my own module.locateFile() function, and inject that too into loam-worker.js. Then gdal-js loads successfully from unpkg.com.

All this monkey patching works, but is rather ugly. Instead, I would like to do something like this:

• Load loam.js like usual.
• Tell Loam how to locate each file: loam-worker.js, gdal.js, gdal.data, gdal.wasm.
• Loam should try and create a new web worker. If it fails due to cross-origin, it could even try to anonymous-fetch loam-worker.js and create a worker using the blob URL. If this is too much though, that's fine: I can create the blob URL myself and set it as the source location for loam-worker.js.

Instead of loading gdal.js directly, loam-worker.js should get its location from the config that we passed to loam.js. Maybe via message exchange, i.e. send the script locations to the promise as a message. Then, loam-worker.js should implement Module.locateFile() so that when it loads gdal.js, it will get the right URLs to the .data and .wasm files.

Overall, that would be a much nicer user experience, although it could be an overkill — you could just say that it is a requirement that all files, loam and gdal-js, should be hosted on the same origin as the web page executing the script.

[0xTimepunk]

That's exactly what I'm doing! Here is an example code pen.

What I'm doing is the following:

1. Load loam.js from unpkg.com via an anonymous cross-origin script.
2. Call loam.initialize(), but passing in unpkg.com as the prefix would try to create a cross-origin worker which will fail.
3. So instead, I do an anonymous fetch() to get the contents of loam-worker.js to create a blob.
4. Create a blob URL, and append # so that the suffix that loab.js adds would be ignored. This is the hacky part.

But there is more tricks to do: loam-worker.js will try to call importScript('gdal.js'), which will also fail since it will try to resolve gdal.js using a relative URL. So instead I do a search & replace in the contents of loam-worker.js, to replace the importScript('gdal.js') with importScript('https://unpkg.com/…/gdal.js'), which will also load gdal.js from unpkg.com.

But then, gdal.js will try to load the gdal.data and gdal.wasm files. To provide the right file paths for this, I implement my own module.locateFile() function, and inject that too into loam-worker.js. Then gdal-js loads successfully from unpkg.com.

All this monkey patching works, but is rather ugly. Instead, I would like to do something like this:

• Load loam.js like usual.
• Tell Loam how to locate each file: loam-worker.js, gdal.js, gdal.data, gdal.wasm.
• Loam should try and create a new web worker. If it fails due to cross-origin, it could even try to anonymous-fetch loam-worker.js and create a worker using the blob URL. If this is too much though, that's fine: I can create the blob URL myself and set it as the source location for loam-worker.js.

Instead of loading gdal.js directly, loam-worker.js should get its location from the config that we passed to loam.js. Maybe via message exchange, i.e. send the script locations to the promise as a message. Then, loam-worker.js should implement Module.locateFile() so that when it loads gdal.js, it will get the right URLs to the .data and .wasm files.

Overall, that would be a much nicer user experience, although it could be an overkill — you could just say that it is a requirement that all files, loam and gdal-js, should be hosted on the same origin as the web page executing the script.

Hey @attilaolah

Thanks for this codepen, I am having an issue where gdal.js, gdal.data, gdal.wasm aren't being loaded properly so I tried to do it your way by looking at your code.

This is what it resulted in for loading loam for use in a Create React App application:

• Added the following links in public/index.html:

...
    <link rel="prefetch" href="https://unpkg.com/[email protected]/gdal.js" />
    <link rel="prefetch" href="https://unpkg.com/[email protected]/gdal.data" />
    <link rel="prefetch" href="https://unpkg.com/[email protected]/gdal.wasm" />
    <link rel="prefetch" href="https://unpkg.com/[email protected]/lib/loam-worker.js" />
...

• loadLoam.js:

const PREFETCH = Array.from(document.querySelectorAll('link[rel=prefetch]')).map(
  (link) => link.href,
);

const prefetched = (file) => PREFETCH.filter((path) => path.match(new RegExp(`/${file}$`))).shift();

const prefetchedJS = `
      "use strict";
      const PREFETCH = ${JSON.stringify(PREFETCH)};
      Module.locateFile = ${prefetched.toString()};
    `;

export default function InitLoam() {
  return new Promise((resolve) => {
    console.log(prefetched('loam-worker.js'));
    fetch(prefetched('loam-worker.js'))
      .then((res) => res.blob())
      .then((blob) => blob.text())
      .then((text) => {
        loam
          .initialize(
            `${URL.createObjectURL(
              new Blob([
                // Prevent the worker from trying to load 'gdal.js'.
                // Instead, tell it to load from a prefetched blob resourse.
                // Also inject Module.locateFile that locates files based on prefetched links.
                text.replace(
                  /\bimportScripts\(["']gdal.js["']\)/g,
                  `importScripts("${
                    // Load a script that will populate Module.locateFile().
                    URL.createObjectURL(new Blob([prefetchedJS]))
                  }", "${
                    // Now load the actual gdal.js, but from a blob.
                    // URL.createObjectURL(gdal)
                    prefetched('gdal.js')
                  }")`,
                ),
              ]),
            )}#`,
          )
          .then(() => resolve(loam));
      });
  });
}

However loam appears as not defined by the compiler, so I'm wondering where are you importing/requiring it?

@ddohler I replied in this issue as well as I thought using unpkg could solve my problem.

(here is my original issue: #62)

Thanks

EDIT
@ddohler

I managed to fix my error and initialize loam by just importing loam at the top of loadLoam.js

import loam from 'loam';

All of this seems to work now!

[ddohler]

Hey @attilaolah -- Sorry for the delay here; I still had trouble understanding the use-case the first time I read your comment, but now that I'm looking at it again, it's pretty clear.

I think the tricky part will be getting the absolute path into the worker--the only way that the main Loam library can communicate with the worker is via message passing, so in order to inject an absolute path for gdal.js, gdal.data, etc., the only two options I can see are:

1. Change the worker initialization so that it waits for a second "configuration" message before requesting the gdal assets, and then use Module.locateFile() as you did to set the correct path based on the configuration message. This would probably work but it'd add some complexity to the worker code setup process because it'd have to be split into two stages. It also seems a bit dangerous because in the failure case the worker could potentially just sit, waiting for a configuration message forever, which could lead to hard-to-diagnose bugs. So I'm not super enthusiastic about this option, for that reason.
2. Do something similar to what you did and either request the worker code and alter it, or construct it dynamically from the main library. Despite how hacky this feels, I think it might end up being the right solution -- it would avoid the issue with the worker "hanging" from the previous option, and I bet the dynamic part could be split out into a small "bootstrap" worker that just contains enough logic to pull in the other files, while the remainder of the logic remains the same. So I think this is probably the direction I would go in.

So I think the solution you came up with, hacky as it feels, is probably pretty close to how it would look with native Loam support, though obviously you wouldn't have had to implement it yourself if Loam already supported this 🙂.

The other wrinkle I'm seeing here is that Loam's Web Worker handling is hand-written because there were no libraries at the time I started the project to make that part easier. Now there are things like ComLink (#49 ), and webpack version 5 has also added improved Web Worker support. So I'm hesitant to add a lot of complexity to the web worker handling before taking on #49 because it seems like it could be wasted effort if ComLink has its own way of handling the problem. But this is useful to know going in to #49 because I don't want to switch if using ComLink would make this problem harder to solve in the future.

Anyway, I think my next focus now that 1.0.0 is released is going to be on developer quality-of-life issues, so I'll prioritize this and #49.

[attilaolah]

Thanks! Splitting the worker in two, one that would just bootstrap the other, sounds like a reasonable way forward. Also, focusing on #49 first sounds good to me.

[ddohler]

@attilaolah I've implemented this and I'll release a new version with the changes soon (hopefully within the next two weeks)! Thanks for your help and input!

[ddohler]

This is implemented as of version 1.1.0. Sorry, forgot to close this when that was released!

[attilaolah]

Oh coolies! Thank you!