Connection with Google Colab

[esgomezm]

Hi there!

In CellTracksColab (a tracking analysis tool), we often use Google Colab due to its ease of use for non-expert users. It would be nice to have the chance to connect it also with inTRACKtive, however this is not straightforward as data in Google Drive is private and the localhost opened in Google Colab as well.

I found a workaround to open the localhost with ngrok (a popular API gateway) but it feels to me that the link does not work with InTRACKTive. Here is some code in case you want to give it a try (you'd need to log-in in ngrok though):

# Install pyngrok
!pip install pyngrok

# Import and configure ngrok
from pyngrok import ngrok

# Add your authentication token obtained after signing in
!ngrok config add-authtoken TOKEN-KEY

# Configure the tunnel with correct parameters
try:
    # Kill any existing tunnels
    ngrok.kill()

    # Create new tunnel with correct configuration
    public_url = ngrok.connect(
        addr="127.0.0.1:8000",
        proto="http",
        bind_tls=True    # This ensures HTTPS
    )
    print(f"Ngrok tunnel established: {public_url}")

except Exception as e:
    print(f"Error: {e}")

# Display active tunnels
tunnels = ngrok.get_tunnels()
print("\nActive tunnels:", tunnels)

This will provide an output similar to the following (links broken as I closed the session in colab):

Authtoken saved to configuration file: /root/.config/ngrok/ngrok.yml
Ngrok tunnel established: NgrokTunnel: "https://ffee-34-46-46-10.ngrok-free.app/" -> "http://127.0.0.1:8000/"
Active tunnels: [<NgrokTunnel: "https://ffee-34-46-46-10.ngrok-free.app/" -> "http://127.0.0.1:8000/">]

Then one can launch inTRACKtive:

!python serve_directory_http.py data_bundle.zarr

> INFO:root:Serving data_bundle.zarr at http://127.0.0.1:8000/

and the localhost http://127.0.0.1:8000/ can be accessed with https://ffee-34-46-46-10.ngrok-free.app/ but it does not work in inTRACKtive. This is how the content looks like:

@guijacquemet

[aganders3]

Thank you - cool workflow! I'm traveling but will try to look into this next week if others don't get to it before me.

[TeunHuijben]

Thanks @esgomezm!

I was able to recreate your results, by first using serve_directory_http.py to serve a local directory, and then using ngrok for setting up a tunnel. I have the same result as you, I can see the folder structure in my browser via the ngrok tunnel, but it doesn't load into inTRACKtive.

I suspect that the ngrok tunnel somehow breaks the CORS access, and I wasn't able to quickly solve this. @kyleawayan, do you have any experience with this?

Zooming out, do you think that the localhost+ngrok would be your preferred solution for allowing collaborators to view your data? In my understanding, both the localhost and the ngrok tunnel need to be stay alive during this procedure, which will probably not work for sharing data for a longer period of time. It might be worth looking into options of putting the data in an online repository (like the ftp.ebi server you mentioned in a previous conversation). Would you be able to upload an example csv and zarr_bundle.zarr file there for us to try it out? (although HTTP servers are preferred, since ftp server usually have problems with CORS)

One thing to keep in mind is that, when you use the our serve_directory_http.py function, the correct syntax would be:

python serve_directory_http.py folder

Where folder is the folder that contains the Zarr bundle, the the path of the Zarr bundle itself

[kyleawayan]

Yep, it may be a CORS issue. Do we have any console logs to confirm it?

[esgomezm]

Hi @TeunHuijben

Zooming out, do you think that the localhost+ngrok would be your preferred solution for allowing collaborators to view your data?

Sorry, this would only be to connect with Google Colab, one of the standard tools for courses with life scientists and many use it as a tool by default. To share data, I'm working on the Bioarchive to provide a link. This will take me some time, sorry

[aganders3]

I think (for ngrok) this is not exactly a CORS error, but that on first-visit ngrok shows a warning page. Thus when trying to open the zarr we get:

You are about to visit 40a2-104-196-194-212.ngrok-free.app, served by 104.196.194.212. This website is served for free through ngrok.com. You should only visit this website if you trust whoever sent the link to you. (ERR_NGROK_6024)

I found an alternative way to tunnel to a server running in colab. You can find the tunnel URL with:

from google.colab.output import eval_js
print(eval_js("google.colab.kernel.proxyPort(8000)")

Unfortunately this is giving me a different error - again it works if you navigate there via browser, but I get a 404 when trying to access it via inTRACKtive.

[aganders3]

Okay I have a method that seems to work. I put my zarr bundle in ./sample_data/tracks_bundle.zarr and opened a tunnel using cloudflared. This is a "quick tunnel" which has some limits (for example number of concurrent requests) but does not require any authentication:

!wget https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64
!chmod +x cloudflared-linux-amd64
!./cloudflared-linux-amd64 tunnel --url http://127.0.0.1:8000 & python ./inTRACKtive/tools/serve_directory_http.py ./sample_data

This gives some output like this, where you can see the quick tunnel URL:

...
2024-11-18T19:05:54Z INF Requesting new quick Tunnel on trycloudflare.com...
2024-11-18T19:05:57Z INF +--------------------------------------------------------------------------------------------+
2024-11-18T19:05:57Z INF |  Your quick Tunnel has been created! Visit it at (it may take some time to be reachable):  |
2024-11-18T19:05:57Z INF |  https://sir-cope-jamie-few.trycloudflare.com/                                              |
2024-11-18T19:05:57Z INF +--------------------------------------------------------------------------------------------+

Pop this URL into inTRACKtive, and it works! For me it was https://sir-cope-jamie-few.trycloudflare.com/tracks_bundle.zarr/. This still has the limitations around needing to keep it running in colab, etc.

I was also able to get this to work with ngrok, but unfortunately it required some ngrok-specific modifications to inTRACKtive as well as the local file server. I'm also not happy with the performance.

[TeunHuijben]

Hi @esgomezm,

We have updated the Python side of inTRACKtive, which has been merged with main today. This might affect your current notebook implementations.

Some takeaways:

• easier command-line interface after pip install git+... (see readme)
  • intracktive convert --csv-file file.csv
  • intracktive serve path/to/folder
• napari plugin to directly open inTRACKtive from a tracks layer
• example notebook to directly launch inTRACKtive from a notebook cell (unfortunately not yet fully integrated with Google Colab, as illustrated in the discussion above)

My plan is to embed inTRACKtive in a jupyter notebook cell using anywidget, I'll keep you posted on the progress.