Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Headless Mode #275

Closed
henrypinkard opened this issue Apr 1, 2021 · 19 comments
Closed

Headless Mode #275

henrypinkard opened this issue Apr 1, 2021 · 19 comments
Labels
enhancement New feature or request

Comments

@henrypinkard
Copy link
Member

Picking up on #262:

Want to add "headless mode" so that pycromanager can start all required java libraries from python (i.e. without having to run micromanager desktop app)
@henrypinkard henrypinkard added the enhancement New feature or request label Apr 1, 2021
@henrypinkard
Copy link
Member Author

@tlambert03

with #274 and #276, pycromanager 0.12.8 now has changes to automatically find the correct JRE version on windows (which comes bundled with MM installer).

With Mac OS you may have to install the correct version of Java runtime environment (JRE) manually to ensure compatibility. I'm on Mac OS 11.2.3, and I can find JRE by going to: System preferences -- Java -- (launches new window) -- Java -- view. Which shows:

Screen Shot 2021-04-01 at 1 16 31 PM

So I'm using JRE 1.8.0_251 which is at the path /Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/bin/java

If you install this same version, you should be able to tell headless mode to run it by following this updated example: https://github.com/micro-manager/pycro-manager/blob/master/test/headless_demo.py

Side note: there might be some issues with the Pycro-Manager image viewer depending on JRE version: micro-manager/NDViewer#10

@tlambert03
Copy link

awesome thanks, will give it a shot soon. I was able to get headless running recently, (but not yet using the start_headless function... I had to start it manually, directly with java on the command line).
will try this out soon. thanks agagin

@carlkesselman
Copy link
Contributor

Is there any way headless mode could be used to start multiple core instances for managing multiple cameras?

@henrypinkard
Copy link
Member Author

Yes, but would also need to add ability to run multiple different Java-python bridges in parallel (i.e. change the default port numbers on which each operates)

@carlkesselman
Copy link
Contributor

Right... that would seem to be necessary. Does this seem like something that would be generally useful, and if not, would there be some other way to achieve this? Ideally, I would like to use a single pycromanager to coordinate image acquisition from three, independent cameras.

@henrypinkard
Copy link
Member Author

It wouldnt be too hard to add, and probably nice to have the option in general to manually set it. The "correct" solution to what youre describing should involve changes to the micromanager core...but that would be a lot more involved.

So this might allow you to hack together a way of doing what you want, but could you maybe describe what you're trying to do in a bit more detail? Do you need them to be synchronized with each other? Are you using TTL triggers?

@carlkesselman
Copy link
Contributor

I'm open to suggestions, but here is what I'm trying to do. I'm using this microscope (https://dx.doi.org/10.1063%2F1.5144487). We have three cameras for our current experiments, one for the SPIM images and two that are taking heartbeat and tail motion data from a live fish. We are taking time series over several hours and for this experiment, each camera is independant. The cameras used for behavioral data are different than the SPIM camera, and ideally each of them will operate at a different framerate so as to avoid motion blur on the tail images. We are using hardware triggering to drive the Z stack acquistion on the SPIM, and also to minimize jitter in the acquistion of the behavoral data.

Right now the data is being acquired manually via three seperate instantiations for micromanager, which for many reasons is not a very good solution. What I want to do is basicly drive three independent data streams from a single python controller script. This will allow me to completely automate our experiments, which we want for reproducability and minimizing the potential for error. All things being equal, it seemed to me that a multithreading approch would be the most elegant solution, but it was not clear to me that MMCore was threadsafe, as the server seems to keep the current camera as part of its global state. My second choice would be to create three independant instances of core, one for each camera and to control them via a single script. That is what lead me down the headless mode path. Ideally, each instances would run on the same server, and a different port. I could then encapsulate each in a seperate thread and go from there. It will be helpful to timestamp each file, so having everything on the same computer/OS will simplify that. Of course agreegate I/O or memory bandwidth might make this a bad solution, but I suspect I will be ok.

Does this infomation help?

@henrypinkard
Copy link
Member Author

Yeah that clarifies things thanks.

I added in the ability to manually specify port numbers to pycromanager (new pip version) and micromanger (tomorrows nightly build)

#293

micro-manager/micro-manager#1265

This script shows how to use it: https://github.com/micro-manager/pycro-manager/blob/master/test/headless_demo.py

This is all untested so I'm not sure what you'll run into. I would recommend getting it going with a single instance first, then multiple on multiple processes, then maybe multiple on the same process.

Also a note on port numbers, there are actually multiple ports used during an acquisition, and they count incrementally up. So dont give your different ports consecutive numbers (4827, 4828, etc), but instead seperate them by at least 10 (4827, 4837) to avoid any possibility of collision.

I'd be curious also to know how headless mode works for you with just a single process. It works on my laptop but this issue remains open because I'm unsure if others have been able to use it

@carlkesselman
Copy link
Contributor

carlkesselman commented Jun 23, 2021 via email

@carlkesselman
Copy link
Contributor

carlkesselman commented Jun 24, 2021 •
edited
Loading

let me know if you would like me to file a separate issue for these. I found the startup issue. It seems that on my platform, java doesn't like the extra quote around the class path. Changing:
classpath = '"' + mm_app_path + '/plugins/Micro-Manager/*"'
to
classpath = mm_app_path + '/plugins/Micro-Manager/*

solves the problem. I'm using python 3.9.4 and java version "9.0.4". I think the reason why this is the case is that the subprocess.Popen function actually looks at the arguments and escapes them using subprocess.list2cmdline(). As it stands, this returns:
'java -classpath \\"/Applications/Micro-Manager-2.0.0-gamma1/plugins/Micro-Manager/*\\" -Dsun.java2d.dpiaware=false -Xmx2000m org.micromanager.remote.HeadlessLauncher'
which is not what you want (e.g. the quoted double quotes), given that the args are passed directly to the subprocess.

Also, once I get things started, I'm getting the following exception:
Exception in thread "main" java.lang.ClassCastException: java.base/jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to java.base/java.net.URLClassLoader at org.micromanager.internal.zmq.ZMQServer.<init>(ZMQServer.java:74) at org.micromanager.remote.HeadlessLauncher.main(HeadlessLauncher.java:41)

This probably makes sense to someone who knows Java (which would not be me). I think things still run after the exception, but might be worth checking.

@carlkesselman
Copy link
Contributor

Let me know how you wouild like me to proced... I can file new issues, or try to fix these things myself and do a PR. The following is I believe an issue:

def __new__(cls, *args, **kwargs): """ Only one instance of Bridge per a thread """ if hasattr(Bridge.thread_local, "bridge"): return Bridge.thread_local.bridge else: return super(Bridge, cls).__new__(cls)

It would seem that we don't want to have only one bridge per thread now.... or perhaps there is a different granularity which is one bridge per port?

@henrypinkard
Copy link
Member Author

let me know if you would like me to file a separate issue for these. I found the startup issue. It seems that on my platform, java doesn't like the extra quote around the class path. Changing:
classpath = '"' + mm_app_path + '/plugins/Micro-Manager/*"'
to
classpath = mm_app_path + '/plugins/Micro-Manager/*

solves the problem. I'm using python 3.9.4 and java version "9.0.4". I think the reason why this is the case is that the subprocess.Popen function actually looks at the arguments and escapes them using subprocess.list2cmdline(). As it stands, this returns:
'java -classpath \\"/Applications/Micro-Manager-2.0.0-gamma1/plugins/Micro-Manager/*\\" -Dsun.java2d.dpiaware=false -Xmx2000m org.micromanager.remote.HeadlessLauncher'
which is not what you want (e.g. the quoted double quotes), given that the args are passed directly to the subprocess.

Gotcha. It still works for me when changing that, so I went ahead and merged it (#294).

Also, once I get things started, I'm getting the following exception:
Exception in thread "main" java.lang.ClassCastException: java.base/jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to java.base/java.net.URLClassLoader at org.micromanager.internal.zmq.ZMQServer.<init>(ZMQServer.java:74) at org.micromanager.remote.HeadlessLauncher.main(HeadlessLauncher.java:41)

This probably makes sense to someone who knows Java (which would not be me). I think things still run after the exception, but might be worth checking.

Based on this answer, this seems to be related to different versions of Java running. All the libraries were compiled with Java 1.8 and you're using 1.9. I don't know enough about Java internals to say if this different class loader will be problematic, but based on the Exception arising this might mean that the server running on the java side might not discover the Java classes that pycro-manager is calling under the hood, so best to try to resolve it.

I think your best bet would be to install the same Java version that I've been developing this all on: 1.8.0_251. This will also avoid the potential for errors with the pycro-manager image viewer caused by different Java runtime environments (micro-manager/NDViewer#10). Since you're giving the path to the java installation directly when launching headless, It shouldn't cause any problems to have this version installed alongside the version you have now--I don't think you need to make it the system default or anything

@henrypinkard
Copy link
Member Author

Let me know how you wouild like me to proced... I can file new issues, or try to fix these things myself and do a PR.

Either opening new issues or staying on here is fine with me. And PRs are much appreciated if its something straigthforward

def __new__(cls, *args, **kwargs): """ Only one instance of Bridge per a thread """ if hasattr(Bridge.thread_local, "bridge"): return Bridge.thread_local.bridge else: return super(Bridge, cls).__new__(cls)

It would seem that we don't want to have only one bridge per thread now.... or perhaps there is a different granularity which is one bridge per port?

The one bridge per thread thing was to try to avoid ZMQ errors, and avoid unnecessary traffic on the port. I don't have a clear idea of why this would need to change for your use case (I very may well be missing something). In any case, trying to get it working with different pycromanager instances on different processes as a first step might avoid any issues from this for now I think

@carlkesselman
Copy link
Contributor

I think the issue is that given this new function, I will never get a bridge that points to a different port, and hence, the acquisition will never use a different instance. I think if you just add a check to include the port number when deciding to reuse the Bridge, it should be fine.

@henrypinkard
Copy link
Member Author

Ah right. Makes sense. Do you want to take a crack at it and PR?

@carlkesselman
Copy link
Contributor

sure.

@henrypinkard
Copy link
Member Author

@carlkesselman did you ever figure this out?

@carlkesselman
Copy link
Contributor

carlkesselman commented Oct 7, 2021 via email

@henrypinkard
Copy link
Member Author

Ok cool, headless mode is official!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tlambert03 @carlkesselman @henrypinkard