m2e 2.4.x bug: m2e incorrectly uses older transitive dependency if intermediate project is imported into workspace #1501
Comments
garretwilson
changed the title
|
This is draining; I don't know what to do I'm trying to downgrade to Eclipse EE 2023-06 with the old m2e to see if that fixes the problem. I even upped |
|
I finally got the downgrade to Eclipse EE 2023-06 to work the old m2e. Here again m2e seemed to be stuck with the #1414 error, even after changing the versions. I finally had to remove the project and delete all Eclipse project-related directories and files (e.g. Since things were working again, I switched back to m2e 2.4.0.20230802-0719. Things still work. But I found out what confuses it. Look at the changes on this branch: https://github.com/globalmentor/globalmentor-base/tree/issues/JAVA-313 I changed the structure of the POMs to add an intermediate POM, and this puts m2e 2.4.0.20230802-0719 into a huge confusion when I remove the project and re-import it with the new structure. I'll need to see whether I can get around it by deleting the settings files again, and whether this happens in m2e 2.3.x. But I'll have to do that tomorrow. |
|
OK, here's what I know. I start out with an aggregate POM like this:
The modules in <modules>
<module>core</module>
<module>management</module>
</modules>Then I add an intermediate
"Foo" and "Bar" use <modules>
<module>parent-pom.xml</module>
<module>core</module>
<module>management</module>
</modules>An explanation of the reasoning for this, along with complete examples, are in by blog post Improving the Maven Bill of Materials (BOM) Pattern. The problem is that if I remove the project, make the change, and then import the project back to Eclipse without removing the original Eclipse project files, Eclipse gets very confused about the version. As explained above, projects that reference this project inside Eclipse wind up referencing a previous version (perhaps brought in transitively by some other dependency), even though that does not reflect that The only way I've found to move forward after changing the POM hierarchy is to 1) remove the project, 2) delete all the Eclipse project files, and then 3) import the project back into Eclipse. It would appear that something set up in the project files from the first hierarchy aren't being updated when re-importing the project back into Eclipse. I performed these steps in the plain Eclipse EE 2023-06 with m2e 2.3.x, and then switch to the same Eclipse that had v2.4.x installed, and it still worked, so I can't say whether this m2e confusion is related to any v2.4.x changes or not. |
|
Oh, no—this problem surfaced again, and this time it's not an issue of corrupted Eclipse projects. I just published the Here are steps to reproduce:
You will see the Eclipse EE 2023-06 with m2e 2.4.100.20230802-0719 shows an error in
However You'll also see that building from the command line using OpenJDK 17 Please investigate this problem. I'll be happy to answer any more questions. Right now this is breaking the project, and it looks like to get around this I'll have to publish another version of Note that SLF4J is getting ready to release a BOM using this type of POM hierarchy, as I discussed with the SLF4J author already, so we may need to make last-minute changes in the next SLF4J release if this is going to cause problems with m2e. See SLF4J-437. |
Have you tried Project > Update Maven Project and just updated all projects in the workspace? Beside that it seems this is not really something the usual committer/contributors encounter and thus handled with very low priority, so probably you like to provide a PR to fix that together with a test-case to verify it works? Beside that, sponsoring is a good way to get specific issues fixed, so if this is crucial to your business and you likes to speed up the development of m2e in general a sponsoring would allow me to assign more time-slots to bug fixing or contact me for an individual contract for example fixing a specific issue. |
Yes, so many times I've lost count. I should have emphasized more in that last comment that I have removed But I have more information that makes this more mysterious: my steps to reproduce the problem do not work! If I clone the same projects and import them into a new Eclipse installation, the problem does not appear! And even more mysterious: the m2e Maven Dependency Hierarchy for And even stranger, removing the But still I can't reproduce this in a brand new Eclipse installation. I realize you can't fix a strange problem if you can't reproduce it, so I'll keep trying to reproduce it. If by reading these descriptions you have any ideas of what might be causing this, please let me know. |
|
Here's something perhaps significant, and perhaps related. When I clone
However in the old Eclipse installation (the one giving the error), this project shows the
(Also note that the old installation correctly shows Could someone clone |
I think I found the reason for this, and it appears to be a bug in itself. The I'll have filed this separately as #1504. And at least I found out why this difference between my old and new test installation: in the latter the imported projects were inside the workspace directory. See #1504 for more details. |
|
Bingo! I have reproduced this problem!! 🎉It may in fact be related to #1504. You must clone these repositories to some directory outside of your workspace directory.
As soon as you import If you delete the |
|
I have verified that this bug was introduced with m2e 2.4.x. A fresh installation of Eclipse EE 2023-06 with m23e 2.3.0.20230523-2033 does not exhibit this bug. If you add the snapshot repository Apparently m2e 2.4.x introduced some sort of dependency lookup bug for projects that are imported into the workspace. |
garretwilson
changed the title
Nice work. It would be great if you would setup up a dev-Eclipse for m2e, launch the provided launch config for the m2e-product and reproduce the problem there. You could then probably also try to investigate the reason for this in the sources and then a fix is usually relatively close. :) |
I'm sorry, but I must get back to my main project. I've lost several days with this and #1414 and #1504 and filing JGit bugs and filing Eclipse compiler bugs and filing XML editor bugs … It makes me happy that I can put in sufficient work so that your team can reproduce this, but sincerely at some point I have to do my own work. If you have any questions or doubts let me know and I'll be happy to provide additional information. |
|
At this point I suspect (hope 🤞) this problem never had anything to do with a corrupt project or the POM structure, but it is simply a new bug introduced in m2e 2.4.x. As we've seen, the problem only surfaces after 1) updating to v2.4.x, and 2) updating a Maven project with m2e.
|
|
Maybe similar/related to this bug #1474 |
|
@garretwilson Do you think you can create an automated test case with minimal projects derived from your earlier comment? Having a test case that verifies this story will certainly help in fixing it and preventing further regressions. |
As I don't know what is causing the problem, I don't even know what "minimal projects" means. Maybe we could reproduce this with a single project. Maybe what I've given you is already minimal. It's hard to guess without knowing what is causing this. Seeing that I've already spent days to come up with at least some set of projects that should reproduce this, I would appreciate it if someone could please at least take five minutes to follow the test case I provided and confirm that they have successfully reproduced the issue. |
|
For example, I'm wanting to release a new version of It took a long time to find something reproducible, and I'm currently putting some of my own projects on hold waiting for someone to tell me the can reproduce this before I make any changes. |
I'll try to reproduce it tomorrow. |
I have tried to reproduce your problem and cloned all three git repos you mentioned into a folder outside of my workspace and just imported all projects from it. The project names are not exactly as mentioned them, but if there is an error with the workspace resolution the project name should not matter since it has actually nothing to do with the GAV. I did not notice exactly the errors you got, but for me the Maven Dependencies Classpath container failed to resolve the
I'm not sure if this exactly your problem, but maybe m2e simply does not continue to build after the first of such errors and in my case the order is different due to the different project names. If there is really a problem with the workspace resolution I assume it is due to your relatively complex parent+bom structure in the same workspace. But that should then be reproducible with a stripped down version of your projects. |
|
The error remains if I close all projects except for |
First of all @HannesWell , thank you so much for taking the time to try to reproduce this. However I'm a little confused about some of the errors you got. I'll try to see what I can do stripping down this set of projects, but in the meantime let me respond to a couple of your questions.
After cloning the projects separately, I use File > Import > Maven > Existing Maven Projects. I also see that one of your errors is that it failed to read an artifact descriptor for Plus I see a long list of errors from missing libraries, some of which aren't even mine, and some of which have been available on Maven Central for years! No, those are not the errors I'm getting. I have no idea why you are getting those errors. That is basic m2e load-from-Maven functionality.
I see you're looking at the "Package Explorer", not the newer "Project Explorer", so that might be why that is happening. It's one thing if you can't reproduce the problem, but I'm seriously at a loss to explain what you're seeing. Surely we might have missed a step somewhere. (I just download the Java EE 2023-06 ZIP file and expand it in a directory and run it from that. I'm not using the OOMPH or whatever. It shouldn't make a difference, but I thought I'd mention it.)
If you mean that you just imported the parent directory of the three projects, then no, I can't guarantee that will reproduce the problem. You must import each of the projects separately, as per the instructions. Did you import each one of them separately using File > Import > Maven > Existing Maven Projects? (When you import each of the projects mentioned, e.g. |
|
I have created a tiny project that easily reproduces this bug. I am attaching the ZIP file with the project to this ticket as m2e-core-bug-1501-test-project.zip. Here are the reproduction instructions, which are also present in the readme file in the project.
At this point m2e will become confused, and use the wrong version of
If you're still skeptical, go to the command line and issue You can even compile the project using Java: mvn clean verifyThe part that confuses m2e is in the POM: <dependency>
<groupId>com.globalmentor</groupId>
<artifactId>globalmentor-awt</artifactId>
<version>0.6.1</version>
</dependency>
<dependency>
<groupId>com.globalmentor</groupId>
<artifactId>globalmentor-core</artifactId>
<version>0.7.4</version>
</dependency>You'll see that (Note that if you switch from m2e 2.3.0.20230523-2033 to m2e 2.4.100.20230802-0719 or vice versa, you'll afterward need to tell m2e to refresh the Maven projects using This shouldn't be hard to track down. It was introduced in 2.4.x. Just go look at the new code that was added relating to the dependency graph regarding loaded projects. |
I followed you steps exactly and I indeed already have errors after only importing this simple reproducer project and I have similar errors like before. I first assumed these were follow-up errors on the failure in the Maven Depenency container, but copying the their description revealed the following (Note to our-self, these stack-traces could be made more visible, easier accessible): So the root cause here is that globalmentor-base:0.7.4 cannot be resolved.
Are you sure the release fully succeeded?
MvnRepository also shows 0.7.3 as latest version: https://mvnrepository.com/artifact/com.globalmentor/globalmentor-base So I wonder have you installed globalmentor-base:0.7.4 in your local repository or do you have another repository specified in your users |
Hi, @HannesWell . Yes I just realized that this morning. It appears that there was some problem with Maven Central and that POM did not get published. I have filed ticket I believe this is a separate issue unrelated to the bug described in this ticket. I will completely understand if you want to wait until we find more answers in In the meantime, though, if you want to continue reproducing the issue despite the missing
I don't want to confuse this ticket. If you want to do the workaround, awesome; but if you want to wait until |
|
I'll try to find some time tomorrow, for now its too late here. |
After I have done this, I can now reproduce the issue exactly as you have described in #1501 (comment) and can confirm everything you mentioned. This even happens in a simple Eclipse SDK plus m2e respectivly in an IDE launched from a m2e-dev workspace (makes it simpler to debug 👍🏽). But I don't think that m2e really get confused about the version of globalmentor-core, because looking at the Therefore I assume that this is more an issue JDT Classpath configuration, maybe the order of entries or what is exported by referenced projects. |
|
Bingo, this is indeed a regression from #1424. and call @laeubi I think #1424 needs adjustments or we need to revert this. |
|
@garretwilson you can work-around this problem by moving the dependencies in the workspace to the end of the list. This is of course not a solution, but should unblock you for now. |
|
Just created #1517 with a reproducer. |
|
@HannesWell I think this is the probably a bug of JDT because it searches deep-first while bread-first is what we want and one can expect here. |
I assume think this is a bug but more by design. As far as I understood it and simplified, the container creates a flat list of jars/directories as classpath and in this case it starts with adding project-a and its transitive) dependencies and then continues with the other dependencies of project-b. And then the Compiler searches that flat class-path as usual, like the ClassLoader does. |
|
JDT must take precedence of directly added dependencies, this has nothing to do with the Classlaoder (that is created as part of this later on). So it must first add al level-one (directly declared), then decent into that transitive ones and so on... otherwise such situation can always happen. One can even argue that if we say it is not an error, the order must be defined in the pom.xml otherwise as you have shown it to work already. |
Is that written anywhere? I think on the very basic level the JDT Classpath is not that smart and its the task of container creators like M2E to properly order the entries. But even if it turns out to be a bug in JDT I'm relatively sure that it won't be fixed for 2023-09. Therefore I suggest we revert #1424 and accept this loss in extra convenience for remote-debuggers and then either fix JDT or think of another solution for that. Because obviously this is causing regressions. |
|
It turns out that the |
Its either a bug (then it should be changed) or a feature (the the container creator can't do much), the container itself correctly declares the order reflected from the pom and m2e should expect it to handle this appropriately. I'm not sure if it is even specified in full detail how maven handle the case, but changing the order by m2e seems not very useful as it might be undesired then as well. For me this more looks like a limitation of the Workspace resolution, because just consider project A is in the workspace and uses an older dependency, project B references A and includes a newer one (that has removed a method), then this will fail at runtime (e.g. unit test executed) but will not show any compile error on the project itself. So even though the project B shows a problem now it previously has just hidden the problem of project A needing an update of its dependency to work in this scenario. I actually prefer to see the problem in the IDE than somewhere in runtime ... |
The dependency resolution mechanism is well documented for Maven:
While this it is definitely a desirable goal to get to now about such runtime conflicts in the IDE and your example is valid, the classpath-container is the wrong place to solve it and not even a standalone Maven knows it and AFAIK Maven is not designed to solve that problem. If one really want's to prevent such runtime errors one needs to fully obey Semantic Versioning in all own artifacts and dependencies and needs a build-tools that checks compatibility and SemVer correctness, but we both know from OSGi that even then things are not that simple to solve in the IDE. :) In my opinion the Maven Dependency classpath container should behave just as the maven resolver and should resolve breath-first regardless of if a dependency is in the workspace. Otherwise users just get confused and more issues like this will be opened. Can't you achieve your goal of #1424 with another solution that does not affect the classpath so that we can revert #1424? |
As said that's nothing m2e can solve, it is not the classpath container that do this, it is JDT / user code that reads the entries of a classpath container, so one would need a flag for the container to influence how that behaves, especially m2e might want to ignore all.
I won't fully revert it but set the |
Fixes eclipse-m2e#1501 This partially reverts "Export the entries of transitive dependencies", commit-ID d31ccee
Done. |
This partially reverts "Export the entries of transitive dependencies", commit-ID d31ccee And adds a test-case to reproduce the issue and prevent future regressions. Fixes eclipse-m2e#1501
|
@garretwilson thanks for this bugreport. |
Yes! 🎉 I can confirm that v2.4.100.20230823-0832 fixes this bug! What a relief. I have tested this both with an Eclipse installation upgrading from v2.4.100.20230802-0719 to v2.4.100.20230823-0832, as well as a plain Eclipse EE 2023-06 installation upgrading from v2.3.0.20230523-2033. Interestingly after upgrading from the buggy version v2.4.100.20230802-0719, I needed to refresh not only the project using the dependencies, in this case @HannesWell thank you for taking this bug seriously and taking the time to reproduce it and then fix it. @laeubi thank you for working with @HannesWell and providing input to help get this resolved quickly once it was reproduced. |
|
As a side note, it turns out (see |
Great. Thanks for testing!
That's good to know. Thanks for letting us know. Do you know if this problem will be addressed in the Nexus Staging Plugin? Out of curiosity, what structure are you using now? I havn't looked it up, but wouldn't it be simpler to to have the bom-project as a sibling module of the other artifacts, use it as parent of the root pom and import it as bom there as well and at the same time aggregate it with the other artifacts, i.e. list it in the root pom's module section? As you have mentioned in your blog-post aggregation is independent from inheritance. |
It's actually the same logical structure. The difference is that instead of having
I get a little confused reading your suggestion, as I have a separate "root POM" as a completely separate project, which serves as the "bootstrap" POM for all the projects. I'm calling the per-project "root" the project "top-level POM". Maybe that's what you mean here. In that case, you're proposing that the BOM be 1) the parent of the top-level POM (wait, so wouldn't the BOM be the top-level POM then?), 2) the sibling of the other artifacts, 3) import the BOM into the top-level … wait I'm sorry, I'm totally confused. I think there's a terminology problem, too, regarding "root" and "top-level'. Anyway do you think we're getting off-topic in this ticket? Or is it OK since it's been closed? (I sort of think it might be better elsewhere.) Maybe you'd like to send me a direct email to discuss the blog post? Or even better, send an email to the Maven User List. I already requested feedback there, and I would really love to know if there is a better BOM pattern. |
No idea yet. I only discovered what what triggering the bug this morning, and they said they'll send it over to the developers. I may have to file a separate bug for the plugin (this bug was for Maven Central). I imagine there's a long, winding road before we get any answers. 😅 I'll be sure and update MVNCENTRAL-8292) with any news. |
I'm using Eclipse EE 2023-06 with Maven 3.9.1 on Windows 10 with OpenJDK 17. I had to upgrade m2e to v2.4.0.20230802-0719 because of #1414. I don't know if that snapshot version of m2e is causing the problem, but this was working fine before, so it wouldn't surprise me if the new m2e has a bug. I've spend a frustrating, tiring time of trying to coax Eclipse to act right, to no avail.
The summary (I'll explain how to reproduce it below) is that in the https://github.com/globalmentor/globalmentor-base project, which is at
com.globalmentor:globalmentor-core:0.7.4-SNAPSHOT(a subproject), I recently (see commit history) added acom.globalmentor.collections.iterables.Iterables.toStream()method, but another project in Eclipse which is usingcom.globalmentor:globalmentor-core:0.7.4-SNAPSHOTis not seeing that method. Instead it seems to be using v0.7.2, not v0.7.4-SNAPSHOT. That other project is https://github.com/globalmentor/guise-mummy .Reproducing this isn't simple, because multiple projects are in play. Let me try to give you a scenario; if you can't reproduce it with this, I can try to put together something simpler, but that will take a lot of work and at the moment I'm already exhausted from trying to straighten this out.
Start by cloning and importing all these projects into Eclipse:
All these are at SNAPSHOT versions. You'll notice that
globalmentor-basehas a subprojectglobalmentor-core. There you should easily find the latestcom.globalmentor.collections.iterables.Iterablesclass, and note that it has atoStream(@Nonnull final Iterable<T> iterable)method. Note also that the version iscom.globalmentor:globalmentor-base:0.7.4-SNAPSHOT.Then in the
guise-mummyproject you'll see thatguise-mummyis actually a subproject, which has theio.guise.mummy.deploy.aws.S3class. On line 166 you'll see that it uses the newtoStream()method above:However Eclipse shows an error message for this line:
And if you click on the
import static com.globalmentor.collections.iterables.Iterables.*in theS3class, and then use "View in Project Explorer", you'll see that it's using theIterablesclass fromcom.globalmentor:globalmentor-core:0.7.2!Open the root
guise-mummy-bomproject from theguise-mummyrepository. Inpom.xmlyou'll see that in<dependencyManagement>it pulls in v0.7.4-SNAPSHOT:But you don't have to take my word for it. Just run
mvn dependency:tree. You'll see that the onlyglobalmentor-corethat appears in the output iscom.globalmentor:globalmentor-core:jar:0.7.4-SNAPSHOT. So Eclipse and m2e are wrong to be linking to v0.7.2.Finally go into each of the projects on the command line and do a
mvn clean install(in the order listed above). On my machine they all compile with no errors. But in Eclipseguise-mummyis pulling in the wrong version ofglobalmentor-core.The text was updated successfully, but these errors were encountered: