Incremental build building more than one project?

[laeubi]

Currently m2e has an incremental builder what works quite fine when one project is changed, but it seems it also happens that multiple projects change/needs rebuild and then m2e might be able to optimize some tasks.

I therefore wonder if there is a way to process more than one project I take a look at

org.eclipse.core.internal.events.BuildManager.basicBuildLoop(IBuildConfiguration[], IBuildConfiguration[], int, MultiStatus, IProgressMonitor, int, boolean)

but it seems there is not much to find out about other pending builds that might happen...

[mickaelistria]

I believe the incremental build can be invoked on multiple project from API perspective; calling from workspace.build(BuildConfig[], IncrementalProjectBuilder.INCREMENTAL_BUILD) is a valid thing to do. Not sure whether this combination is actually invoked or not -but yet it's valid to call it and should work and thus is supported-.
However, I don't fully get your question: the IncrementalProjectBuilder is project-specific, it doesn't know much about its caller and about the ongoing build process (which other projects are involved and so on). So how do you think you can optimize it according to it? I imagine you can make m2e keep some interesting state about the workspace and Maven projects in its own builder to optimize how 1 particular builder instance will perform according to that data that has been made accessible to it; but then you don't really care about how builds are invoked if you already have some state available; do you?

[laeubi]

However, I don't fully get your question: the IncrementalProjectBuilder is project-specific, it doesn't know much about its caller and about the ongoing build process (which other projects are involved and so on).

That's actually the question, also the builder is specific to one project, it is very common that many projects in the workspace share the same build config! So currently it works like this (if I understand right), there is a set of projects that needs to be (incrementally) build, then a build order is computed and then the builder is called for A, B, C, D ...
What I'm looking for is some kind of batch instead, so if my builder is called for A, I know there is more to come or even process them as one batch instead of each single chunk.

So how do you think you can optimize it according to it? I imagine you can make m2e keep some interesting state about the workspace and Maven projects in its own builder to optimize how 1 particular builder instance will perform according to that data that has been made accessible to it; but then you don't really care about how builds are invoked if you already have some state available; do you?

One example is setup of the maven session and lookup of artifacts, one might even see an incremental build of multiple projects as a special form of invoking maven with mvn -pl <my list of changed projects>

[mickaelistria]

What I'm looking for is some kind of batch instead, so if my builder is called for A, I know there is more to come or even process them as one batch instead of each single chunk.

I believe it's how JDT does it: basically, the build keeps track of everything dirty from all Java projects and decides to rebuild whatever part makes sense to it when a project does request a build. This build may include generating .classes from other projects.
So the IncrementalProjectBuilder is more a way to request "I want this project in a built state", not a particular request to "build that particular project"; but nothing in the builder does prevent from processing other content. The only thing to take care of is to minimize resource modifications, or at least group them into a single WorkspaceOperation to not trigger too many downstream actions.

[laeubi]

So you mean the builder should just enqueue the project and build it later would be sufficient?

[iloveeclipse]

builder should just enqueue the project and build it later would be sufficient

Builder is supposed to build a project. I guess I simply can't understand what do you want to ask.

[iloveeclipse]

@laeubi : if I understand your question right, you might want to listen to org.eclipse.core.resources.IResourceChangeEvent.PRE_BUILD resource events in some "global" m2e "build manager" and do whatever you want in a singleton there (ideally not directly on the listener callback), just "wait" in your builders for the common code to process whatever you want to process.

[laeubi]

@mickaelistria @iloveeclipse

To summarize your suggestions would/should something like this work:

1. Listen to org.eclipse.core.resources.IResourceChangeEvent.PRE_BUILD and create a "context"
2. in the incremental builder just add projects/delatas/whatever to this "context"
3. listens to IResourceChangeEvent.POST_BUILD and schedule the "context" to perform a batch update of everything collected in step (2)?

[laeubi]

Lets assume I have a really powerful source generation framework but it is costly to start the generation (e.g. requires a native process to start), but is fast afterwards.

Now project A + B needs an update, I can perform it serial (requires two slow startups) or I can collect tasks for A + B and then process them as one batch (only one slow startup).

Afterwards the generated data needs probably be build, but that should also be the case with running it serially?!

[iloveeclipse]

I still don't get what do you want to do.
Can the "slow native process" be started only once per session, or it needs to be re-started on every build?

Do you want to change/generate something that is common to all N projects during same build cycle - you can generate that on first project build and re-use on N-1 remaining projects. In this case no rebuilds are needed.

If that what is generated changes every N project resources, and you want to "postpone" the build somehow - on next build cycle you still would need to start this "powerful generator" again, or how would you know that it doesn't need to run?

[laeubi]

In maven you can build a single project, or you can build many projects at once, of course the need to have things in common (e.g. used maven version), but the later is more efficient because before the project can be build maven needs to setup certain things.

So lets say I have 10 projects and they take 30 seconds to build together, calling maven for each single project separately will take 90 seconds (just as an example).

[iloveeclipse]

Still no clue what do you want. So you want to run maven "external" build for all/some affected projects? And after this build you want to pick up changes generated by maven?

Eclipse build works either triggered by resource changes of the workspace or programmatically.

So if you have N projects depending on some common maven build, where ther original change is coming from?
I don't get where the "external maven" build is supposed to be triggered.

File changed on file system? Some magic maven property changed globally and you explicitly trigger a build? Or just a dedicated "user" action "do some maven related task"? In later case you can just run maven and let the incremental build proceed (after maven is done) as usually.

If you generate something "externally" by maven during the build in the builder, and your builder depends on that changed resources, and your builder knows that, after running maven, the builder probably should request rebuild via IncrementalProjectBuilder.requestProjectRebuild(boolean) or even IncrementalProjectBuilder.requestProjectsRebuild(Collection<IProject>)? Same for other builders? Or other builders could check some "global" singleton if the maven was triggered and if changes in the delta require "external maven" process to be started or not, and might also don't do anything and call IncrementalProjectBuilder.requestProjectRebuild(boolean) too?

Or is the every builder supposed to trigger "external maven" process on every project because every project might contain some maven related config?

[laeubi]

Actually there could be different triggers, so the triggering is not really something I like to care about, but maybe it helps to show some code parts e.g. in m2e we have this:

https://github.com/eclipse-m2e/m2e-core/blob/0dcc3325762b357a362b13ad1a5182ba56c527a3/org.eclipse.m2e.core/src/org/eclipse/m2e/core/internal/builder/MavenBuilder.java#L193

as you can see it is just about building one project.

Now assume I choose Project > Clean (but as said the trigger is not that important) this results in it being called inside a loop

eclipse.platform/resources/bundles/org.eclipse.core.resources/src/org/eclipse/core/internal/events/BuildManager.java

Lines 501 to 521 in fca4635

	for (IBuildConfiguration config : configs) {
	if (config.getProject().isAccessible()) {
	IBuildContext context = new BuildContext(config, requestedConfigs, configs);
	// Inner loop over builders in one project
	basicBuild(config, trigger, context, status, Policy.subMonitorFor(monitor, projectWork));
	builtProjects.add(config.getProject());
	// Check if we should continue with other projects
	if ((rebuildRequested || !projectsToRebuild.isEmpty())
	&& isEarlyExitFromBuildLoopAllowed()) {
	if (lastIteration) {
	// run build for all projects at least once
	continue;
	}
	// Don't build following projects if one of the predecessors
	// requested rebuild anyway, just start main loop from scratch
	break;
	}
	}
	}

and in the debugger I can see that the array has > 500 build configs (its quite a large project).
So MavenBuilder is getting called > 500 times with one project and because I don't know that (in the MavenBuilder) it start/shutdown "maven" > 500 times.

So what I want to archive is that

• I get the 500 Projects as one chunk and can process them in one run instead of one after the other

or alternatively

• I get some clue "there is more to come this is 1 of ..." so I can keep at least some things between invocations of the builder