Bug 321667

The workaround doesn't even really work. It works for "kdesrc-build nepomuk-core", but not for "kdesrc-build", which then says

Error updating kdelibs/nepomuk-core, removing from list of packages to build

due to trying to get it with svn

# kdesrc-build running: 'svn' 'co' '--non-interactive' 'svn+ssh://svn@svn.kde.org/home/kde/trunk/KDE/kdelibs/nepomuk-core' 'nepomuk-core'

I see the need for a better semantic to this. What would you propose as being most useful and memorable?

Something like

module-options
    # Some scheme to select modules as a set, could be 'use-modules' also
    applied-to nepomuk-core
    cmake-options -DCMAKE_FOO=BAR
end module-options

It could also be made to override individual modules by specifying a name (perhaps would make the 'applied-to' option an error since it's implicit)?

The way I see it allowing for option sets would allow you to also "stack" option sets for a module multiple times that way you can set common options for a given subset and then more specific options for a narrower subset.

But the problem is what to name the construct to make it clear that such options should override *existing* modules, as otherwise we're back to the same syntax we have now.

(In reply to comment #2)
> I see the need for a better semantic to this. What would you propose as
> being most useful and memorable?
> 
> Something like
> 
> module-options
>     # Some scheme to select modules as a set, could be 'use-modules' also
>     applied-to nepomuk-core
>     cmake-options -DCMAKE_FOO=BAR
> end module-options
> 
> It could also be made to override individual modules by specifying a name
> (perhaps would make the 'applied-to' option an error since it's implicit)?
> 
> The way I see it allowing for option sets would allow you to also "stack"
> option sets for a module multiple times that way you can set common options
> for a given subset and then more specific options for a narrower subset.
> 
> But the problem is what to name the construct to make it clear that such
> options should override *existing* modules, as otherwise we're back to the
> same syntax we have now.

Yes, "module-options" seems clear and unambiguous to me. It's "options for an existing module", so if the module isn't known at that point, it will give an error.

I like the two possibilities: "module-options kdelibs"  to apply a set of options to kdelibs, and module-options with a list of modules that these options apply to (but I'm not sure about the name of the key there... you suggested applied-to or use-modules.... at least "applies-to" would be an improvement of the first one, no? Or even "apply-to", same grammar as "use-modules". Alternatively "options-for", just "modules"... OK nothing is really great. Let's say "apply-to"?
I'm afraid reusing "use-modules" with a slightly different meaning could be a bit confusing.

Stacking options is very useful indeed. But on that topic, it would be useful to be able to remove a cmake option or cxxflag in a given module-options... :)
My use case for that is that have -DQT_STRICT_ITERATORS in cxxflags in the global section, but I need to remove it for one bad module, calligra. Hmm tell me if I should open a separate report for that, I'm mixing 3 issues here (module search, module-options sections, and removing options for a specific module).

Cheers,

Having a way to filter out options from cxxflags/cmake-options should probably be a separate ticket.

However, have you considered adding -UQT_STRICT_ITERATORS to the cxxflags for just calligra? That might work, and *should* do the right thing already.

Ah, I thought I tried it, but maybe that was with cmake-options, or maybe I was confused between adding and replacing cxxflags. It adds, indeed. Perfect.

So back to the main issue here: I can't compile attica, soprano, or nepomuk-core anymore because it's trying to get them from svn :)

Somehow kde/applications/kde-baseapps isn't getting updated nor built, either....

If this is still a big problem, then a workaround that can be used until this is properly implemented is to mention the module you want to override the options for within the use-modules

e.g.,

module-set
    repository kde-projects
    use-modules nepomuk-core attica soprano kde
end module-set

module nepomuk-core
    # Override options as normal
end module

Unfortunately you have to make sure you mention the modules you wish to override before any parents due to bug 321883, but this syntax should work even now (only for a module-at-a-time though).

OK, thanks for the workaround for overriding options.

Please note that the main blocker now is comment #6 though:
when just specifying "use-modules kde", many modules are not being built at all (like kde-baseapps, but also many many more).
Tell me if I should start splitting this up into multiple reports; I've been overloading this one with all the things that break when using "use-modules kde".

The issue here is a side-effect of an implementation decision to avoid recursing for repos under existing repos.

The canonical example were the "SuperBuild" style repositories where you could have a base module within kde-projects (something like kde/kdefoo) which would be a real git repository that had a CMake script to checkout the rest of the module as git submodules and run the build that way. The components of kdefoo would then be real git repositories in kde/kdefoo/*.

It doesn't make sense to build both so kdesrc-build stops at the higher-level repo (since the leaf repos can be manually requested with the kde/kdefoo/* syntax).

Calligra is another example. "calligra" itself points to a git repository for the whole office suite, but it also contains calligra/krita-extensions/* and calligra/calligra-history (though this is marked inactive, at least). Prior to the addition of the ignore-modules option there was no way to avoid pulling these repos-upon-repos in without stopping at the topmost repo level (it was still possible to be specific to ask for these subrepos).

ANYWAYS, it looks like even with all this in mind, that it is faulty not to fully recurse when the requested entry is not even a real repository at all. I.e. the current impl seems to improperly reset the 'do-not-exceed-repo-level' flag to always be the lowest of any found repository, even when not looking at a direct descendant chain.

In the example of looking for "kde" kdesrc-build does fine for kdeadmin, as all of its actual repos are 3-level (kde/kdeadmin/*). Next it hits kdeplasma-addons which is 2-level (kde/kdeplasma-addons) and from then on kdesrc-build will only allow 2-level git repos under kde/, which gives the remainder of the module list (kdepim, kdeexamples, kdelibs, kdepimlibs, kdepim-runtime, kde-runtime, kde-workspace, superbuild, kdegraphics). The modules are then put in order, filtered, etc.

I'll try to figure out the best way to solve tonight, but there you have the diagnosis.

Created attachment 80986 [details]
Removes problematic kde-project module filter

This patch is actually sitting on my disk as a git commit and should be perfect for your use case.

However I would want to get the new "ignore-modules" option to *also* act fully recursively. Without that there's a lot of extraneous modules I seem to get from things like extragear/graphics (e.g. if I ignore 'digikam' I still get digikam-doc, likewise for kipi-plugins). I would like to fix that as well before pushing to master.

Also there is the newly-interesting question of "how do I build *just* kdelibs?" (try it, it's fun ;). But this should give something to experiment with while I'm off vacationing for half a day tomorrow.

Updating bug metadata now that more info is available on the issue.

This patch helps indeed. I went from 92 modules to build, to 272.

Indeed, building just kdelibs isn't possible anymore. Can't "kdesrc-build kdelibs" do a recursive search for a module with that name?

Same thing for "module kdelibs" (to override some options).

(In reply to comment #13)
> Indeed, building just kdelibs isn't possible anymore. Can't "kdesrc-build
> kdelibs" do a recursive search for a module with that name?
> 
> Same thing for "module kdelibs" (to override some options).

Both should be possible. The command line can prefer actual modules to guessed kde-projects paths; overriding module options would imply a single module if we adopt the syntax you proposed in your comment 3.

If you wanted just kdelibs (and not nepomuk-core, kactivities, etc.) then we can still use the single module notation:

module kdelibs
    repository kde:kdelibs
end module

But perhaps it would be prudent to have something like 'kdefoo/' (trailing slash) in use-modules ask for a given module and not any children? Seems like overkill, though it might be useful at the command-line (unless there is a better way to disambiguate between kdelibs/* and kdelibs.git).

Git commit a033866429b62c29fd93aea843c2f21427634ec8 by Michael Pyne.
Committed on 06/07/2013 at 04:50.
Pushed by mpyne into branch 'fix-recursing'.

kde-projects: Don't auto-filter git-repo's direct children.

Kind of a mouthful, but remove a filter that was added to avoid
automatically including an actual git repository's children that were
also git repositories. E.g. think of something like a SuperBuild git
module that also had its component git modules as logical children
within projects.kde.org.

This filter is unnecessary now that kdesrc-build supports both kde.org
metadata (kde-build-metadata/build-script-ignore) and user-configurable
module filtering (ignore-modules option).

More importantly this filter precludes many other desirable types of
group syntax (e.g. including all of kdebindings just by asking for
kdebindings).

So we remove the filter. The trouble I've seen so far is that it is
now rather difficult to build *only* kdelibs, since kde/kdelibs is a
logical parent of nepomuk-core, nepomuk-widgets, and kactivities. This
can be worked-around by using a normal single module declaration. E.g.

    module kdelibs
        repository kde:kdelibs
    end module

On the other hand, the recently-added config file option
"ignore-modules" should now remove any kde-projects modules that contain
a path component(s) matching the ignored module. You may have to be
specific with a given ignore atom for this reason (e.g. ignore
playground/libs, not 'libs' otherwise you'll also ignore
kdegraphics/libs at this point... though that was also true before this
commit).

M  +24   -0    doc/index.docbook
M  +1    -1    modules/ksb/BuildContext.pm
M  +10   -13   modules/ksb/KDEXMLReader.pm

http://commits.kde.org/kdesrc-build/a033866429b62c29fd93aea843c2f21427634ec8

I've pushed a more fleshed-out version of the patch to the 'fix-recursing' branch of kdesrc-build if you wish to try.

I think I'll have to treat the idea of selecting the module (instead of group of modules) from the command line as a separate issue. I don't think that should be too difficult to adjust but that whole portion of the code probably needs yet another refactoring to get the code to match the intent.

Git commit 5d638acab096f17dd1d136868e674d36eed6f21e by Michael Pyne.
Committed on 23/07/2013 at 22:58.
Pushed by mpyne into branch 'fix-recursing'.

kde-projects: Accurately track parent module set.

This is unfortunately a giant change, as all of the functionality that
is encompassed into module-sets currently had to migrate over to
multiple separate classes, including the new ksb::ModuleSet class and
subclasses.

This was a long-overdue change, however, and should allow for accurately
tracking a source module-set for a given module.

On the other hand this migration of logic has made it easier to
understand each of the individual pieces where they stand (e.g. there is
no longer a separate expandXMLModules and expandModuleSets).

In addition we can properly handle ignore-modules with wildcards just as
we do with use-modules (they even use the same matching logic) which
means that it is safe to integrate this into master (assuming no extra
boogs get added, of course).

This will also help with fixing some of the extant module-selection bugs
(321883, 299415).
Related: bug 321275

M  +151  -403  kdesrc-build
M  +67   -1    modules/ksb/BuildContext.pm
M  +52   -19   modules/ksb/KDEXMLReader.pm
M  +4    -2    modules/ksb/Module.pm
A  +182  -0    modules/ksb/ModuleSet.pm
A  +222  -0    modules/ksb/ModuleSet/KDEProjects.pm
A  +33   -0    modules/ksb/ModuleSet/Null.pm

http://commits.kde.org/kdesrc-build/5d638acab096f17dd1d136868e674d36eed6f21e

Seems not fully fixed?

# GET ME ALL OF KDE SC!
module-set
    repository kde-projects
    # WORKAROUND FOR BUG 321667
    #use-modules kdesupport attica soprano kde
    use-modules kdesupport kde
    ignore-modules kdewin-installer kdewin emerge emerge-history
end module-set

module attica
  cmake-options -DATTICA_ENABLE_TESTS=TRUE
end module

kdesrc-build attica leads to an unexpected svn checkout:

# kdesrc-build running: 'svn' 'co' '--non-interactive' 'svn+ssh://svn@svn.kde.org/home/kde/trunk/KDE/attica' 'attica'

If I uncomment the workaround, it works.

That behavior was actually on purpose; you must actually mention the module you want to override to try to avoid having your module declaration magically do something different because a new framework is added to kde_projects.xml (yet another reason to use a specific keyword for this, I think).

It would be easy enough to default to always using a matching KDE project, should I try for that?

Yeah, it's a bit annoying to have to specify the module in two places in order to change its options. Not very modular in terms of writing up the kdesrc-buildrc file.

Is it also OK to use a specific keyword then? This is the time to ensure it's right before it needs to be supported til the end of time. ;)

I'd suggest a simple "options", e.g.

8<==============
module-set
    repository kde-projects
    use-modules tier1
end module-set

options kcoreaddons
    cmake-options -DFOO=BAR
end options
8<==============

The option-reading code is already module-declarator agnostic so this, at least, should not be very difficult to do.

Thoughts?

Yes! +2 for options.

It would remove the ambiguity of "module", which gives me svn checkout commands when misused :)

Thanks!

Git commit 38ea997502e18313e6fe0f7120dee9ed8544bd82 by Michael Pyne.
Committed on 11/01/2014 at 20:28.
Pushed by mpyne into branch 'master'.

rc-file: Add specific "option override" entry type.

Add a specific config file grouping (which acts just like a module
declaration), to allow for specifying options to override a
previously-declared module.

The use case here is for a module-set: You can specify options which
apply to an entire module-set when declaring the module-set, and then
override those options with any changes in a later "options"
declaration.

These declarations can stack too, so this can also be useful for
multi-level file includes (but this is less useful since an "options"
declaration requires a specific module, it doesn't work on module-sets;
in that case you'd want to have the different module-sets in your
most-specific included config files instead of in a base file).

Tested on my personal test case for bug 321883, and on a --pretend run,
and with a bug 321883 test case modified to not pre-declare the
overridden module first.

Example:
    module-set kde-mm
        repository kde-projects
        use-modules kde/kdemultimedia
    end module-set

    options kmix # Not mentioned before this line
        branch KDE/4.11
    end options

In this case kmix would use KDE/4.11 branch while juk (and the rest of
kde-mm) would use whatever the global branch or branch-group was
(probably 'master').

M  +67   -1    doc/index.docbook
M  +30   -20   modules/ksb/Application.pm

http://commits.kde.org/kdesrc-build/38ea997502e18313e6fe0f7120dee9ed8544bd82

My hero!

This is so much more readable now, being able to differ actual external modules with options for kde git modules.

"As soon as" everything compiles (ok, so not very soon yet) I'll make a blog with more details, but here's what the "compile everything" rc file looks like now:
http://www.davidfaure.fr/2014/kdesrc-buildrc

Sounds great! I've updated the website documentation for kdesrc-build (and made an apologetic note on its homepage).

If you don't mind my asking though, was it your intention to mark the bug as REOPENED?

Either way please continue to let me know how I can help. I don't get tons of time nowadays but I will keep chipping away at what I can.

Hmm can't remember why I reopened it. Sorry about that.

Git commit 26461b97d3f53c09024bc602ddaca3c904a3b043 by Michael Pyne.
Committed on 12/07/2013 at 03:41.
Pushed by ashark into branch 'docbook_historied_per_file'.

kde-projects: Don't auto-filter git-repo's direct children.

Kind of a mouthful, but remove a filter that was added to avoid
automatically including an actual git repository's children that were
also git repositories. E.g. think of something like a SuperBuild git
module that also had its component git modules as logical children
within projects.kde.org.

This filter is unnecessary now that kdesrc-build supports both kde.org
metadata (kde-build-metadata/build-script-ignore) and user-configurable
module filtering (ignore-modules option).

More importantly this filter precludes many other desirable types of
group syntax (e.g. including all of kdebindings just by asking for
kdebindings).

So we remove the filter. The trouble I've seen so far is that it is
now rather difficult to build *only* kdelibs, since kde/kdelibs is a
logical parent of nepomuk-core, nepomuk-widgets, and kactivities. This
can be worked-around by using a normal single module declaration. E.g.

    module kdelibs
        repository kde:kdelibs
    end module

On the other hand, the recently-added config file option
"ignore-modules" should now remove any kde-projects modules that contain
a path component(s) matching the ignored module. You may have to be
specific with a given ignore atom for this reason (e.g. ignore
playground/libs, not 'libs' otherwise you'll also ignore
kdegraphics/libs at this point... though that was also true before this
commit).

Original commit: a0338664
https://invent.kde.org/sdk/kdesrc-build/-/commit/a033866429b62c29fd93aea843c2f21427634ec8

M  +12   -0    doc/getting-started/kde-modules-and-selection.docbook
M  +12   -0    doc/kdesrc-buildrc/conf-options-table.docbook

https://invent.kde.org/sdk/kdesrc-build/-/commit/26461b97d3f53c09024bc602ddaca3c904a3b043

Git commit a825ec00f754dae8edc158f90405fa9b61ad611e by Michael Pyne.
Committed on 11/01/2014 at 21:06.
Pushed by ashark into branch 'docbook_historied_per_file'.

rc-file: Add specific "option override" entry type.

Add a specific config file grouping (which acts just like a module
declaration), to allow for specifying options to override a
previously-declared module.

The use case here is for a module-set: You can specify options which
apply to an entire module-set when declaring the module-set, and then
override those options with any changes in a later "options"
declaration.

These declarations can stack too, so this can also be useful for
multi-level file includes (but this is less useful since an "options"
declaration requires a specific module, it doesn't work on module-sets;
in that case you'd want to have the different module-sets in your
most-specific included config files instead of in a base file).

Tested on my personal test case for bug 321883, and on a --pretend run,
and with a bug 321883 test case modified to not pre-declare the
overridden module first.

Example:
    module-set kde-mm
        repository kde-projects
        use-modules kde/kdemultimedia
    end module-set

    options kmix # Not mentioned before this line
        branch KDE/4.11
    end options

In this case kmix would use KDE/4.11 branch while juk (and the rest of
kde-mm) would use whatever the global branch or branch-group was
(probably 'master').

Original commit: 38ea9975
https://invent.kde.org/sdk/kdesrc-build/-/commit/38ea997502e18313e6fe0f7120dee9ed8544bd82

M  +67   -1    doc/kdesrc-buildrc/kdesrc-buildrc-overview.docbook

https://invent.kde.org/sdk/kdesrc-build/-/commit/a825ec00f754dae8edc158f90405fa9b61ad611e