I work on pretty complicated scenes, sometimes with several hundred layers. Since the only way to get the magic wand selector working correctly in my environment is to hide all layers besides the lineart and a plain white background, you can imagine how this can be extremely cumbersome.

It would be nice if you could check a box on a layer to mark it as a 'selection reference' layer or similar, and in the magic wand/similar color/magnetic wand selector tools have an option to 'limit to reference layer(s)'

if only one layer would be allowed at a time that would be a nice start, but if multiple could be chosen that would make it even better.
Or even better, integrate it into the layer colors. No new UI needed, just have it only reference layers with a certain color assigned to them.