SUMMARY
Hovering over a grouped task icon slows down the tasks plasmoid temporarily 

STEPS TO REPRODUCE
Let's assume the following configuration on the icon-only task manager. However, the issue also occurs in the normal task manager:

Appearance -> Show tooltips: false
Appearance -> Highlight windows when hovering over tasks: false
Appearance -> Mark applications that play audio: false

Max rows: 2

Behavior -> Group: by program name
Behavior -> Show only tasks -> From current activity: true

Now comes the issue. Bear in my mind however, it's a bit difficult to describe:
Let's say I start several different programs, each of those get their icon in the task list. Let's say I then start 15 konsole windows. Therefore, those konsole windows will be grouped.

If I click on the konsole icons, I experience a lag before the group popup menu shows. When I start hovering over the items, I at first also experience a delay before the first entry gets highlighted. After that however, there is no lag. All other elements will be highlighted (i. e. the background of the icon changes...). If howver I close the group popup and then I open it again, I experience the same issue.

In fact, I get the delay before I even click the grouped icon. Just hovering over the konsole group icon will cause a delay, and if I move my mouse to other icons, there will be a delay before they get highlighted. So, hovering over a grouped task seems to be triggering something that slows the plasmoid.

In short, the more grouped windows, the more delay when I hover over the icon. With just a few, this isn't noticable. 

I hope it's clear enough to understand what I mean. I could maybe also create a short video if necessary.

POSSIBLE SOLUTION

Anyway, I tried to get to the bottom of this and I believe I found something in /usr/share/plasma/plasmoids/org.kde.plasma.taskmanager/contents/ui/Task.qml.

If inside PlasmaCore.ToolTipArea, in onContainsMouseChanged, I simply insert a return at the beginning (so it does nothing), the issue is gone. 

In fact, the line "toolTipDelegate.rootIndex = tasksModel.makeModelIndex(itemIndex, -1);" appears to be the actual problem. If I comment it out, I don't experience this issue. I now have more than 60 instances of konsole open and experience none of the delays mentioned above.

Keep in mind, that as mentioned I have tooltips disabled, so I am not sure whether ToolTipArea should even be involved here. Nevertheless, seems the actual delay is caused by tasksModel.makeModelIndex().


SOFTWARE/OS VERSIONS

I created a new user profile with a fresh config in kubuntu 20.10, and also briefly reproduced it in a voidlinux with Plasma 5.20.3.