SUMMARY
When panel opacity is set to "adaptive", and desktop 1 has a window touching the panel (panel is opaque), desktop has no window touching the panel (panel is translucent), then when switching between these desktops, Icon Task Manager updates about PlasmaCore.Units.longDuration (200ms) later after the desktop switch.

Worse, when both desktops have a window touching the panel (panel is opaque in both desktops), task manager updates about 2*PlasmaCore.Units.longDuration (400ms) later after the desktop switch.

When both desktops have no window touching the panel (panel is translucent in both desktops), there's no such delay.

Also, when panel opacity is set to "opaque" or "translucent", there's no such delay.

STEPS TO REPRODUCE
1. Edit panel, set opacity to "adaptive".
2. Create 2 virtual desktops.
3. In desktop 1, maximize a window.
4. In desktop 2, minimize all windows.
5. Switch between desktops. Notice the delay between desktop switching and task manager updating.
6. In desktop 2, maximize a window.
7. Switch between desktops. Notice the even longer delay between desktop switching and task manager updating.
8. In both desktops, minimize all windows.
9. Switch between desktops. Notice no such delay.
10. Set panel opacity to "opaque" or "translucent".
11. Repeat step 1-9. Notice no such delay.

EXPECTED RESULT
Task manager updates the same time as virtual desktop switches, regardless of panel opacity setting.

SOFTWARE/OS VERSIONS
Linux/KDE Plasma: 
(available in About System)
KDE Plasma Version: 5.27.1
KDE Frameworks Version: 5.103.0
Qt Version: 5.15.8

ADDITIONAL INFORMATION
The relevent code:
plasma-desktop/desktoppackage/contents/views/Panel.qml
line 120, line 185

Seems that:
1. Task Manager updates after the opacity animation, thus the delay.
2. When panel is opaque in both desktops, the opacity animation somehow runs twice (although not visible), thus twice the delay.

It can be fixed by letting Task Manager (and other applets) updates before the opacity animation, but I don't know how.