When a vertical panel is set wide enough to allow 2 columns of system tray icons, app icons in Icons Only Task Manager become too large. On a densely populated panel with a few pinned apps, opening one or two unpinned apps causes all the icons to shrink (which is good) and shift position as they squeeze together to fit (not so good). As a result, icons are constantly shifting position back and forth as apps are opened and closed, instead of staying where the user expects them. Obviously, some shrink and shift is necessary to accommodate a large number of opened apps, but space on vertical panels in already limited and the large icon size makes the problem worse.
May I ask for some screenshots? Also, have you tried the "Always arrange tasks in columns of as many rows" option by any chance?
Created attachment 157051 [details] pinned apps only, icons full size
Created attachment 157052 [details] extra apps opened, icons shrink and shift up
Screenshots added. I prefer not to use multiple columns of icons. The size of the shrunken icons in the second screenshot are about the "correct" size for me, just a matter of personal preference.
Ok, looking at the screenshots, this is intentional. The task manager icon size always follows the size of the application launcher. If you'd like those icons to be smaller you can make the margins of the task manager bigger in your plasma theme tasks.svg file or use a different plasma theme with (slightly?) bigger margins
...Or you can surround your Task Manager widget with Margins Separators to increase its margins and therefore make the icons smaller. Beyond that, we don't plan to bring back an explicit user-configurable size setting for Task Manager icons, as when we had it in the past, and produced a never-ending stream of bug reports about size issues anyway because people would use it, get odd sizing due to interactions with other settings, and get confused.