5.1.0-prealpha (git af77a5f) But I've noticed this as far back as 4

When having any sized reference image up, if you zoom too far in while the ref is still on-screen, the FPS can drop as low as 10% of the normal canvas FPS (220 down to 22). 

To reproduce:
- Open a canvas
- paste or open a reference image of any size
- monitor canvas FPS when panning at various zoom levels on the ref image.

Things that affect the FPS directly:
- The relative resolution of the canvas docker on the screen (4k screens and full-screen mode have it bad, less noticeable on 1080p screens.)
- The percentage of the screen taken up by the reference image. (This variable can go over 100% if more than one reference image is overlaid on top of each other, try placing 5 refs in the same place and make them take up the full screen.

Things that DON'T seem to affect the FPS directly:
- Canvas zoom level
- Scale of the reference image
- resolution of the reference image (2000x2000 vs 10x10)
- Total number of refs (only percentage of screen taken up)
- Color mode or channels of ref images
- Whether they're referenced to a file or copied off clipboard
- Whether they're overlaid on or off the side of the canvas