Simply bumping the table your mouse is on will reset the timer. I think the sensitivity should be cranked down quite a bit. The timer should only reset if the mouse is move an inch or so.
I am also running into this issue, but its a bit more severe. For some reason my mouse (USB) always sets off the activity timer even when not moving. It doesn't stop by machine from sleeping and I have confirmed its the mouse by unplugging it during the countdown. I have been able to workaround this by un-checking "flash on activity", but it would be nice to be able to have more control here (like dialing in the sensitivity as poperigby@mailbox.org suggests and/or disabled activity tracking (while keeping keyboard) all together. Possibly by settings mouse sensitivity to zero.
that issue is more or less unsolvable at the rsibreak level: it uses the kidletime library, which uses whatever the windowing system reports to it. so basically, you need to file feature requests against xorg/xserver and potentially numerous wayland compositors. rsibreak can't track the events itself, as security measures against key-logging (are supposed to) prevent that. though it might be possible to configure a special authorization - i didn't check what options the display servers offer nowadays. that would also have the side effect of potentially fixing #392679.