The neo2 layout provides arrow keys on a fourth layer (see the image at http://www.neo-layout.org/ -> "Ebene 4"). Using those does work perfectly fine in general and specifically in normal Kate's vi mode's normal mode. However, it doesn't work at all in visual mode. That's because when in visual mode the modifier key (mod4) needed for layer 4 is pressed, Kate deletes the currently selected text, and leaves visual mode. xev gives the following output when pressing the modifier key:
KeyPress event, serial 35, synthetic NO, window 0xc00001,
    root 0xac, subw 0x0, time 23763779, (256,358), root:(1218,381),
    state 0x0, keycode 108 (keysym 0xfe11, ISO_Level5_Shift), same_screen YES,
    XKeysymToKeycode returns keycode: 94
    XLookupString gives 0 bytes: 
    XmbLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyRelease event, serial 35, synthetic NO, window 0xc00001,
    root 0xac, subw 0x0, time 23763882, (256,358), root:(1218,381),
    state 0x20, keycode 108 (keysym 0xfe11, ISO_Level5_Shift), same_screen YES,
    XKeysymToKeycode returns keycode: 94
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False


Reproducible: Always

Steps to Reproduce:
1. Use neo2 layout (setxkbmap de neo on most modern linux distributions)
2. Enter some text in kate
3. enter visual mode
4. press the mod4 key (on most keyboards it is AltGr)
Actual Results:  
The currently selected text is deleted.

Expected Results:  
No text is deleted. Using the arrow keys on layer 4 is possible.