I've got an experimental implementation of an InputMethod on Android to inject key-events from desktop-to-mobile. After some testing and improvements it seems to work pretty well for printable keys and some selected special keys (like Backspace/Arrow-keys/Return etc) and I wanted to ask your opinion on some points, Aleix and Albert (especially because I'm not an Android programmer -- yet ;-):

Some misc. remarks/questions:

* InputMethod and activation:

For now the InputMethodService is baked into the MousePadPlugin. As soon as installed it is visible and selectable from the IME list in the Android settings.

Once selected it remains the active InputMethod until it is deselected again by the user or kdeconnect is uninstalled. That means, it remains active also if not currently paired (with a remote device that supports the plugin).

For me this behaviour was fine, but I'm not sure if it should remain like that. Depending on what is possible on android we could consider to ...

... De/Activate InputMethod automatically if not paired (if possible)?
... De/activate it based on capabilities of the paired device

For now keys are only accepted when the keyboard is visible (InputMethodService.onStartInputView()). I think about adding a setting to enable accepting keys always (e.g. for selecting and starting apps on the home screen with arrow-/return-keys.

On the visual side the keyboard only contains one row with a settings- and a keyboard-button for entering directly the settings dialog and the keyboard selection-dialog, resp. I think about adding a isPaired indicator and maybe a Return and a Backspace key.

* On-the-wire format:

I experimented a bit with the structure of the NetworkPackage payload (type "kdeconnect.keyboard"). Mere transferring of printable characters turned out to be insufficient, because it would become hard to handle special keys like backspace, Shift, etc. Therefore for now the payload basically contains a QKeyEvent serialized to JSON. This carries all information necessary to handle special and modifier keys. The Qt keycodes are then mapped to KeyEvent.KEYCODE_XXX values and some selected keys are handled in a special way (like Backspace, Left, Shift-Left, Home, etc.) and result in KeyEvent sequences on the android side.

One could also think about using a (single-)char utf8 payload for the keyevents, mapping (selected) special keys to (corresponding) unicode-values. But this makes serialization on desktop-side more complex, and I'd prefer to stick to the QKeyEvent way.

If a keyevent was successfully delivered to the InputMethod it is reported back to the remote keyboard (if requested), to allow for visualizing what arrived on the other side.

It might be useful to add additional packages to signal keyboard-activation and disable typing on a remote keyboard if it is not possible to deliver anything or/and error-reporting to notify a remote typer why his keys did not reach their destination.

* Plasmoid:

To the plasmoid I added a TextField for inputting the remote keys and a second readonly one that echoes the keys confirmed from the remote side. I'm not yet completely happy with that. Ideally I think of a single TextField (or TextInput) not echoing keypresses immediately but only after having been ack-ed by the remote device, thus the user can see immediately what he typed remotely. But that would mean to interpret the echoed kepress-event equivalently to what is done on the remote side. I tried to manually inject a QKeyPress via postEvent directly to the TextField but that was not evaluated by the TextField, dunno why.

Also added a command-line flag to kdeconnect-cli for sending keypresses (interactively) to a remote.

* Plugin:

For my POC everything lives inside the MousepadPlugin on both sides. As you already factored out the sendnotifications I wonder if I should add a new plugin for this direction of the remote-keyboard feature as well?

* More possibilities: Add support for defining special keys on the smartphone (e.g. F1-F10) and make them execute special actions when pressed on the remote keyboard (Press menu button, start app X, ...)

* Finally: Would you be interested in accepting it in KDE Connect?

Please let me know what you think about it. After these first tests it looks promising and might be a nice extension especially for computer people that hate typing on an on-screen keyboard, like me :-P

Thanks, regards,
  Holger

(In reply to Holger Kaelberer from comment #1)

> * On-the-wire format:

Oups, I just realized that I forgot doing the most obvious thing: look at how keys are serialized in the mousepad plugin for mobile-to-desktop key events. I did now and found, that what you used could also serve my purposes, if I map the QKeyboardEvent.key to the specialKey property as used in a PACKAGE_TYPE_MOUSEPAD_REQUEST.

I think this is a needed addition to kdeconnect. About the format of the packets, ideally it should be the same as the one used for desktop to phone communication. This way, it will work desktop to desktop (eg:sending keys to a Raspberry from my main desktop) and phone to phone too. Splitting it in separate plugins makes sense from

[continues previous comment] makes sense for capabilities to be able to disable the part that is actually not available without disabling other features that do work (that's what happens if we put more than one feature in the same plugin). 

Sent from a phone, excuse typos!

Oh, and about the UI... I'm not sure what's the best way to present it to the user on the desktop, but the text field in the plasmoid probably works. About the phone side, registering it as a keyboard is the only choice available that I'm aware of. I'm not sure if it's even possible to make it work when the keyboard is not shown, but that would be even better.

Thanks for your feedback.

(In reply to Albert Vaca from comment #3)
> I think this is a needed addition to kdeconnect. About the format of the
> packets, ideally it should be the same as the one used for desktop to phone
> communication. This way, it will work desktop to desktop (eg:sending keys to
> a Raspberry from my main desktop) and phone to phone too. Splitting it in
> separate plugins makes sense

Ok, then I will

1. split the mousepad-plugin into mousepad and remotekeyboard (new; mobile-to-desktop) and

2. add the desktop-to-mobile remote-keyboard part to the (newly created) remotekeyboard plugin.

Right?

mobile-to-desktop and desktop-to-mobile should be in different plugins (like in the case of notifications), because they are not "symmetrical": a device can support only receiving keys but not sending keys, so we want to enable/disable them separately. Sorry if that was not clear in my previous message.

Ideally, as you said, keyboard and mouse should also be different plugins, but don't worry about splitting them now if you don't want: On the Android side they are quite coupled so it might be a bit involved to separate them.

Ok, got it. That makes it easier.

Give me some time to clean things up and prepare a reviewable version ...

Awesome :)

I'm happy to read about your plans - thank you!

For a functionality that lets me control the phone on my desktop (the best would be just "mirror" one Android app into a window on my desktop) another bug report would be better, right? Or would that fit into here, too?

(Background: using Android applications like single-device Telegram with encrypted chats on the desktop to not always switch between devices)

kde: https://git.reviewboard.kde.org/r/129727/
Android: https://git.reviewboard.kde.org/r/129728/

(In reply to OlafLostViking from comment #10)
> For a functionality that lets me control the phone on my desktop (the best
> would be just "mirror" one Android app into a window on my desktop) another
> bug report would be better, right? Or would that fit into here, too?

Sounds like a new (and different) feature, like a vnc connection or so. ;-) Not sure what exists in the Android world for such use-cases.

Git commit 040ad7357b5a630f2fffface6a29d1be5103c663 by Aleix Pol, on behalf of Holger Kaelberer.
Committed on 10/01/2017 at 20:12.
Pushed by apol into branch 'master'.

kdeconnect-kde: Add remotekeyboard plugin

Allow to inject keypress events to remote peers (most notably Android devices)

Notes / open issues / possible improvements:

- For the json-payload I used the syntax of the key-events as sent by mousepad-plugin with the addition of a "sendAck"-flag. If "sendAck" is set to true the remote peer should echo a key-event if it could be handled, thus allowing the local client to find out whether the key was accepted. For performance reasons, it's allowed to send multi-char strings in the "key" property (performs much better if you send a whole string via "echo '...' |  kdeconnect-cli ..." e.g.)

- kdeconnect-cli: For now takes a string and transforms it into single key-events for visible characters only. In a first implementation I used a kbhit() helper that used termios.h to catch and relay keypresses interactively (including some special-events), which was not optimal. A better approch might be to use linux input-api directly. Would this be an option regarding cross-platform compatibility or can I assume to develop for Linux only? Being a command-line guy, I'd really like to have a fully featured kdeconnect-cli interface ;-)

- Factor out the Qt::Key-to-internal keymap to some core-helper because it corresponds to the mapping in the mousepad-plugin?

- The plasmoid is not perfect as it is: A single line containing a non-echoing TextField (i.e. it eats all the KeyPress events), and only ack-ed keypress-packets from the peer device are injected if they contain visible keys. Advantage: the user sees whether his key-presses are accepted by the peer device. Disadvantage: The echoed text does not correspond 1:1 to what is shown on the peer's display, user might be confused when typing without success. I played around with different variations each of which with its proper shortcomings:
1. An echoing Textfield for typing: Has the advantage that the user can directly see what he is typing, which makes interaction in the typing field easier, BUT messes up interaction if the Editor on the peer is changed silently and does not notify the user if his keypresses are not handled by the peer.
2. A non-echoing TextField for typing PLUS a readonly one for printing visible echoed keys. Disadvantage: same as for the previous one and uses more space on the plasmoid.
Comments? Ideas?

REVIEW: 129727

M  +20   -0    cli/kdeconnect-cli.cpp
M  +1    -0    interfaces/CMakeLists.txt
M  +7    -0    interfaces/dbusinterfaces.cpp
M  +10   -0    interfaces/dbusinterfaces.h
M  +9    -0    plasmoid/declarativeplugin/kdeconnectdeclarativeplugin.cpp
M  +60   -0    plasmoid/package/contents/ui/DeviceDelegate.qml
A  +71   -0    plasmoid/package/contents/ui/RemoteKeyboard.qml     [License: GPL (v2/3)]
M  +1    -0    plugins/CMakeLists.txt
M  +1    -1    plugins/mousepad/kdeconnect_mousepad.json
A  +8    -0    plugins/remotekeyboard/CMakeLists.txt
A  +23   -0    plugins/remotekeyboard/README
A  +28   -0    plugins/remotekeyboard/kdeconnect_remotekeyboard.json
A  +144  -0    plugins/remotekeyboard/remotekeyboardplugin.cpp     [License: GPL (v2/3)]
A  +65   -0    plugins/remotekeyboard/remotekeyboardplugin.h     [License: GPL (v2/3)]

https://commits.kde.org/kdeconnect-kde/040ad7357b5a630f2fffface6a29d1be5103c663

Comment from comment #13 was reverted, reopening.

It has been implemented, will be part of the next release.

*** Bug 367877 has been marked as a duplicate of this bug. ***