A Matter of Control: The State of Input Device Support in KDE

Guest Post

If you look at the various changes from KDE 3 to KDE 4, two major trends emerge: unification and abstraction. Plasma, for example, unifies the various parts of the desktop and panel. Solid provides an abstraction layer that hides the details of device management from applications, while Phonon does the same for multimedia. Akonadi does both, providing a unified system for handling PIM data and creating an abstraction layer so PIM front-ends don’t need to be concerned with the source or nature of the data they display. And of course the success of KDE 4 is not due solely to these trends, it is also due to developers sitting down and ironing out the current state of the tools in KDE, where they fail, where they work, where they should be, and how we can get them there.

Overall this strategy has proved to be wildly successful. Plasma is now working (at least partially) on everything from cell phones to finance manager applications. The move from HAL to DeviceKit will be pretty much transparent to applications thanks to Solid, and the move from Xine to VLC will be similar for multimedia applications thanks to Phonon.

However, there is one area that I think has been almost—if not entirely—neglected in this process. As you can probably guess from the title, this area is input devices. To clarify, when I say “input devices”, I mean what some would call “Human Interface Devices”. This includes physical hardware like mice, keyboards, remotes, and joysticks, as well as more software-based tools like gesture recognition and speech recognition. For the most part, the state of such devices is largely the same as it was in KDE 3.5. kdelirc has been replaced with kremotecontrol, decent touch-pad support is available, and Simon may one day provide robust speech control, but otherwise not much has changed.

KDE, while on the leading edge of most aspects of computing, has not been for these devices. Some things that have been standard with pretty much every device or desktop environment for years are unsupported in KDE. Even areas where KDE is strongest, such as keyboard shortcuts, there are a number of well-known deficiencies. Attempts have been made to implement certain features, but they have been uncoordinated, inconsistent, and redundant.

In my opinion, the issues related to this aspect of KDE deserve some serious attention, especially with work in many areas of KDE 4 now focused on polish and incremental improvements. Hopefully this article, in which I list some of the problems I see and propose a solution, will get the ball rolling on this front.

I will start by summarizing my take on the state of the major classes of input devices. The details of the problems are not important, the point is to justify my conclusion that the issue exists. At the end I will propose a single solution that I think would solve many of these issues, one that I feel is based off the successes of KDE 4. Unfortunately I am not a developer, so the proposal may or may not be feasible. But at the very least I hope this will at least trigger some serious thought and discussion on the matter.

Keyboards

I would say that the keyboard is easily the class of input device for which KDE’s support is strongest, one of only two classes that I would say actually has good support. That being said, there are a number of issues even here. For example, only one value is allowed for any global keyboard shortcut, and only two for local shortcuts. This makes sense for desktop users, but for laptop users who sometimes use an external multimedia keyboard, they have to decide between using their multimedia keys or having shortcuts that work while traveling. If you use Bluetooth remotes, which tend to be identified as keyboard buttons, this is an even more serious issue since the buttons on the remote may not correspond to those on your keyboard, meaning you can only use one or the other. And amongst the multimedia keys my version of x11 supports, KDE trunk currently cannot recognize 20 of them, about 13% of the total number.

Remotes

This is the other area where I think KDE support is good, although not as good as with keyboards. There are two primary problems I see with the kremotecontrol. First, it requires that the remote already be configured, not necessarily an easy task and one that generally requires fiddling with the command-line and editing text files. Second, it requires an implementation of a shortcut system completely independent from the keyboard shortcuts. This involves a separate and very different user interface to accomplish roughly the same task, and requires that either individual applications or kremotecontrol provide a second, redundant set of commands. Since remotes are nowhere near as common as keyboards, this has left kremotecontrol with far fewer applications it can control directly and far fewer commands it can issue. It is capable of mapping remote control button presses to D-Bus events, but this is very complicated and requires the target application be running.

Mice

KDE’s support for mice, or lack thereof, is for me its most glaring deficiency. For a device as ubiquitous and essential as mice are, they have gotten remarkably little attention from developers. One of the more obvious limitations in KDE’s mouse support is the lack of support for now almost universal back and forward buttons. You would be hard-pressed to find a mouse today, even small, budget, portable, laptop mice, that do not have these buttons. As far as I am aware every modern desktop environment supports these buttons besides KDE. Even Apple, which long resisted even the right-click, now seems to support these buttons. Many mice today, especially high-end mice, have many more buttons than this. Besides the normal three buttons+scroll, my mouse has seven extra buttons, none of which work with KDE. There is no general way to use mouse button presses or combined keyboard and mouse button presses for use as shortcuts. These sorts of things are possible to implement in KDE, rekonq supports back and forward buttons while plasma supports combined mouse/keyboard shortcuts (although only local shortcuts, not global ones), but there is no general, KDE-wide way to use these buttons, the support that does exist is only in one or two applications, and for most buttons there is no solution in KDE at all. Feature requests for better mouse button support are amongst the oldest, highest-voted, and most-duplicated wish-list items in both bugs.kde.org and the KDE brainstorm system.

Touch-Pads

KDE’s touch-pad configuration has greatly improved recently from a basic configuration standpoint. However, it suffers from the same limitations as mice, namely that there is no easy way to map touch-pad events to commands. There are four corners in a standard rectangular touch-pad, but KDE only lets you map the three mouse buttons, wasting at least one corner (or more if you have a multi-touch enabled touch-pad). There are four edges, but only two ways to scroll, meaning edge actions commonly seen in other touch-pad configurations, like back/forward and volume control, are not supported. With two directions per edge and four edges that is eight commands. With circular scrolling, there are two additional directions for each edge and each corner, giving 16 additional shortcuts. With multi-touch enabled touch-pads all of those could be used for non-mouse-related tasks. That is 28 potential commands on every touch-pad. That many is probably too much for users to remember at one time, but it is a lot of untapped capability, much of it capability that has been present in pretty much every touch-pad driver in windows for at least five years.

Joysticks

Joystick support in KDE is also pretty limited. You can calibrate your joystick, it shows the current values for each button, and it shows a trace for two axes (no matter how many axes the joystick actually has). There may be some joystick controls in some games (although I wasn’t able to find any, even for games where it makes sense). Ignoring games, however, many joysticks provide a half dozen or even a dozen buttons and often at least four or five axes, all of which could be put to good use as shortcuts for controlling your computer. In fact many joysticks come bundle with tools that let you do this in windows, and it is probably even more useful for game pads which are smaller, easier to handle, and often wireless.

Bluetooth

I already mentioned the issues with Bluetooth remotes and keyboards. Less-standard devices, like Nintendo Wii remotes, currently need separate software to interpret them, and they have to be mapped to other input devices like keyboard button presses in order to be used. There are some efforts to get native support for these sorts of devices, such as in Plasma Media Center, but once again these solutions are restricted to parts of KDE and thus will not help KDE as a whole.

Gestures

KDE has support for mouse gestures. However, the configuration of these gestures is difficult, relying on the custom shortcut system which then forces you to map the gestures to keyboard shortcuts or D-Bus commands.

Speech Recognition

Simon should provide speech recognition support, but since I have never been able to get it run, and it does not compile at all with Qt 4.7, there is not much I can say. I don’t expect the situation to be much better than with kremotecontrol, though.

A Proposal

As I mentioned at the beginning, I think unification and abstraction have proven to be extremely successful in KDE wherever they have been applied. So the questions are, can these approaches be applied to input devices, and will this make users’ and developers’ lives better? I think the answer to these questions is “yes”.

KDE already has a really good D-Bus-based system for handling input events. This is the keyboard shortcut system I mentioned earlier. It makes it very easy to assign keyboard shortcuts, and automatically provides slots for keyboard shortcuts for any toolbar button or menu entry without any additional effort by developers. However, the interface would work just as well for shortcuts involving other input devices. In fact, almost identical user interface elements are used by Plasma for mouse shortcuts.

So I think the best solution would be to build off the existing keyboard shortcut system, extending it so developers can write plug-ins that provide support for additional types of input devices. The plug-ins would send events to the shortcut system in a standard format. The shortcut system would then combine the events from all the plug-ins and pass those events along to applications. Applications would just see these generic input events, they wouldn’t need to know or care what sort of device sent them (although I suppose this information could be provided if it would be useful).

This approach has a number of advantages over the current implementations:

  1. Users don’t have to map input events to keyboard shortcuts
  2. Users don’t need to go searching for the right tool to configure a device
  3. Users don’t need to learn several different configuration interfaces
  4. Developers have a consistent software interface for all devices
  5. Application developers don’t need to change anything in their applications
  6. Developers don’t need to write a custom user interface just for their device
  7. If one developer programs support for a device the rest of KDE gets it for free
  8. Support for shortcuts that combine two devices, like mouse+keyboard shortcuts, would happen automatically
  9. Plug-ins would not need to be tied to physical hardware devices, so for example a program that triggers a particular event at a particular time would be easy to do without needing to worry about cron jobs or shell scripts
  10. Joystick support for games would happen automatically (at least for buttons)
  11. It should make it easier to keep up with changes in underlying libraries, like x11 keyboard keys
  12. It would be easy to package and distribute plug-ins
  13. It would be easier to support additional operating systems
  14. Plug-ins could be loaded only when the appropriate device is connected
  15. It could be possible to support multiple plug-ins for the same device with different features, although this may not be desirable

One objection may be that there are major differences between different input devices. For instance kremotecontrol supports profiles, which allows you to change the role of each button. However, this turns out to be a benefit of this system, rather than a drawback. It allows developers to focus all their attention solely on the things that are different about the device they are working on. Each device would still need configuration tools to configure properties that are unique to that device, but they can share configuration tools for things they have in common.

So in the case of kremotecontrol profiles, rather than having to worry about mapping the button presses in different profiles to different application events, all the developers would need to do is let the users assign whatever name they want to the buttons in each profile. The same name would be seen by the shortcut system as the same button. If you want a button to do the same thing in different profiles, give it the same name. If not, give it a different name. If you want different buttons to do the same thing, give them the same name.

However, you could even take it a step further. Being able to configure different sets of shortcuts for different situations could be very helpful, maybe even tied to activities. So the kremotecontrol developers could, if they wanted, integrate their profile system into the shortcut system as a whole. That is another advantage of this system: if there is a useful feature, all devices could benefit from it rather than just one.

There is one thing that would still need to change in the shortcut configuration, though. With all of the devices being supported in a single interface, one global shortcut and two local shortcuts per command would no longer be sufficient. Users would need to be able to set an arbitrary number of global and local shortcuts for any command. As I explained earlier, however, I think this is a worthwhile change in and of itself, so I don’t consider this to be a disadvantage.

In the end, this is just an idea about how the sorts of approaches that have worked in other parts of KDE could be applied to input devices. But this is not intended to be the end of the discussion, rather I hope it is a starting point. A sprint dedicated to input devices would be ideal, but I think any sort of focused attention on this area would be of great benefit to KDE. I think that if KDE developers put their heads together and work some of their magic they could make KDE’s input device support second to none, as they have done with many other areas.

by Todd

9 thoughts on “A Matter of Control: The State of Input Device Support in KDE”

  1. >Hi!

    I think this is a good read and definitely an interesting perspective.

    Just one minor correction: simon (Git) does compile with Qt 4.7. If you have any trouble setting it up, you can contact us at any time on the sourceforge page or directly at grasch ate simon-listens dot org.

    Back on topic: I do have to agree that the input stack could be improved.
    Specifically, I would add multi touch support to your list.

    About bluetooth, I have to say that I was quite impressed with BlueDevil actually. It worked perfectly for my bluetooth keyboard and – despite the low version number – felt very polished already.

    Best regards,
    Peter

  2. >I would *love* to control my laptop's speaker volume by "scrolling" the left or top touchpad edge!

    I've never even thought of this possibility before, but now that you've mentioned it, I really want it… 😉

  3. >Great writeup. KDE has many unifying technologies like akonadi and solid and I think that input should be another part of what makes KDE software great.

    Having plugins for each input device would allow more flexibility as well. For example I have always wanted to move windows by pressing the left and right mouse buttons together but with the current system I can't.

    This input plugin system would make a great GSoC project as well. Who knows maybe I might take it up 😉

  4. >One feature of Opera I'm completely addicted to is the ability to move between tabs by holding the right mouse button and scrolling the wheel (or, in a touchpad's case, moving my finger along the right edge). So this kind of thing could definitely be useful. Mainly for advanced users, but I don't necessarily regard that as a minus point, more as a challenge to improve the discoverability of these things somehow. (Many non-computer-centric people I know don't even know that you can open links in new tabs by middle clicking them, a phenomenon which was completely baffling to me the first time I encountered it, but has since turned into a regular occurrence. They're always pleasantly surprised when they discover it, via seeing me do it.)

  5. >You speak the truth. It totally agree on your points and really like your proposal. But as for me as a pure user it makes me a bit depressed to read this because it is very likely that it will take several years until a usable system will appear.

  6. >http://code.google.com/p/wiimotedev/ – Completely and ready to use Nintendo Wiiremote daemon. Need some KDE developer that is interested in Wiiremote support 🙂

    Best regards,
    Bartlłomiej Burdukiewicz
    dev.strikeu at gmail.com

  7. >I proposed something like this a while ago but I lack the skills or reputation to get something to happened. I would really like to be part of this if I can help at all (my skills are improving)
    let me know rmsmit @ rms-mit.dhis.org

  8. >Todd_RME and I have been discussing input enhancements in other venues (mailing lists, email, and so on). He knows my counter-ideas already; this is for the benefit of other readers.

    (Todd, if any of this surprises you- send me an email.)

    KDE uses Qt toolkit for these devices. Qt, in turn, uses X11 (on Linux/Unix platforms). A switch to D-Bus would require new interfaces within or linked to D-Bus, in order to support remotely attached devices (e.g., xterm-like "dumb PCs") running applications on a Server, as in classrooms.)

    X11, using the older xinput 1.5 protocol, provides partial support for up to 31 mouse buttons already. But xinput 1.5 will NEVER handle multiple pointers within the same window; the button mask is inadequate; the KBD can hanve only 255 key codes; and new input devices (e.g. multi-touch screens) can only be added with great difficulty. Many of the mouse usage issues can be addressed easily, but the keycode limitation must (ultimately) be addressed.

    The current overall design, migrated to XI2, is a less risky implementation — although Qt would have to be responsible to translating the new X11 protocol elements into the existing Qt API when using the XI2 interface.

    Unfortunately, although both levels are Xinput are always present in modern versions of X11, a single application (i.e. Qt) needs to use the interface of one OR the other. Not both. Qt must translate the input events into a single API for KDE, and that API must be BC through the KDE 4.x series.

    Big job. 🙁

Comments are closed.