A Matter of Control: The State of Input Device Support in KDE

Guest Post

If you look at the various changes from KDE 3 to KDE 4, two major trends emerge: unification and abstraction. Plasma, for example, unifies the various parts of the desktop and panel. Solid provides an abstraction layer that hides the details of device management from applications, while Phonon does the same for multimedia. Akonadi does both, providing a unified system for handling PIM data and creating an abstraction layer so PIM front-ends don’t need to be concerned with the source or nature of the data they display. And of course the success of KDE 4 is not due solely to these trends, it is also due to developers sitting down and ironing out the current state of the tools in KDE, where they fail, where they work, where they should be, and how we can get them there.

Overall this strategy has proved to be wildly successful. Plasma is now working (at least partially) on everything from cell phones to finance manager applications. The move from HAL to DeviceKit will be pretty much transparent to applications thanks to Solid, and the move from Xine to VLC will be similar for multimedia applications thanks to Phonon.

However, there is one area that I think has been almost—if not entirely—neglected in this process. As you can probably guess from the title, this area is input devices. To clarify, when I say “input devices”, I mean what some would call “Human Interface Devices”. This includes physical hardware like mice, keyboards, remotes, and joysticks, as well as more software-based tools like gesture recognition and speech recognition. For the most part, the state of such devices is largely the same as it was in KDE 3.5. kdelirc has been replaced with kremotecontrol, decent touch-pad support is available, and Simon may one day provide robust speech control, but otherwise not much has changed.

KDE, while on the leading edge of most aspects of computing, has not been for these devices. Some things that have been standard with pretty much every device or desktop environment for years are unsupported in KDE. Even areas where KDE is strongest, such as keyboard shortcuts, there are a number of well-known deficiencies. Attempts have been made to implement certain features, but they have been uncoordinated, inconsistent, and redundant.

In my opinion, the issues related to this aspect of KDE deserve some serious attention, especially with work in many areas of KDE 4 now focused on polish and incremental improvements. Hopefully this article, in which I list some of the problems I see and propose a solution, will get the ball rolling on this front.

I will start by summarizing my take on the state of the major classes of input devices. The details of the problems are not important, the point is to justify my conclusion that the issue exists. At the end I will propose a single solution that I think would solve many of these issues, one that I feel is based off the successes of KDE 4. Unfortunately I am not a developer, so the proposal may or may not be feasible. But at the very least I hope this will at least trigger some serious thought and discussion on the matter.

Keyboards

I would say that the keyboard is easily the class of input device for which KDE’s support is strongest, one of only two classes that I would say actually has good support. That being said, there are a number of issues even here. For example, only one value is allowed for any global keyboard shortcut, and only two for local shortcuts. This makes sense for desktop users, but for laptop users who sometimes use an external multimedia keyboard, they have to decide between using their multimedia keys or having shortcuts that work while traveling. If you use Bluetooth remotes, which tend to be identified as keyboard buttons, this is an even more serious issue since the buttons on the remote may not correspond to those on your keyboard, meaning you can only use one or the other. And amongst the multimedia keys my version of x11 supports, KDE trunk currently cannot recognize 20 of them, about 13% of the total number.

Remotes

This is the other area where I think KDE support is good, although not as good as with keyboards. There are two primary problems I see with the kremotecontrol. First, it requires that the remote already be configured, not necessarily an easy task and one that generally requires fiddling with the command-line and editing text files. Second, it requires an implementation of a shortcut system completely independent from the keyboard shortcuts. This involves a separate and very different user interface to accomplish roughly the same task, and requires that either individual applications or kremotecontrol provide a second, redundant set of commands. Since remotes are nowhere near as common as keyboards, this has left kremotecontrol with far fewer applications it can control directly and far fewer commands it can issue. It is capable of mapping remote control button presses to D-Bus events, but this is very complicated and requires the target application be running.

Mice

KDE’s support for mice, or lack thereof, is for me its most glaring deficiency. For a device as ubiquitous and essential as mice are, they have gotten remarkably little attention from developers. One of the more obvious limitations in KDE’s mouse support is the lack of support for now almost universal back and forward buttons. You would be hard-pressed to find a mouse today, even small, budget, portable, laptop mice, that do not have these buttons. As far as I am aware every modern desktop environment supports these buttons besides KDE. Even Apple, which long resisted even the right-click, now seems to support these buttons. Many mice today, especially high-end mice, have many more buttons than this. Besides the normal three buttons+scroll, my mouse has seven extra buttons, none of which work with KDE. There is no general way to use mouse button presses or combined keyboard and mouse button presses for use as shortcuts. These sorts of things are possible to implement in KDE, rekonq supports back and forward buttons while plasma supports combined mouse/keyboard shortcuts (although only local shortcuts, not global ones), but there is no general, KDE-wide way to use these buttons, the support that does exist is only in one or two applications, and for most buttons there is no solution in KDE at all. Feature requests for better mouse button support are amongst the oldest, highest-voted, and most-duplicated wish-list items in both bugs.kde.org and the KDE brainstorm system.

Touch-Pads

KDE’s touch-pad configuration has greatly improved recently from a basic configuration standpoint. However, it suffers from the same limitations as mice, namely that there is no easy way to map touch-pad events to commands. There are four corners in a standard rectangular touch-pad, but KDE only lets you map the three mouse buttons, wasting at least one corner (or more if you have a multi-touch enabled touch-pad). There are four edges, but only two ways to scroll, meaning edge actions commonly seen in other touch-pad configurations, like back/forward and volume control, are not supported. With two directions per edge and four edges that is eight commands. With circular scrolling, there are two additional directions for each edge and each corner, giving 16 additional shortcuts. With multi-touch enabled touch-pads all of those could be used for non-mouse-related tasks. That is 28 potential commands on every touch-pad. That many is probably too much for users to remember at one time, but it is a lot of untapped capability, much of it capability that has been present in pretty much every touch-pad driver in windows for at least five years.

Joysticks

Joystick support in KDE is also pretty limited. You can calibrate your joystick, it shows the current values for each button, and it shows a trace for two axes (no matter how many axes the joystick actually has). There may be some joystick controls in some games (although I wasn’t able to find any, even for games where it makes sense). Ignoring games, however, many joysticks provide a half dozen or even a dozen buttons and often at least four or five axes, all of which could be put to good use as shortcuts for controlling your computer. In fact many joysticks come bundle with tools that let you do this in windows, and it is probably even more useful for game pads which are smaller, easier to handle, and often wireless.

Bluetooth

I already mentioned the issues with Bluetooth remotes and keyboards. Less-standard devices, like Nintendo Wii remotes, currently need separate software to interpret them, and they have to be mapped to other input devices like keyboard button presses in order to be used. There are some efforts to get native support for these sorts of devices, such as in Plasma Media Center, but once again these solutions are restricted to parts of KDE and thus will not help KDE as a whole.

Gestures

KDE has support for mouse gestures. However, the configuration of these gestures is difficult, relying on the custom shortcut system which then forces you to map the gestures to keyboard shortcuts or D-Bus commands.

Speech Recognition

Simon should provide speech recognition support, but since I have never been able to get it run, and it does not compile at all with Qt 4.7, there is not much I can say. I don’t expect the situation to be much better than with kremotecontrol, though.

A Proposal

As I mentioned at the beginning, I think unification and abstraction have proven to be extremely successful in KDE wherever they have been applied. So the questions are, can these approaches be applied to input devices, and will this make users’ and developers’ lives better? I think the answer to these questions is “yes”.

KDE already has a really good D-Bus-based system for handling input events. This is the keyboard shortcut system I mentioned earlier. It makes it very easy to assign keyboard shortcuts, and automatically provides slots for keyboard shortcuts for any toolbar button or menu entry without any additional effort by developers. However, the interface would work just as well for shortcuts involving other input devices. In fact, almost identical user interface elements are used by Plasma for mouse shortcuts.

So I think the best solution would be to build off the existing keyboard shortcut system, extending it so developers can write plug-ins that provide support for additional types of input devices. The plug-ins would send events to the shortcut system in a standard format. The shortcut system would then combine the events from all the plug-ins and pass those events along to applications. Applications would just see these generic input events, they wouldn’t need to know or care what sort of device sent them (although I suppose this information could be provided if it would be useful).

This approach has a number of advantages over the current implementations:

  1. Users don’t have to map input events to keyboard shortcuts
  2. Users don’t need to go searching for the right tool to configure a device
  3. Users don’t need to learn several different configuration interfaces
  4. Developers have a consistent software interface for all devices
  5. Application developers don’t need to change anything in their applications
  6. Developers don’t need to write a custom user interface just for their device
  7. If one developer programs support for a device the rest of KDE gets it for free
  8. Support for shortcuts that combine two devices, like mouse+keyboard shortcuts, would happen automatically
  9. Plug-ins would not need to be tied to physical hardware devices, so for example a program that triggers a particular event at a particular time would be easy to do without needing to worry about cron jobs or shell scripts
  10. Joystick support for games would happen automatically (at least for buttons)
  11. It should make it easier to keep up with changes in underlying libraries, like x11 keyboard keys
  12. It would be easy to package and distribute plug-ins
  13. It would be easier to support additional operating systems
  14. Plug-ins could be loaded only when the appropriate device is connected
  15. It could be possible to support multiple plug-ins for the same device with different features, although this may not be desirable

One objection may be that there are major differences between different input devices. For instance kremotecontrol supports profiles, which allows you to change the role of each button. However, this turns out to be a benefit of this system, rather than a drawback. It allows developers to focus all their attention solely on the things that are different about the device they are working on. Each device would still need configuration tools to configure properties that are unique to that device, but they can share configuration tools for things they have in common.

So in the case of kremotecontrol profiles, rather than having to worry about mapping the button presses in different profiles to different application events, all the developers would need to do is let the users assign whatever name they want to the buttons in each profile. The same name would be seen by the shortcut system as the same button. If you want a button to do the same thing in different profiles, give it the same name. If not, give it a different name. If you want different buttons to do the same thing, give them the same name.

However, you could even take it a step further. Being able to configure different sets of shortcuts for different situations could be very helpful, maybe even tied to activities. So the kremotecontrol developers could, if they wanted, integrate their profile system into the shortcut system as a whole. That is another advantage of this system: if there is a useful feature, all devices could benefit from it rather than just one.

There is one thing that would still need to change in the shortcut configuration, though. With all of the devices being supported in a single interface, one global shortcut and two local shortcuts per command would no longer be sufficient. Users would need to be able to set an arbitrary number of global and local shortcuts for any command. As I explained earlier, however, I think this is a worthwhile change in and of itself, so I don’t consider this to be a disadvantage.

In the end, this is just an idea about how the sorts of approaches that have worked in other parts of KDE could be applied to input devices. But this is not intended to be the end of the discussion, rather I hope it is a starting point. A sprint dedicated to input devices would be ideal, but I think any sort of focused attention on this area would be of great benefit to KDE. I think that if KDE developers put their heads together and work some of their magic they could make KDE’s input device support second to none, as they have done with many other areas.

by Todd

projects.kde.org got a facelift

Some of you might know, i am always trying to improve the experience of our KDE websites. Last improvement was Userbase (if you don’t know that yet, go and visit it!).

Now we have a new facelift. Some might have seen it, projects.kde.org is now our project management facility. But its look was a plain default redmine one.

But with some help of our artist Eugene Trounev we were able to change that. So take a look. Hope you all enjoy it and still find it usable.