“Command Not Found” Plasma Runner

As promised, an explanation.

This is the “Command Not Found” runner. For those of you who may not be familiar, command-not-found is a magical little command line tool that gets activated when the shell cannot find the binary you wish to launch. It has a database of binaries and package names, and can check to see which package contains the binary you tried to type. If it finds one, it tells you which package to install to get this binary.

But I find that I use KRunner for launching my stuff more often these days. On new installs, I instinctually go to KRunner and try to type in the name of my favorite app.  But then I get nothing; realizing that this is a new install and I have yet to install this piece of software. I mumble, backspace, and type in the name of my favorite package manager. I then type in the name of the application I wanted (again) and mark it for installation.

…but wouldn’t it be nice if I didn’t have to? I don’t anymore, thanks to the Command Not Found runner. I was browsing the KDE brainstorm section of the KDE forums and saw this idea. I thought it was neat and actually feasible, so yesterday I sat down and hacked. By the end of the day, the installer/command not found runner was born!

Here’s how it works: Whenever KRunner cannot find an application via KService, the Command Not Found runner takes what you typed and tells the command-not-found utility to find which package has that program. If command-not-found finds anything, the runner returns a result. Click that result, and it starts qapt-batch, telling it to install that package. You just have to enter your password, and qapt-batch will do all the rest. Neat!

As an added bonus, if the runner detects that the program you are trying to run has a different name than the package it is in, it gives a nice, helpful little subtext that explains that the amarok package contains “amarokpkg”, even though the package isn’t named “amarokpkg”.

A few notes based on the comments my mysterious post last night introduced:

• This is not meant to replace traditional package management. It is just a little convenience feature that can only ever install things that aren’t installed. It will never do anything more than that package management-wise.
• Going back to the first point, it is (in general) a good thing that you have to know the exact name of the program to get this to show up. It limits the usecase of this service to those who forgot that they hadn’t installed $app yet, and will have a very small likelihood of showing up for somebody who is not trying to run $app. There would really be no reason to not enable this by default, especially since KRunner is somewhat of a hidden feature in the first place…
• The KRunner plugin can be turned off.
• It does not do any package management itself. It leaves the installation to a real batch installer program.
• If the user does not have the privileges to install packages, it will not show up.

At the moment, it only works on Debian-based systems, since it requires the Debian command-not-found program. Sorry ’bout that. I just don’t see a way that this could be done in a cross-distro manner right now… It was simple to code, however, so it shouldn’t be too much trouble to make one for another distro, if you know how their command-not-found works (if they have one, of course)

Going back to the KDE Brainstorm post, it is definitely feasible to add this to Kickoff. It does require a one-line patch, however, so that pretty much limits it to people who want to patch kickoff themselves to get the runner to show results there… It would be neat if we could make certain “categories” of runners, and then have kickoff configured to only show the results of certain categories. At the moment there’s a whitelist that you have to add to before they’ll show.

I’ll try to get packages up in the next few days, but due to my impending vacation I don’t know how successful I’ll be in actually getting around to it. You can grab the source here if you want to. It should just need kdelibs5-dev to build.

QApt and Muon 0.4 (1.0 beta)

Two weeks after alpha 2, the betas of both QApt and Muon are released. Both are now feature and string frozen for 1.0, and the focus for this release and all releases on towards 1.0 (scheduled for 2 weeks from now) will focus on fixing bugs as they are found. Here’s what has changed:

QApt 0.4.0:

• A minor optimization in PackagePrivate::searchPkgFileIter(). Should speed up the generation of the changelog url for Package::changelogUrl()
• Minor optimizations in Package::name(), Package::section(), and Package::component() by constructing less unnecessary temporary QStrings. Shouldn’t be too noticeable, but hey.
• Fixed a crash that occurred when a standards-non-compliant package that lacked a package section was present in the system. (Bug 245177)
• Reduce RAM usage (heap) by 1.3 MiB by not caching the default candidate version in the Package object. It is cheap to calculate it on-the-fly and does not justify the amount of memory caching it was taking up.
• Add the Backend::packageForFile() function that enables looking up a Package by a file it installs
• Fix another encoding bug in the QApt Worker caused by not explicitly converting from UTF-8. It should be the last of those, though.
• For qapt-batch, use a KMessageBox::detailedError for presenting our queued errors, since the text isn’t selectable in KMessageBox::errorList.
• Fix a bug in the QApt Worker where it would not report valid progress. (Applications would have blank progress bars) This happened because when I added support for handling fractional percentages (e.g. 60.5% -> 61%), I accidentally made it so that normal percentages would not be handled, making operations reporting fractional percentages be the only ones that would work.

Muon 0.4.0

• Fix a bug that allowed the items in the Filters sidebar to be editable, due to questionable defaults for QStandardModelItem.
• Do not report an error if the user cancels the opening of a markings file.
• Improve the vertical centering of the “Status” and “Requested” columns of the PackageView. Now it’ll be centered for all font sizes, rather than just my own.
• Use a KMessageBox::detailedError for presenting our queued errors, since the text isn’t selectable in KMessageBox::errorList.
• Let software-properties-kde handle reloading the package lists after it edits them, since you already have to give your password to open it, and having to give it again to reload inside muon is annoying.

Packages are building in the usual place and should be available within the day. Sources for QApt and Muon can be found here and here, respectively. A new snapshot of libdebconf-kde should also be available, as Daniel has made a few nice improvements since the first snapshot.

Things are shaping up for release. I’m quite happy. I am leaving this Friday for vacation, and will return August the 11th. I will, however, try to find the time to do a release candidate for QApt/Muon a week from today. A week after that, 1.0 final will be released, so please, test it and file those bug reports!

I’ll blog about my cryptic blog last night in a bit, so stay tuned.

Whatever might this be? (Return of the Whatever)

Mysteriously introducing new Plasma-based technologies…. it’s what we do. (Click shinies for bigger)

…and I’ll leave you with that.

Oh, and I’m planning the QApt/Muon 1.0 beta release for tomorrow. (Well, technically today, as it is just past midnight. Probably within twelve hours.) But I need some sleep now. I’ll also probably be nice and fill in a bit more about the mysterious contents of this blog post.

QApt and Muon 0.3 (1.0 alpha2)

I’m proud to announce the second alpha of both LibQApt and Muon. (For an intro to those who may be seeing this for the first time, read my previous blog post) Packages will be in the same location as before, but are not yet ready. It seems that the PPA service builders have a backlog on the magnitude of 1956 jobs, and won’t get to LibQApt or Muon for another 48 hours.

Tarballs can be found here and here, for QApt and Muon respectively.

Bugs are now being tracked at bugs.kde.org (as opposed to in blog comments ). To report a bug in either LibQApt or Muon, you can now use the “Help -> Report bug” menu item in Muon just like any other KDE application.

Here’s what’s changed since 0.2 (1.0 alpha 1):

QApt

• Fixed a bug where entries like “Replaces: libqapt (< 0.3.0-0ubuntu1)” would get cut off at the ‘(‘, due to the interpretation of ‘<’ as the beginning of an HTML tag.
• Remove intermediary non-file paths from the installedFilesList() function. e.g if the file list were “/usr/, /usr/bin/, /usr/bin/konversation”, it would now be just “/usr/bin/konversation”
• Miscellaneous Krazy compliance fixes
• Fixed the appearance of items in the Depends tab that were localized by APT so that they don’t look like garbage. (Convert explicitly from Utf8)
• Added a downloadSize() function to determine how large the potential download size will be.
• Added an originLabel() function to get the human-readable name of a package archive. (PPAs, etc)
• Add support for finding the internal origin name, given its human-readable name
• Change the return type of Package::installSize() to be a qint64, since it easily can outgrow an int.
• Fix the QApt Worker not downloading package description translation files. Starting the worker as root via DBus does not set the locale env vars to the system setting, but rather to “C” (unlocalized), which causes the translations not to be downloaded
• Add version override support to QApt::Package
• Add a saveSelections() file and readSelections() file to support saving/loading package markings to file, compatible with Synaptic markings files.
• Add an updateXapianIndex() function to the backend and worker, so that apps can update the apt-xapian-index when necessary
• For the QApt Batch installer, queue all fetch warnings and commit errors to be displayed in one dialog at the end of an operation. This fixes potential massive failures causing many error dialogs to appear during operations.

Muon

• Fix the spacing of items in the status bar.
• Set the bug address to bugs.kde.org.
• Miscellaneous Krazy fixes.
• Add a menu item in the “Settings” menu to open software-properties-kde for software sources editing capabilities:

• Fix the runtime warning “QCoreApplication::postEvent: Unexpected null receiver” from popping up on return from downloading/committing something.
• Add support for showing reverse dependencies in the Dependencies tab.

• Fix incorrect alphabetical sorting after changing a filter.
• Show the download/install sizes of the currently-marked changes.

• Add “Filter by origin” support, to show packages only from certain archives/PPAs, like Software Center does:

• Save custom custom splitter sizes for the size of the filter box and the package view.
• Add a versions tab for packages that have multiple available versions. From this tab you can force a specific version. I have not yet implemented version locking, but it is planned for a future release.

• Add entries to the “File” menu to save and load package markings. It uses LibQApt’s functionality for this, and both loading/saving are Synaptic compatible.
• If there are unapplied changes when the user tries to quit, Muon will now ask if they want to save these changes to a markings file. It comes with a “never ask again” checkbox for those who don’t want to be nagged when this happens.
• On startup, check if the apt-xapian-index needs updating, and update it if it needs to be updated.
• Queue all fetch warnings and commit errors to be displayed in one dialog at the end of an operation. This fixes potential massive failures causing many error dialogs to appear during operations.
• Fix a bug where the “All” item in the categories filterbox was not at the top in localized environments.
• Plug a teeny memory leak that could cause viewing screenshots and changelogs to accumulate memory over time.

Roadmap for 1.0

With this, I am hereby announcing that feature freeze and string freeze are occurring tomorrow for both QApt and Muon. I plan on doing a beta release in two weeks, followed by a release candidate the week after that, with a final release on August 12th, just in time for feature freeze for Kubuntu 10.10. I must once again stress that this will not be the default package manager for Kubuntu 10.10, but a stable release needs to be out there to get a maximal amount of testing for 11.04, should the Kubuntu Council and development community choose this as the default package manager for 11.04.

For beyond 1.0, I plan to implement version locking as well as a configuration dialog for various APT system preferences, such as whether or not to treat Recommended or Suggested packages as dependencies or not, etc. It would be nice to eventually implement software sources configuration in Muon itself, and not have to rely on launching software-properties-kde. This is not an exhaustive list, but it’s what I’ve thought of so far.

Introducing QApt and the Muon Package Manager

It has been a while since I’ve blogged. I know I said I would post stuff  about something exciting at UDS, but that sort of never happened.

But! Better late than never. I think that waiting a month or so has also made this post better than it would have been at UDS. Before I do any introducing, I feel that a summary of the history of KDE/Debian package management is in order.

A Brief History of Debian-based Package Management in KDE

It can be said that the hardest part of writing a package manager is not in writing the GUI, but in interfacing with the package system. For a long time, this has been the biggest barrier that has limited the number of Qt/KDE package manager for Debian-based systems. Even among those that do exist, such as Adept and PackageKit, each interfaces with your computer’s underlying package system differently, each with its own set of bugs and unique, not-always-equal feature set.

These factors combined have caused a drought in the number of Qt/KDE packaging systems from the earliest days. When these package managers were good (like Adept 2 in KDE3 days), things were generally fine, and the drought had a limited effect. The transition to KDE4 threw package management GUIs right back to square one, and even two years later there has not been a replacement of the caliber of the KDE3 package managers.

But what if there were a way to alleviate both of the factors contributing to the lack of stellar package managers? What if all Qt package managers for Debianish systems could share one underlying library to interface with the system package cache? What would happen if the success of a package manager was not limited by the difficulty of writing a low-level package management backend, but rather was solely dependent on the quality of the user interface?

Introducing QApt

QApt is the answer to the above question. I had pondered the above questions for quite a while, starting back with the inception of the Project Timelord initiative.

A bit later, the Shaman project was announced, and I was interesting in writing a backend for Shaman. There was already an APT backend for Shaman, though it was in its infancy. That’s when I realized that libapt-pkg was an ancient dinosaur with an awful API. That’s when it dawned on me that I could write a library/backend that any Qt or KDE application could use to manipulate/read the Debian package cache.  If a focus on this potential library could be made, then any Qt or KDE application could use it, from Shaman to a “Software Center” style app to a completely custom, fully-featured package manager.  Developers would be free to focus on making a great GUI, and could all share the same internals.

I started three months ago by spending a few days trying to grok the libapt-pkg API by looking at the internals of Synaptic, the APTcc PackageKit backend, and the python-apt bindings. After a month or so of hacking, the result was QApt. Then came finals season and UDS…

But what is QApt? QApt is actually split into several parts. Half of QApt is LibQApt, which is a Qt wrapper around libapt-pkg that boasts a saner API, as well as a pretty extensive APT cache implementation for fetching information about packages. This constitutes the read-only portion of QApt. The second half, the QAptWorker, is equally important. The QAptWorker is what does all the dirty “needs admin” stuff, and could be considered the “write-only” portion of QApt. It handles everything from checking for updates, doing the package downloads, and actually installing/removing packages. LibQApt interfaces with the QAptWorker over Polkit, giving all the normal security and sysadmin-flexibility of the PolicyKit framework. Apps using LibQApt do not have to run as root, and sysadmins can exercise fine-grained control over which operations they let their users perform.

In addition, QApt ships with a simple batch installer utility called “qapt-batch”. It is a drop-in replacement for install-package, which is a utility that several Kubuntu applications use to install packages from non-packagemanager applications. (E.g. codec install, the firefox installer, apturl-kde, etc) It already has a few neat features that install-package does not, including more informative error handling, security improvements (warn about unsigned packages, etc) and media change support.

For further technical details, you can see the slides from the UDS presentation on LibQApt that I did here, and the neat little diagram I made with umbrello here.

What uses QApt?

At the moment, two external projects use it. There is an infantile Shaman backend based on QApt in the shaman svn branch that can do your basic things (install, remove, check for updates, error handling, etc).  Check out playground/libs/libqapt, install, and you should be able to build the shaman backend.

In addition, I wrote my own package manager, Muon, which I shall spend the rest of this post discussing.

Introducing Muon

Muon is a package manager based on QApt that I started to write about two weeks ago, when I felt that LibQApt was extensive enough to be able to write a package manager with. It follows in the current KDE naming fad of naming your app after a subatomic particle. The purpose of Muon is to cater to the usecase ranging from “user who knows a fair bit about how his computer works but really isn’t a terminal-lovin’ power user” to the “poweruser” usecase. Its goal is to have all the power of the end-all of all Debian-based package managers, Synaptic, while at the same time presenting this functionality in a more usable fashion than Synaptic. It’s never going to be a “Software Center” in terms of  ”Your non-techie family member could use it”, but that’s OK; thanks to QApt, writing a Software Center-style app would be just a matter of writing another GUI shell. Seeing the amount of progress I made in two weeks, this is definitely not something to worry about. (I also plan to write a Software Center/Adept Installer style app once Muon has a stable release)

Currently, Muon sports a quite spiffy feature list. (Especially for a two-week old package manager!)

• A powerful, yet intuitive interface
• Fast, accurate package search using the apt-xapian index and the Synaptic search algorithm
• Support for filtering packages by status and category
• Media change support
• Support for configuring packages through the debconf system
• Warns about/disallows the installation of untrusted packages, depending on APT settings
• Uses Polkit for running privileged actions for enhanced security, convenience, and desktop integration
• Power management suspension during package downloads, installations and removals
• Support for download the latest changelog of a package
• Package screenshots

But, talk is cheap. Bring on the screenshot tour!

Here we see the initial view of Muon. (Click all screenies to enlarge.) Pretty straightforward. Toolbar on the top, filters to the left, search bar/package view to the right.

Here I’ve searched for Konversation and clicked the first result. A details tab bar popped up as a result. The first tab houses the short description, a button to fetch a screenshot, buttons for marking konversation for removal, purge, or reinstallation, the long description, and finally a short blurb telling how much longer Canonical will support konversation. (Since it’s in main)

Unfortunately, screenshots.debian.net lacks a screenshot for Konversation.

Dolphin has one, though.

I couldn’t fit all the details in the first tab, but the second tab has a bunch of neat technical details.

Here we can see Konversation’s dependencies. For packages that provide virtual packages, you can also view those packages by selecting “Virtual packages provided” from the combobox. But Konversation doesn’t provide any virtual packages, so this screenshot can’t show you.

The changelog tab. (Hey, that’s me! )

Since I already have Konversation installed, I can see what files the package has.

Filterting by category works.

You can filter by category and by status at the same time. In this picture, I am filtering on installed KDE packages.

After I marked amor for installation, I hit the  ”preview changes” button in the toolbar, which graciously shows all packages to be installed/removed, and turns the preview button into a “back” button.

Gotta give my password before I can hit apply. (Polkit remembers the password for a few minutes, so you can install/remove something again directly after this install without being annoyed by the dialog again.)

Downloading. This could stand some improvement, (Per item progress, get rid of the URIs, etc) but is functional.

Installing amor. It happened so fast that it was already cleaning up by the time ksnapshot could get a picture.

If the need ever arises, a debconf GUI shows up. Please don’t mind the Debian branding. That should change before any stable release. Muon uses the Debconf KDE Library written by dantti, which should hopefully also be usable in the next version of KPackageKit.

Untrusted packages, while usually just a matter of forgetting to add a GPG key for a PPA, are a security concern. (E.g. DNS Cache poisoning attacks…) It does follow APT configuration, though, and will totally disallow installing unsigned packages if that is set in /etc/apt/

Muon also asks if you really want to remove that essential package

..and also prevents you from making changes that would break the package cache, like any good package manager should. (It undoes the attempted change after you press OK)

For all its features, Muon and QApt are not yet, in my opinion, ready. They still need support for showing reverse-depends, package downgrade and pinning support, and in general just need some more polishing and bughunting. Muon still is only two weeks old, you know. For this reason, I’m going to say right now that this will not be the default package manager for Kubuntu 10.10.  Every single time we’ve jumped on the latest-n’-greatest piece of package management shininess, it’s never ended well. Adept 3 was decent in my opinion, aside from search accuracy and a somewhat scattered UI, but it was rushed to be made ready for release and really never stood up to the quality that Adept2 provided. Similarly, K/PackageKit was still young when we picked it up in Kubuntu 9.04, and while in 10.04 it’s not the disaster it was in 9.04, it’s still not as nice as a package manager could be. So let’s just let Muon and QApt cook for a while, and then we can see where we stand for Kubuntu 11.04. It’ll be worth the wait.

In the meantime, I plan to go through a prerelease cycle for both QApt and Muon. It is conceivable that at the least qapt-batch could be made ready to replace the python install-package for Kubuntu 10.10, but aside from that nothing big will come. I do plan to make packages for both QApt and Muon available for lucid and maverick throughout the prerelease cycle.

You can grab Kubuntu packages of the first alphas of both Muon and QApt from my PPA for both lucid and maverick here. Muon should also build and work just fine on Debian, but I don’t have packages for it. It requires Qt 4. 6 and up, so users of Debian unstable should be able to build and use it from svn just fine. (muon is in /playground/sysadmin/muon) Tarballs for QApt 0.2 (1.0 alpha1) and Muon 0.2 (1.0 alpha1) can be found here and here respectively.

Please beware that while things should in theory be safe, Muon has not yet had widespread testing, so there is still the slight chance that a bug could still be lingering, ready to kill your kittens. Just keep that in mind, and have fun. I look forward to feedback on Muon, and will continue to work on improving both Muon and QApt for a brighter future for package management.

Oh, and does Muon go moo? It might.