More KRatingPainter optimizations

Following the optimization I made last night to KRatingPainter’s paint function, (KRatingPainter is what paints those nice little rating stars in Dolphin, Gwenview, and some Muon-related project that I’m not quite ready to receive UI criticism for :P) I discovered that there were also a suspicious amount of KColorScheme-related calls coming from somewhere. Since KCacheGrind is such a lovely tool, I was able to deduce quickly that these were coming from the KIconEffect::apply() calls that KRatingPainter::paint() was calling.

There are several ways to call KIconEffect::apply(). (See http://api.kde.org/4.x-api/kdelibs-apidocs/kdeui/html/classKIconEffect.html) They all differ in what parameters you supply to the function. Some are more efficient than others. In our case, KRatingPainter was using the following apply() function:

QPixmap  apply (const QImage &src, int effect, float value, const QColor &rgb, bool trans) const

This function takes your picture, the effect you want the strength of your effect, a color to use should the effect support a separate color, and a boolean value to determine whether or not to apply transparency. This is actually a convenience function, since icon effects that require colors require two different colors in some cases. In these cases KIconEffect uses KColorScheme to determine what the best color to use in these cases would be.

The effect in use by KRatingPainter, the “to gray” effect, does not require any additional colors to be specified at all, so by using the wrong apply() function KRatingPainter was spending time finding a color that would never be used. I remedied this by using this overload of the apply() function:

QPixmap apply (const QPixmap &src, int effect, float value, const QColor &rgb, const QColor &rgb2, bool trans) const

This one makes you specify two colors, so time is not wasted looking in your color scheme configuration to determine the best color. I just specified two colorless colors (null QColor()’s, to be technical) and everything was fine and dandy from there on out. It sped up KRatingPainter at least another 10x, on top of the 500x gained from last night’s optimization. In my delegate’s case, this optimization cut the time my ItemView delegate spent painting in half. (A bit more, actually, though data for my delegate specifically is not included in the below screenshot.)

I am fairly sure that rating painting is no longer a bottleneck in my ModelView Delegate class anymore. Compared to last night, I can make it much bigger before seeing noticeable losses of scrolling fluidity.  It should be really fast now, and other applications should be able to benefit from these changes in KDE 4.6 as well.

So, in conclusion, valgrind/callgrind and KCacheGrind (valgrind/callgrind file viewer) are invaluable tools, and are a must-have for any KDE developer. This is by no means the first time that they’ve helped me discover bottlenecks, but this is one of the more drastic optimizations that they have helped me find.

Sharing (KIconEffects) is caring

This evening I made an optimization to KRatingPainter (a class in libnepomuk in kdelibs) that I just have to blog about. I was using the KRatingPainter class to paint one of those 5-star rating dealies onto the item delegate of a listview. When I maximized the window, scrolling this list view became very slow. I broke out callgrind for a debugging session and scrolled down the list using the arrow key whilst my other hand assisted me in eating a jelly doughnut whilst I waited for the (now considerably slower thanks to callgrind) view to scroll enough to get sufficient data from callgrind.

Looking at callgrind’s output, the culprit was obvious. The paint() function of KRatingPainter was 20% of all computing time that had been spent during the program’s run. More specifically, the KIconEffect constructions going on in paint() were the culprit.

So I delved into the source to figure out why so many KIconEffects were being constructed during paint(). It turned out that KRatingPainter was using the KIconEffect class to apply enabled/disabled/hover effects to the stars it painted, but was creating a new KIconEffect each time it wanted to paint(). It ended up creating 2 KIconEffect objects every time paint() was called. KIconEffects are quite expensive to construct, as they have to read configuration files each time they are made. (The above picture illustrates the expensiveness nicely.)

But, as it turns out, KIconLoader provides a shared KIconEffect object that all KDE apps can use (unless they want a custom one for some reason). Using a pointer to this shared KIconEffect object rather than creating 2 per paint resulted in a speedup of near 500x.

Whereas before KRatingPainter had consumed 20% of all computing time spent, the second time around KRatingPainter painting took only 0.04% of all computing; meaning that, if it wanted too, it could have done 500 times more paints in the same amount of time it took to paint the first time around. Notice too from the picture that KIconEffect::KIconEffect (the constructor) was only called once after the fix, whereas it was called 400-something times without the fix. So take this as a lesson: always use the shared KIconEffect unless it is absolutely necessary not to.

This was just to epic a win not to blog about. 😛

Muon and QApt 1.0.1

After a somewhat difficult day release-wise, I am relieved to announce the release of Muon 1.0.1 as well as QApt 1.0.1, the package management library that Muon uses. For those of you unaware, Muon is a powerful Debian-based package manager with a sane GUI for KDE. A full rundown of its main features can be found here. Packages are available in Maverick proper as well as in my PPA for Lucid. Source tarballs can be found here and here.

Here’s a list of what’s changed since 1.0.

QApt 1.0.1

• Fixed what is hopefully the last of “libapt-pkg gives LibQApt null const char pointers if for some reason libapt-pkg can’t get the string we want” crashes. (Bug 247439)
• Fixed a typo in the API documentation
• Slightly optimized the QApt::Package::isSupported() function by using better logic to reduce the amount of queries we make about package info.
• A few coding style fixes
• Fixed a bug where the worker would never send a “finished” signal if a download ended in error. (Bug 247534)
• Report errors that occur while APT is initializing the package system. (This was an oversight… the error type was there, but was never used…) This prevents crashes on startup when errors like invalid lines in the package sources list being malformed and such. (Bug 247533)
• A small memory leak was fixed by mlaurent. Thanks! This leak should not have had an impact on virtually any application using LibQApt that I know of, since the Backend class that had this leak on deconstruction does not get constructed/deconstructed a lot in practice.
• Quieted some console debug messages.
• Enhance QApt Batch’s handling of InitErrors to show the newly-available details about these errors, and quit after the user has been presented with the error and has closed the error message box.
• In step with the Aptdaemon project, allow unprivileged users to check for updates by default. This can still be configured in the PolicyKit settings for those who wish to lock it down more, but really it shouldn’t be a security risk.

Muon 1.0.1

• Show the newly-available details about InitErrors from QApt, and quit after the user has been presented with the error and has closed the error message box to prevent Muon from being a useless shell.
• Add a throbber next to the “Get Screenshot” button while fetching the screenshot to enhance usability via feedback.
• Fix a bug that crept in where unmarking a dependency of a package would not unmark the dependent package itself. (Bug 248376)
• Disable the screenshot directly after the user clicks the button so that the user can’t go into a crack-induced trance, and click the screenshot button multiple times and create a ton of windows. Muon; saving KDE users from crack since 2010. 😛 (Bug 248382)
• Allow purging residual system configuration files of uninstalled packages. Previously you could only purge these files if the package was installed. In 1.1 I plan to add a filter for uninstalled packages that still have residual system configuration files.

This release was a bit tough. It turns out that the CamelCase-style header for KPixmapSequenceWidget, the widget I am using for the busy throbber, was missing in KDE 4.4. This meant that Muon failed to compile in Lucid, which is where most of the users of this software probably are right now. 😦 This was after I spun the initial tarballs. New tarballs with the fix for KDE 4.4 are up as of now.

Similarly, (In that I discovered this after making the tarball) the libapt-pkg headers in Lucid are a bit different than they are now in Maverick, and the new error handling bits for APT initialization were missing an include that needed explicit inclusion in lucid.

But, they’re both out now, with a nice batch of fixes to make things more solid. Both QApt and Muon have moved to extragear, and once I figure out how (or who to speak with) to get an application branched in svn I plan to reopen trunk for new features and string changes. If crashers are found with 1.0.1 I will do a 1.0.x releases to remedy them, but for the next 6 months I plan on adding some new features to Muon, creating a more update-based GUI, as well as an application-based manager similar to Software Center.

Oh, and for Kubuntu users of Muon, in 10.10 the Debconf GUI has as of today received Kubuntu branding thanks to Dantti’s nifty branding feature:

Plasma theme is Kaleban, for those who are interested

I think it’ll even use the Ubuntu logo if you don’t have kubuntu-default-settings installed. (E.g. in GNOME) And of course it’ll still be the Debian logo in Debian. Dantti’s done some neat work on this front, and you should check out this blog to see how KPackageKit is using libdebconf-kde. I think it’s neat that we can share this component. 🙂

Anyways, expect to hear a bit less from me in the coming few weeks. School is starting up next week, so that’ll be a big time sync. I’ll still be around, though, just not as much.

QApt/Muon 1.0 released

I am excited to announce the immediate availability of QApt and Muon 1.0.

This marks the point where both QApt and Muon are ready for general use. Here is a list of changes from the release candidate:

QApt 1.0

• Fixed a crash that occurred when trying to get the version of a package that lacks a version due to APT pinning rules. (Bug 246799)
• General improvements to the LibQApt API documentation. All public functions and members should be completely and accurately documented as of this point.
• Think forward a bit and make sure we can handle rare-as-of-now ultra-large packages, the size of which in bytes can not be represented by an int. (E.g. a couple of gigabytes) Use a double instead. (qint64 typedef)

Muon 1.0

Felix Geyer deserves a big thank you, as he is responsible for most of the improvements I am about to list. Thanks again!

• Select the “All” item in the filter widget views by default for clarity’s sake. Thanks for Felix Geyer for the patch.
• Improve reliability of clicking on the filters.  (Felix strikes again)
• As a nice little convenience, if we revert all changes inside the preview widget, Muon will now return you to the main few. (Felix once again)
• Improve the reliability of the changelog fetching code, reducing the risk of files not getting deleted/the wrong changelog being displayed when quickly scrolling through packages with the keyboard. (Felix)
• My chance to shine 😉 Only listen to progress reports from the QApt Worker if we are actually the one that asked. Otherwise we could try displaying progress from another application such as QApt Batch and crash since we don’t have the progress widget set up.

Packages are available from the usual location, with the exception of Maverick, where things are delayed a bit as I prepare to get Muon in the official archives for 10.10. They should be available by tomorrow if everything goes well. Source for QApt and Muon are available here and here, respectively.

The future

I have some ideas of what I’d like to accomplish for 1.1. Here’s a list of some of them:

• Implement version locking support
• Improve the download view, with per-item progress
• Implement a history log viewer, now that APT has a consolidated history log. (Will require APT from 10.10)
• Add a configuration dialog for configuring things like whether or not to treat recommended packages as dependencies or not, and other APT settings.
• “Slow” search that does not rely on Xapian and can search in specific package fields. This would supplement fast search, which will remain prominently in the main GUI. (Slow search will probably only be accessible from a menu in the menubar)

Of course, I’d also like to write a more Software Center-esque custom GUI for application installation. For that I will have to write a parser for app-install-data, and use that parser to filter what is shown in a GUI that is geared towards the technically non-elite. I’ll also have to figure out a clever way to limit the Xapian quick search to the applications. I am a bit unsure how to accomplish this as of yet.

So, enjoy Muon. QApt will be getting a bit of use via QApt Batch in 10.10, but until 11.04 we really need to test the heck out of QApt and Muon to ensure that Muon will be a good choice for inclusion by default in 11.04. (It’s not that I doubt that they are good and ready now, but every single time we have jumped on the latest and greatest it’s come back to bite us, even if it was a popular decision beforehand.) If we can make sure that they are good, we will ensure that they rock for 11.04. 🙂 Muon will be available for install in the official repositories for 10.10, so there is no excuse to go out and test it while waiting for 11.04. 😉

Thanks, and enjoy.

QApt in Maverick

Almost home from vacation; another day in the car and I shall be home!

I’d like to take this chance to share what has been going on with QApt in Maverick, and what you can expect to see. While Muon will not be included on the CD in Kubuntu 10.10, once the “new” queue has been cleared out it will be available for install as any other application. More on that Wednesday, though.

What you will see in Maverick is the QApt Batch utility. This is a little application that acts as a batch installer/update application that other applications can use when they need to check for updates or install new packages without having to become a full-blown package manager. In Maverick, I have used QApt Batch to fix bug497803. If it doesn’t look too much like a bug, or even a problem, that is understandable.  Her is a bit of an explanation.

The original intent was to use KPackageKit to replace install-package, our batch installer utility. This (and GDebi-KDE) are two technologies that have been slated for removal in the Project Timelord plan due to lack of maintenance. KPackageKit was, however, not quite able to do all that we wanted with it, and also brought a lot of the KDE stack when it was used for installing packages in kubuntu-debug-installer. (What Dr. Konqi uses to get debug packages) QApt Batch is a drop-in replacement for install-package, its predecessor. It acts and looks similar as well, but with some marked differences:

Security

QApt Batch is safer to use than install-package was from a security standpoint. QApt Batch will inform you if you are about to install something from an untrusted source and either error out or ask if you want to proceed (depending on your APT settings), whereas install-package will merrily go along with the install without any concern. Best case, you just forgot to add the GPG key for some PPA. But worst case, you are being hit with a DNS cache poisoning attack or are in a country where the government cannot always be trusted, and you may be receiving a maliciously modified package.

Robustness

QApt Batch is also more robust in several ways. Primarily, it will catch and report more kinds of errors than install-package did. Being a python application, install-package always tended to have a RuntimeError or few hiding in the code, usually in the error handling… In addition, QApt Batch also handles debconf where install-package didn’t. This means that install-package would fall over on itself if it tried to install the Java packages, which have a Debconf EULA. QApt Batch handles debconf through libdebconf-kde. Furthermore, QApt Batch theoretically has Media switch (CD change) support. I say theoretically because I’ve not had the opportunity to test it…

Here’s a list of all programs in Kubuntu now using QApt Batch that were using install-package:

• Apturl-kde, which used install-package’s batch install features for the apt:/ protocol
• Kubuntu Notification Helper, which used the batch install features for installing proprietary addons, if requested.
• Language Selector, which used install-package for installing language packs.
• Software Properties KDE, which used install-package to refresh the package cache after you finished editing software sources.
• Kubuntu Debug Installer, which used KPackageKit to install debug packages and made installing any KDE app bring in KPackageKit/PackageKit in 10.04. 😦 The Kubuntu Debug Installer now uses LibQApt for seeing which debug packages need installing instead of hackishly using the output of “dpkg -s”
• Kubuntu Firefox Installer, which used install-package to install Firefox plus the KDE integration

All in all, quite a win for robustness, consistency, and optimization if I do say so myself.

I plan to look at my email once I get back from vacation tomorrow. (Or maybe I’ll look at it Wednesday) so please don’t feel offended if you’ve sent an email in the past few days and have not gotten a response as of yet.

See you all Wednesday!

QApt/Muon 0.5.0 (1.0 RC)

The release train keeps rolling on… In one week both Muon and QApt 1.0 will be released. Until then, here’s one last pre-release to iron out any remaining issues before the stable release. Packages are in the normal place, and sources for QApt 0.5.0 can be found here. (Muon 0.5.0 sources will be uploaded in the usual place once I get back to the hotel… unfortunately I cannot do the GPG signing of the tarball on my grandparent’s computer) Here’s what’s new since last week:

QApt 0.5.0

• No coding changes, but there are now a weeks worth more translations for the qapt-batch utility. 🙂 Go, go kde-i18n!

Muon 0.5.0

• A teeny new feature, but I added a nice little busy indicator overlay on the changelog viewing tab while it fetches the changelog. This improves usability, so that you can tell when it is really working, etc. Try it out. 🙂
• Fixed a bug where an error such as an authentication error or download error would not properly return Muon to a usable state. (Grayed-out toolbar, getting trapped in the preview view.) Bug 246596
• A minor code cleanup for the software-properties-kde launching code. Nothing too special.
• Fixed a typo in the dependencies tab.

As you can see, the list of changes and bugfixes has slowed significantly. This is a sign that Muon really is ready for a release in one week; I’d be a bit afraid if I was still fixing major bugs at this point. 🙂

So, yeah, final release in a week. I’ll be getting the RC into maverick in the next few days (keep in mind that it won’t be default just yet) and will update to the final release the day it becomes available. (“Coincidentally” also being the day of Ubuntu feature freeze ;)) Thanks for all the support, testing and bug reporting. At this point all reported bugs against Muon have been fixed, and Muon has been translated in to over half a dozen languages. See you in a week!