Last week we had one hell of a wonderful KDEPIM spring with about 30 people attending. It was my second sprint, and it’s certainly something i will keep attending.
So, what did i do during the sprint? The first day was spend entirely on my pet project of improving the entire file browsing stack in kdelibs. That means:
I’m doing this because the current stack doesn’t scale very well. It works fine for the every day usage, but begins to show it’s limitations when you use massive folders (100.000 till 1 million entries). I know that optimizing for this is “crazy”, but it’s also fun and seems to work really well thus far. The intention here is to have a rewritten stack that is as good as the current stack, but does scale well. Even with millions of files in one folder. However stupid that might be. I’ve rewritten all of the above classes since i wasn’t about to keep backwards compatibility. And rewriting them has the benefit that i can change the structure however i want to do that. The new names are:
- KDirectory (contains a list of KDirectoryEntry objects for each entry)
- DirModel (is going to be KDirModelV2)
KDirectory and KDirectoryEntry vs KFileItem. In terms of memory usage KFileItem was horrible. It was having one full UDSEntry and at least one KUrl. All sucking up memory that really could be used more effectively. I’ve taken a different approach here. KDirectoryEntry (the KFileItem replacement) is not storing a UDSEntry or a KUrl/QUrl anymore. It’s only storing:
QString with only the file name (not the full url)
mode_t with the details that tell what kind of entry this is (file, folder, symlink…)
bool that tells me if i have more information available like file size, permissions and a bunch of other details
FullStatData* pointer that contains the actual detailed data.
The “FullStatData*” pointer is filled based on lazy loading. It’s only filled when the data is actually needed. So that means the KDirectoryEntry object is very small compared to KFileItem and only loads additional data when it’s actually required to see that data. Even when that data is loaded, the memory footprint is massively lower then what you would see if you where using KFileItem. Some numbers will follow later in this post.
KDirectory is a glorified container class. It contains a list of KDirectoryEntry objects and is being spammed by KDirListerV2 with new entries.
KDirListerV2 is the real big dragon in this picture. What KDirListerV2 is basically doing is create a new KDirectory object for every new directory it wants to index. Then it attaches some signals from KIO::listDir and KDirWatch. Besides that it also contains logic to get a KDirectory object based on an “int index” and some logic to get that index. It’s based on int indexes because that makes it very easy to use it in a QAbstractItemModel. Besides KDirWatch (it seems to mark everything as dirty -_-) it works fairly well. No issues thus far anyway.
KDirModelV2 (DirModel) is the real head breaking part of the code. Playing with the QModelIndex, parent and index functions is just pure pain. Just a difficult part in this pet project, but one i will probably get working at some point in time.
Now for a few numbers.
If you would put a KDirModel (with a KDirLister) on a folder with 100.000 entries then you would be using about ~700MB of memory. If you do the exact same thing using KDirModelV2 (with KDirListerV2) then you are only using ~160MB of memory. That’s quite a big improvement. Even more curious is the place where the memory is being used. It “seems” like most of the 160MB is being used in the part that building up the actual UDSEntry list and the part that’s receiving the raw blob data from the respective KIO slave. That’s the only part i haven’t touched! In terms of speed it’s filling the list very rapidly. I haven’t done any numeric benchmarks in that area yet, but visually it certainly seem as fast (or faster) then the current KDirModel + KDirLister approach.
That’s it for that “little” pet project.
Now this is where the other 2 sprint days where spend. In my last sprint i was already working on the QML Calendar and it was progressing nicely. However, i had a bit of a knowledge gap in how to use QModelIndex and Qt’s models. I’ve been trying very hard to fill that gap in (mainly with the pet project above). At this sprint it was time to take it to the next level. Previously i was trying to access the calendar data through C++ and making it available through C++ in QML. That was working but is not really the way to go forward since other applications could very well benefit of the same things as well.
So we needed QML components! Since John Layt is also having some calendar related wishes it was time to just sit together with a group and discuss each others calendar needs. Tobias, John, Kevin, Andreas, a few others and me sat together during lunch and drafted up an QML Calendar API. I worked that out with Tobias in this wiki page: Calendar API QML. This API consists of two QML components:
The CalendarData component is providing exactly what it’s name implies. “Calendar Data”. You simply put in a date range from which you want to receive data and the types (Events, Holidays.. or nothing which will fetch all types). You then get a model back with a bunch of properties. This properties part is still one of the parts that i have to finish. So what this component allows you to do is – for example – show all todo items for tomorrow. Or all event entries for this month. It’s that easy and that powerful.
The Calendar component is a different beast. That component is meant to show a month overview. All you do is say when to start the overview and which data it should contain. The model will do the rest. A very important thing to remember here is that this component (both components) are just data components. They provide models and won’t show anything. If you combine the two components you can implement the current plasma clock calendar popup in pure QML without a single line of C++. And that is actually my current testcase. A full calendar clock popup applet (what’s the real name of that thing anyway?) in plain QML. Below is a screenshot of how it currently looks. An obvious note: heavily work in progress!
Just to explain what you see in terms of components. The left part of that image (the actual calendar) is the “Calendar” component from the above mentioned wiki link. The previous and next buttons actually work and are tied to the previous() and next() function in the Calendar component. The colored 4, 5, 6 and 13 are actually colored from actual akonadi calendar data. This is real stuff! I didn’t just give those specific cells the purple color. All cells with any event in them are colored purple.
The right part you see here is the CalendarData component. When you hover a date the data on the right is filled in. This is even more work in progress as you can see.
While this is already very awesome and already allows the clock popup to be fully rewritten in pure QML, this is only the beginning. The idea (John’s idea to be honest) here is to be able to modify and add actual calendar data through QML. Those bits haven’t been written yet, they haven’t even been discussed in depth yet, but the end goal here should be a more interactive calendar where you can just pick a date and add an event as you wish straight from the popup applet. Now that is really tapping into the power of QML and Akonadi.
For the components as they are currently described in the wiki. I aim to have them in KDE 4.11 under the “org.kde.pim.calendar” namespace. I’m also planning to have the calendar popup re-implemented in QML for KDE 4.11. I aim for those, but it could very easily end up being 4.12.
That’s it for my very lengthy blog post. I hope to post some more updates in the coming days regarding my “file browsing stack” rewrite since there is a lot more going on in that area.
* fixed the API link. The PIM Sprint photo is nice, but the API is the intention here ;)