KIOFuse: June in Review

The coding period has now extended over a month and quite a few improvements have been merged into KIOFuse. In my last post I mentioned the development of a KIO error to FUSE error mapping and 32 bit support.

However, interestingly enough it took quite a long time for the 32-bit support branch to be merged. This was because of a test that didn’t fail nor pass – it froze. The test suite would never finish and the process would only respond to SIGKILL. After days of debugging it was determined that fuse_notify_inval_* functions don’t play well when writeback caching is enabled and hence there is now a patch to disable it. Of course this will incur a performance hit as writes will go straight to KIOFuse, and hence straight to disk (although the kernel may cache our write requests to our own cache). Whilst this is unfortunate, seeming as most KIO slaves are network based, switching from a writeback caching policy to a writethrough one is unlikely to hamper performance too much.

In other news, KIOFuse can now handle SIGTERM, SIGINT and SIGHUP signals. Signal handlers can only call async-signal-safe functions. However in Qt there is a bit of hack one can perform, as inspired by this tutorial. Hence, in response to these signals, KIOFuse will flush all dirty nodes to disk, meaning no sudden data loss!

Mounts can now have their password changed.

The lookup function has now been optimised. Previously a lookup would call KIO::listDir, which was totally unnecessary – a KIO::stat would suffice and this is what the patch has switched to. It also increased the data buffer from 1MB to 14MB.

Unmount support has currently been postponed. It is proving problematic to implement reliably and unmounting only really provides a marginal benefit, so it is yet to be seen if we will implement it at all. The current WIP patch can be found here.

It has been decided that slaves that do not implement KIO::stat will not be supported. It’s a bit of a hassle to implement with extremely marginal benefit. There are only a few slaves that don’t implement KIO::stat, such as fonts and thumbnails.

An issue with KDE Connect not working properly has been fixed upstream. I haven’t 100% confirmed which patch has fixed this for us, but I’m placing my bets on this one.

Currently, the Google Drive API reports a size of zero for files that are not supported by GDrive, such as odt files , and their proprietary formats – i.e. Google Docs. Whilst we can update the size quite easily by downloading the file, the file turns out corrupted, and is only openable if the program has a repair option (such as LibreOffice). Unfortunately, I’ve not been able to find out why exactly this is happening, and have not come up with a fix. Currently this is being shelved and I hope to revisit it at a later date with a fresher mind.

We’d welcome anyone to use and test KIOFuse. Feel free to notify us on any bugs or performance issues by opening an issue and you can even contribute a patch if you wish!

Advertisements

KIOFuse: 32-bit Support

The first two weeks of the GSoC coding period are now over.

Firstly, a mapping between KIO errors and FUSE errors has now been established. Previously all KIO Job errors were simply sent back to FUSE as EIO, which isn’t entirely accurate. The mapping now provides more accurate error replies.

A major new addition is 32-bit support. KIOFuse did not compile on 32-bit but these compilation errors have now been alleviated. They mostly stemmed from the fact that size_t has a different size on different architectures, and that file sizes should always be represented as off_t anyway.

Another big question was whether files larger than 4GiB could be sent without data corruption. A switch from the C standard I/O functions to UNIX I/O functions was necessary to avoid data corruption. However, this alone was not sufficient. On 32-bit files larger than 4GiB I noticed that no write call was occurring, although it occurred on 64-bit and on 32-bit with smaller files. I couldn’t point this down to anything in KIOFuse (seeming as KIOFuse simply responds to requests passed by FUSE, what can we do if we don’t receive the necessary request?). Luckily one of my mentors, fvogt, spotted a kernel bug report, which fortunately was fixed about a month ago. This bug was the cause of the problem I was seeing. This means that transferring of files larger than 4GiB via KIOFuse is now supported on distros running kernels 5.1.5+ only. Because of this, we do not recommend packaging or using KIOFuse on kernels less than the mentioned version, due to likely data corruption when transferring large files. This patch is still not in master and is under active review. It is being looked into whether adding a unit test is feasible.

I’d like to thank my mentors, fvogt and chinmoy for their help and guidance so far and look forward to improving KIOFuse further. Over the next week, signal handling and password change/unmount (if necessary) support are the features that will be worked on.

KIOFuse – GSoC 2019

It’s been a great pleasure to be chosen to work with KDE during GSoC this year. I’ll be working on KIOFuse and hopefully by the end of the coding period it will be well integrated with KIO itself. Development will mainly by coordinated on the #kde-fm channel (IRC Nick: feverfew) with fortnightly updates on my blog so feel free to pop by! Here’s a small snippet of my proposal to give everyone an idea of what I’ll be working on:

KIOSlaves are a powerful feature within the KIO framework, allowing KIO-aware applications
such as Dolphin to interact with services out of the local filesystem over URLs such as fish://
and gdrive:/. However, KIO-unaware applications are unable to interact seamlessly with KIO
Slaves. For example, editing a file in gdrive:/ in LibreOffice will not save changes to your Google Drive. One potential solution is to make use of FUSE, which is an interface provided
by the Linux kernel, which allows userspace processes to provide a filesystem which can be
mounted and accessed by regular applications. ​KIOFuse is a project by fvogt that
allows the possibility to mount KIO filesystems in the local system; therefore exposing them to
POSIX-compliant applications such as Firefox and LibreOffice.

This project intends to polish KIOFuse such that it is ready to be a KDE project. In particular,
I’ll be focusing on the following four broad goals:
• ​Improving compatibility with KDE and non-KDE applications by extending and improving
supported filesystem operations.
• ​Improving KIO Slave support.
• ​Performance and usability improvements.
• ​Adding a KDE Daemon module to allow the management of KIOFuse mounts and the
translation of KIO URLs to their local path equivalents.