While working on the Woodchuck support in gPodder, I decided to profile the code. Reading the Python manual, I thought it would be as easy as:

    import cProfile
    cProfile.run('foo()')

On both Debian and Maemo, this results in an import error:

    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/lib/python2.6/cProfile.py", line 36, in run
      result = prof.print_stats(sort)
    File "/usr/lib/python2.6/cProfile.py", line 80, in print_stats
      import pstats
    ImportError: No module named pstats

To my eyes, this looks like I need to install some package. This is indeed the case: the python-profiler package provides the pstats module. Unfortunately, python-profiler is not free. There's a depressing back story involving ancient code and missing rights holders.

If you're on Debian, you can just install the python-profiler package. Alas, the package does not appear to be compiled for Maemo.

Happily, kernprof works around this and is easy to use:

    # wget http://packages.python.org/line_profiler/kernprof.py
    # python -m kernprof /usr/bin/gpodder

Kernprof saves the statistics in the file program.prof in the current directory (in this case, it saves the data in gpodder.prof).

To analyize the data, you'll need to copy the file to a system that has python-profiler installed. Then run:

    # python -m pstats gpodder.prof
    Welcome to the profile statistics browser.
    % sort time
    % stats 10
    Tue Nov  1 13:09:54 2011    gpodder.prof

             105542 function calls (101494 primitive calls) in 117.449 CPU seconds

       Ordered by: internal time
       List reduced from 1138 to 10 due to restriction <10>

       ncalls  tottime  percall  cumtime  percall filename:lineno(function)
            1   57.458   57.458   69.012   69.012 {exec_}
            1   16.052   16.052   26.417   26.417 /usr/lib/python2.5/site-packages/gpodder/qmlui/__init__.py:405(__init__)
            1    8.591    8.591   13.790   13.790 /usr/lib/python2.5/site-packages/gpodder/qmlui/__init__.py:24(<module>)
           60    7.041    0.117    7.041    0.117 {method 'send_message_with_reply_and_block' of '_dbus_bindings.Connection' objects}
            3    6.357    2.119    7.469    2.490 {method 'reset' of 'PySide.QtCore.QAbstractItemModel' objects}
           36    2.636    0.073    2.636    0.073 {method 'execute' of 'sqlite3.Cursor' objects}
            1    2.283    2.283    2.284    2.284 {method 'setSource' of 'PySide.QtDeclarative.QDeclarativeView' objects}
            1    1.848    1.848    1.848    1.848 /usr/lib/python2.5/site-packages/PySide/private.py:1(<module>)
            2    1.789    0.895    1.789    0.895 {posix.listdir}
            1    0.765    0.765    4.234    4.234 /usr/lib/python2.5/site-packages/gpodder/__init__.py:20(<module>)

The statistics browser is relatively easy to use (at least for the simple things I've wanted to see so far). Help is available online using its help command.

Khweeteur is a great twitter and identi.ca client for Maemo. One feature I particularly like is its support for queuing of status updates, which is useful when connectivity is poor or non-existent (which, for me, is typically when something tweet-worthy happens). It also supports multiple accounts, e.g., a twitter account and an identi.ca account.

Khwetteur can automatically download updates and notify you when something happens. Enabling this option causes Khwetteur to periodically perform updates whenever there is an internet connection---whether it is a WiFi connection or via cellular. This is unfortunate for those, who like me, have limited data transfer budgets.

Deciding when to transfer updates is exactly what Woodchuck was designed for, and recently, I added Woodchuck support to Khweeteur. Now, if Woodchuck is found, Khweeteur will rely on it to determine when to schedule updates (of course, you can still manually force an update whenever you like!).

While modifying the code, I also made a few bug fixes and some small enhancements. Two improvements that, I think, are noteworthy are: displaying unread messages in a different color from read messages, and indicating when the last update attempt occured.

You can install the Woodchuck-enabled version of Khweeteur on your N900 using this installer. You'll also need to install the Woodchuck server, to profit from the Woodchuck support. Hopefully, the version in Maemo extras will be updated soon!

Other Woodchuck-enabled software for the N900 include:

• FeedingIt: An RSS reader
• APT Woodchuck: A software update scheduler for Maemo5

If you are interested in adding Woodchuck support to your software, let me know either via email or join #woodchuck on irc.freenode.net.

I'll be at the N9 Hackathon this weekend in Vienna. Sunday morning (October 9th) at 10am, I'll give a presentation about Woodchuck. I'll talk a bit about Woodchuck's motivation and a fair amount about Woodchuck's architecture as well as what we hope to learn from the user study and how we planning on using it to evaluate different scheduling algorithms. If you are around, you should come by!

I've finished an initial port of Woodchuck to Harmattan. To get it, you need to manually add the source repository: Harmattan's application manager does not support .install files. Add the following to /etc/apt/sources.list.d/hssl.list:

deb http://hssl.cs.jhu.edu/~neal/woodchuck harmattan harmattan

Then, run apt-get update.

The following packages are available: the Woodchuck server (package: murmeltier), the Python bindings (package: pywoodchuck) and the Glib-based C bindings (libgwoodchuck and libgwoodchuck-dev).

smart-storage-logger, the software for the user behavior study, has not yet been ported: I'm still trying to figure aegis out.

If you are interested in adding Woodchuck support to your software, see the HOWTO and the documentation. You can also email me or visit #woodchuck on irc.freenode.net (my nick is neal).

At the recent GNU Hackers Meeting, I gave a talk about Woodchuck. (I'll publish another post when the video is made available.) The talk resulted in a lot of great feedback including a question from Arne Babenhauserheide whether Woodchuck could be used to automatically synchronize git or mercurial repositories.

I hadn't considered using Woodchuck to synchronize version control respoitories, but it is a fitting application of Woodchuck: some data is periodically transferred over the network in the background. I immediately saw two major applications in my own life: a means to periodically push changes to a personal back up repository; and automatically fetching change sets so that when I don't have network connectivity, I still have a recent version of a repository that I'm tracking.

I decided to implement Arne's suggestion. It's called VCS Sync. To configure it, you create a file in your home directory called .vcssync. The file is JSON-based with the extension that lines starting with // are accepted as comments. The file has the following shape:

    {
      "directory1": [ { action1 }, { action2 }, ..., { actionM } ],
      "directory2": [ { action1 }, { action2 } ],
      ...
      "directoryN": [ { action1 } ],
    }

That is, there is a top-level hash mapping directories to arrays of actions. An action consists of four possible arguments: 'sync' (either 'push' or 'pull'), 'remote' (the remote repository, default: origin), 'refs' (the set of branches, e.g., +master:master, default: 'master') and 'freshness' (how often to perform the action, in hours).

Here's an example configuration file:

    // To register changes, run 'vcssync -r'.
    {
        "~/src/woodchuck": [
          // Pull daily.
          {"sync": "pull", "remote": "origin", "freshness": 24},
          // Backup every tracked branch every few hours.
          {"sync": "push", "remote": "backups", "refs": "+*:*", "freshness": 3}
        ],
        "~/src/gpodder": [
          // Pull every few days.
          {"sync": "pull", "remote": "origin", "freshness": 96}
        ]
    }

VCS Sync automatically figures out the repository format and invokes the right tool (currently only git and mercurial are supported; patches for other VCSes are welcome).

After you install the configuration file, you need to run 'vcssync -r' to inform Woodchuck of any changes to the configuration file.

You can use this on the N900, however, because this is a programmer's tool and you need to edit a file to use it, it is not installable using the hildon application manager. Instead, you'll need to run 'apt-get install vcssync' from the command line (the package is in the same repository as the Woodchuck server). If you encounter problems, consult $HOME/.vcssync.log.

I also use this script on my laptop, which runs Debian. Building packages for Debian is easy, just check out woodchuck and use dpkg-buildpackage:

    git clone http://hssl.cs.jhu.edu/~neal/woodchuck.git
    cd woodchuck
    dpkg-buildpackage -us -uc -rfakeroot

This (currently) generates eight packages. In addition to vcssync, you'll also need to install murmeltier (my Woodchuck implmentation), and pywoodchuck (a Python interface to Woodchuck).

One of the arguments for Woodchuck is that it can save energy. In this post, I want to examine that claim a bit more quantitatively.

As part of some Woodchuck-related work, I've done a fair amount of Python programming on Maemo. Python, being an interpreted language, runs the source code; there is no need to compile it to some binary representation as is the case with C. This is a great convenience when developing for a device such as the N900: there is no need to compile the code and copy the resulting binaries; I just edit the code on the device and run it. The trade-off is that I need to edit the files directly on the device: but, I want my Emacs (qemacs is not enough!), git and the regular GNU tools. It turns out that I was able to get pretty close.

The following text is from the introduction of the HOWTO I've written explaining how to modify a program to use Woodchuck. The focus is on the Python interface, but it should be helpful to anyone who wants to modify an application to use Woodchuck. This document, unlike the detailed documentation, should be a bit easier to digest if you are just getting started with Woodchuck. If questions still remain, feel free to email me or ask for help on #woodchuck on irc.freenode.net.

Woodchuck is a framework for scheduling the transmission of delay tolerant data, such as RSS feeds, email and software updates. Woodchuck aims to maximize data availability (the probability that the data the user wants is accessible) while minimizing the incurred costs (in particular, data transfer charges and battery energy consumed). By scheduling data transfers when conditions are good, Woodchuck ensures that data subscriptions are up to date while saving battery power, reducing the impact of data caps and hiding spotty network coverage.

At the core of Woodchuck is a daemon. This centralized service reduces redundant work and facilitates coordination of shared resources. Redundant work is reduced because only a single entity needs to monitor network connectivity and system activity. Further, because the daemon starts applications when they should perform a transfer, applications do not need to wait in the background to perform automatic updates thereby freeing system resources. With respect to the coordination of shared resources: the cellular data transmission budget and the space allocated for prefetched data need to be allocated among the various programs.

Applications need to be modified to benefit from Woodchuck. Woodchuck needs to know about the streams that the user has subscribed to and the objects which they contain as well as related information such as an object's publication time. Woodchuck also needs to be able to trigger data transfers. Finally, Woodchuck's scheduler benefits from knowing when the user accesses objects. In my experience, the changes required are relatively non-invasive and not difficult. This largely depends, however, on the structure of the application.

...

I designed Woodchuck's API to be easy to use. A major goal was to allow applications to progressively add support for Woodchuck: it should be possible to add minimal Woodchuck support and gain some benefit of the services that Woodchuck offers; more complete support results in higher-quality service.

To support Woodchuck, an application needs to do three things:

• register streams and objects;
• process upcalls: update a stream, transfer an object, and, optionally, delete an object's files; and,
• send feedback: report stream updates, object downloads and object use.

The rest of this document is written as a tutorial that assumes that you are using PyWoodchuck, the Python interface to Woodchuck. If you are using libgwoodchuck, a C interface, or the low-level DBus interface, this document is still a good starting point for understanding what your application needs to do.

Read the rest.

A couple of weeks ago, I was chatting with Michael Banck about DebConf. He told me that one of the sponsors provided everyone a SIM card with 5 units of credit, and that the first time he established a data connection was also his last: he got bit by Maemo's automatic repository update misfeature; because, he had gone more than 24 hours without checking for software updates, Maemo checked even though he was using a cellular data connection and only had a few megabytes worth of data transfer credit.

A simple workaround for this bug is to disable updates. This has the unfortunate side effect that the user is no longer informed when updates become available. A better solution is one that fetches updates when background updates are acceptable. This is exactly the type of scheduling problem that Woodchuck was designed to help applications with.

Over the past few days, I've developed APT Woodchuck, a small Python script, that does exactly this: APT Woodchuck lets Woodchuck determine when it should check for updates. On installation, APT Woodchuck disables HAM's automatic update feature and registers itself with Woodchuck. When Woodchuck decides APT Woodchuck should perform an update, it starts it (using DBus). APT Woodchuck then updates the package list (it uses HAM's apt-worker utility to ensure that all of HAM's usual update mechanisms are performed, including tickling the update widget, if necessary). APT Woodchuck also prefetches packages for which an update is available.

In total, APT Woodchuck is about 700 SLOC and took about 2 days to write. Most of the time was spent figuring out how to use Python APT. That seems to me like a pretty easy solution to a hard scheduling problem.

I've made packages for APT Woodchuck for Maemo available.

If you are thinking about including Woodchuck support in your application, APT Woodchuck is a fairly good example of how to go about going it.