I must warn Planet Maemo purists (if there are still any) that it is not, stricto sensu, Maemo-related.

It is, however, a follow-up on previous posts dating back to 2009, so it is arguably not completely off-topic either :-)

You can read it on my new host & blog:

w4.monbo.net/blog/index.php?article33/a-foolish-consistency-is-the-hobgoblin-of-little-minds

1 0

I use about 1% of its wealth of functionality, and not often enough to become proficient at it. It’s also slow as molasses on this cheap host I’ve been using.

Moving to a better host (for many other reasons) also made me aware of the hassle involved in carrying over database contents for such apps.

So instead I painstakingly extracted my posts from WordPress and ported them to a much simpler, lighter and database-less blog app called PluXml.

There will be a post with actual content appearing shortly on this new media, which makes it two posts this year, a huge improvement on the previous ones :-)

The new IBTC is at http://w4.monbo.net/blog/ and the old one will probably disappear shortly.

2 0

This post is a follow-up on my “Season one” series about the value of web apps in general (and Python with Web2py in particular) as time-resistant, cross-platform tools, carrying over from Windows/Linux PCs to handhelds like the Zaurus, the Nokia tablets (770/N800/N810) and phones (I stopped at the N900, but I’m pretty sure the N9 and N950 would have worked just as well).

Although I still follow the happenings on TMO, like many former ITT old-timers I finally grew tired after seven years of Nokia switch-and-bait politics, the Meego debacle and Windows alliance being the final straws.

My N900 was feeling orphaned and getting old, so earlier this year I regretfully and reluctantly switched to an Android phone, the Samsung Galaxy Note. From the looks of this thread on TMO, I’m not the only one.

I guess my opinion about the system is similar to that of most former N900 users.

On the one hand, the sheer mass of the Android user and developer base (the only one on par with iOS/iPhone) has turned it into a rich and lively user-centric “market” : even at the cost of weeding out the chaff, the number and variety of useful (and often free, even open source) day-to-day apps is incredible, so just using the phone is easy and pleasurable enough.

On the other hand, for a born tinkerer and former Maemo lover, there are a lot of irritants beneath the pretty skin.

Technically, the multi-tasking is crap, but that is hardly news.
Worse, Google has put quite some effort into hiding and burying the open, Linux roots under piles of “shiny-shiny”… which means hordes of individual users are putting back many times that amount of effort into unearthing it !

Coming from Maemo, the level of obfuscation and the number of mazes needed to solve (normally) simple tasks is a real shock : flashing alternative firmwares, rooting, obtaining a terminal with a proper shell and basic utilities… not to mention “simple” add-ons like ssh and Python !

Community-wise, the level of fragmentation is astronomical : no ITT/TMO equivalent, just a myriad of user-level forums around the ultra-nerd XDA, the sum total of disinformation so high it’s really hard to tell real answers from wishful thinking.

But this post is not about Android itself : let’s just say that things that were so simple and effortless in Maemo, like having a shell, a python interpreter and running your own scripts, have turned into needlessly complicated tasks, with significant learning curves, in Android.

But once you’ve done that, lo and behold !

You just move over your web2py archive (complete with apps, code and data), unpack it in the SL4A scripts folder, and run your (painstakingly modified) launcher script through the SL4A UI, and… that’s it.

Your old web apps from all previous mobile devices just start working again (only faster, as usual).

The same can’t be said, unfortunately, for the couple of PyQt apps I made for the N900… so : QED :-)

3 0

This post is a shameless plug for a new project started in this TMO thread.

The goal is to take advantage of the “USB Host” (h-e-n) package to stream digital audio through the USB port.

In the portable music player scenario, the N900 is used for digital music storage (32BG internal + µSD), user interface software (like Rockbox) and decoding music files (like mp3 or flac) to uncompressed PCM format.

This stream is then sent out through the USB port to an external digital-to-analog converter (DAC) and headphone amplifier, bypassing the N900′s own electronics, to achieve maximum sound quality (see the first post in the thread for a more detailed explanation).

Such DAC/amp “combos” exist in portable form, such as the Fiio E17 and iBasso D-Zero.

Another (technically similar) scenario is to stream sound in through the USB port, for example from a portable mixing console, and use the N900 as a high-quality recording device.

Right now we have proof-of-concept for both use cases, as h-e-n with power-kernel automatically supports such USB audio functions, and the corresponding ALSA device is created.

Using command-line tools in xterm (like mplayer, MOC or arecord), sound can be streamed to/from this ALSA device, with very good results.

One limitation of this method, however, is that it doesn’t work with Hildon GUI apps (like the stock Maemo media player, or the Rockbox port).

Another is that it doesn’t use the N900′s DSP, doing all the encoding/decoding in software, with higher than necessary CPU usage.

Our present understanding, so far, is that we would need to:

• tweak PulseAudio to make the USB “sound card” available system-wide,
• and use gstreamer to take advantage of the mafw framework and the DSP.

Unfortunately none of us end users really understand how this alsa/mafw/gstreamer/pulseaudio puzzle fits together inside Maemo. Conventional wisdom for regular desktop Linux does not seem to apply, and all experiments to date have failed.

If you are reading this and are knowledgeable about the specifics of the Maemo sound system… and enjoy a challenge… please do chime in ! :-)

8 0

Ah, the moon’s too bright
The chain’s too tight
The beast won’t go to sleep
I’ve been running through these promises to you
That I made and I could not keep
(Leonard Cohen)

1 5

On the surface, it’s all about available free time and (lack of) know-how, as always. As soon as the app was usable I made it available as a tar.gz archive, with a helper shell script inside for installation. Clearly the vast majority of my three-to-six self-declared “users” got away with it, insofar as they immediately found a bug… Then one politely inquired if/when it would be available in the maemo “repos”.

My terse, on-the-spot answer was: “I don’t think so… I have strictly zero experience in packaging .debs and of course even less in how the maemo repos work. I have a feeling it would take me more time to learn all that than to code the app itself, and I already have too little time as it is. The entire source is in the .tgz package however, so if someone more knowledgeable wants to make a .deb out of it, even if it’s not in a repo, that could at least help some.”

Actually that is not the entire, candid truth. I did give it a try a few years ago, in the early days of the 770 tablet, when I knew no better. And what a painful, not to mention unsuccessful, experience that was. I wasn’t alone in that, either, and stronger souls than me lost hair over seemingly trivial issues like icons and task bars and such.

Of course one might argue that things are better now, with more documentation and more collective knowledge to share, which is true… but last time I looked, the actual process itself hasn’t become any simpler, and for good reason : it’s passed down as word of law from Maemo‘s debian roots, just like Meego will get rpm, and those are just facts of life. In some ways there have even been regressions : in Maemo4 there was a GUI helper app by Khertan, PyPackager, to help beginners seeking to craft their first .deb package, but it’s not available (in that form) for Maemo5.

So yes, I still feel strongly against spending scarce time and energy getting that stuff right, when I could use it to improve the app itself — I’m the first user, after all. But that’s not the entire story, far from it.

I also question the very logic of packaging a PyQt application.

You see, there is a huge difference between such apps (any Python app, actually) and regular, binary apps compiled from C or C++ : under Linux, they install and run just fine in “user-space” (i.e. somewhere below /home/user/), even in non-Linux filesystems (like MyDocs) because they have no real executable files. Just type “python appname.py” and you’re go.

Another important difference is that Python code is inherently portable : Python aficionados are not just wimps who can’t handle a compiler, they also tend to be platform nomads as well.

Combine the two factors above, and suddenly that good old .tar.gz archive starts to make a whole lot of sense. I can drop it just about anywhere on any of my Windows or Linux PCs, my N900 of course, and even on good old N8xO, and it runs. My guess is it would also work on MacOS X. Heck, I could even keep it in a DropBox folder and run it from there…

Everything grouped underneath a single app folder: code, user settings and data. Full automatic backup with standard platform method. Instant migration between devices. Almost as good as a web app, if it weren’t for that tiny PyQt requirement :-)

Now, what’s to be gained by packaging such an app ?

One package per platform, for starts. Debian standard or not, a Maemo5 package would likely install only on a N900 (if only because of /opt), likely not on Maemo4 or on a regular debian-like PC. Not to mention rpm-based Linux, Windows, or MacOS. Talk about code reuse and leveraging skills…

More dumb scaffolding code, for seconds : a simple app that expects to find everything inside its own cosy nest will need platform detection and additional logic to store and retrieve all its bits and pieces according to local conventions… whereas SnXM has exactly two #ifdef equivalents, and that’s only because it’s optimized for looks on the N900. Twidi‘s excellent (and much, much more complex) GRead needed exactly three lines tweaked to run on the older Qt available on Maemo4, etc.

Dependencies ? PyQt itself is “in the repos”, along with many platform-specific python modules, and the Python “batteries included” model reduces the need for third-party modules. These can always be bundled inside the archive anyway: SnXM has exactly one.

Updates/upgrades ? With basic precautions in place (no default .ini file or empty database, etc.), it’s as easy as download the new archive and unpack on top of the previous version. You get the new code and functionality, with existing user settings and data. For GRead on the N900, I even do a git pull from twidi‘s Github URL and that’s that.

So what’s left ? An icon and a .desktop file placed in the right spots that need root privileges ? Isn’t that a rather high price for something “sh snxm.setup” does in six lines of shell ?

Okay, rant over. I don’t dispute that for an individual user who just wants to use an app on his one device, a platform-native installer is a bonus, especially as he’s often been trained to believe it is the Only True Way :-)

However, in this case, and specifically for the N900, I believe the only reasonable compromise would be a .deb package which :

• keeps the monolithic aspect of the app, i.e. installs the whole enchilada to /home/user/MyDocs/SnXM for example,
• just takes care of placing the launcher, desktop file and icons in the usual places.

‘

I’m not even sure this is technically possible, and it certainly isn’t standard/approved practice as I understand it.

But if someone wants to do it, I’ll gratefully be proven wrong :-)

5 1

What didn’t fail to happen, however, is something I anticipated in a previous post… Since that time I became ever more engrossed with the speed of Python on the N900, the release of a Nokia-sponsored, beautifully “Hildonized” version of PyQt for Maemo5, and the wealth of great community applications that ensued : KhtEditor, DropN900, GRead, to name but a few.

Until, of course, I found an “itch to scratch” and a plausible excuse to make my own.

The itch was Simplenote, a web service that lets you jot down quick text notes and access them from anywhere. Think of it as a KISS version of Evernote that handles only plain text, but fast and well, especially when used off-line through native clients. There are quite a few such apps, for Windows, Mac OS and some mobile platforms : iOS and Android obviously, also Palm webOS, but no Maemo, as usual.

That was bait enough for me to take the plunge. After all, it did have “simple” in it, didn’t it ?…

So here is version 1.0 of SnXM, my very first PyQt app for the N900. And onwards, I hope : I’m confident it’s almost as future-proof as a web app, and the name reflects that — XM is 990 in Roman numerals :-)

It may not be the most polished UI out there (I still have much to learn about Qt), but it works. If you are a Simplenote user, or were waiting for a Maemo client to try it out (as I did with Evernote, Dropbox and Google Reader), check out the talk.maemo.org release topic and pipe in…

Enjoy!

4 0

So today, on a rainy day off, I set out to actually do that demo, while also recording it. I am not well versed at all in the mysteries of video recording, editing and such, so it took me quite a few trials and errors to find something that I could manage. Initially I thought it would be simpler to record a screencast on each machine, and then somehow paste them side-by-side in the same video. That failed abysmally, so I went back to filming both devices together, finger included, with my digital camera. I am not an expert photographer either, so there were problems with lighting, focusing and whatnot. Then, over the course of an afternoon, I learned plenty of interesting stuff: how to convert the huge MOV file from the camera to a more manageable AVI; how to create a file with the subtitles in it (I really didn’t want to tackle a voice overdub!); how to generate a second AVI with those subtitles “burnt into” the video; how to create a hosting account and upload the masterpiece, etc. All reusable skills, I’m sure :-)

The result is two minutes long rather than the five planned for the lightning talk in Amsterdam, but I guess it’s enough to get the gist of it. In retrospect, I’m actually relieved I didn’t have to do this “live”…

I chose the N810 for the demo because it is the smallest tablet and would block less of the laptop’s screen. Both machines are connected to a third host (10.10.10.10), the tablet on port 9000 to web2py, and the laptop on port 8000 to orbited. Both browsers (Tear and Firefox, respectively) are full screen.

Here you go: enjoy, feedback welcome!


Hyakutake demo from fpp on Vimeo.

9 1

We will conclude Season Two with a quick but much-needed reality check:

Does the system work ? Yes. Surprisingly well, one might say, for something that should not be possible by design.

Is it useful ? Definitely. Once you see it in action it tends to give you ideas for other uses: displaying digital images; a family agenda; sending short messages or reminders from outside to those at home, and so on. There is much potential in having a full-blown web app server on the remote-command side; using it just to display a menu of icons is admittedly trivial, but you can stray from that initial approach for more elaborate actions. Like HTML-scraping the target and displaying a custom version on the remote instead of the original page; forms for data entry; remote controlling a gallery; automatically displaying sites depending on the weekday and hour, cron-like. And certainly a lot more I just didn’t think of.

Fine, but did it solve my original problem ? Not really, because in the meantime an N900 joined the family, so I still have too many devices :-)

After this rather high-level view of the what and why, I plan (if there’s interest…) to follow up with a more technical, hands-on Season Three about the how: the innards of Hyakutake as an application, its various bits ‘n pieces, their installation, configuration, and even — gasp — actual code. And screenshots :-)

Any takers  ?

10 2

There weren’t that many to start with, and I already knew that my own use case was about the simplest possible. That meant I didn’t need an industrial-strength, 900-pound gorilla: just something that worked, and that I could understand enough of to use it within a reasonable time.

Quickly enough, I had it narrowed down to Orbited, which I chose, essentially, for three reasons:

• it’s written in Python, which usually means few platform-related issues and easy installation. Using it on my Windows desktop for learning and coding, then on a Linux box for production, was painless. It also meant that most examples of interfacing with a web app server also featured Python servers (generally Django), which made it easier to grasp what I’d need to do on the web2py side.
• the latest version (0.7.10) came with its own bundled STOMP server (MorbidQ). As usual, it is pronounced “not scalable and for testing purposes only”, but despite the weird name it worked perfectly for my modest needs. One less thing to worry about.
• above all, it was, at the time, the less undocumented one. There is, at least, a web site with attempts at some documentation, and a Google group to mine. It’s also well enough established that several people have written articles, blog posts and Howtos elsewhere. It was by no means a quick hit ‘n run, but it was enough to cobble together a basic understanding for my simple use case.

Orbited also provides a STOMP client library in Python, that you can use in the web app server side of things (web2py here) to send messages (to the COMET server, via MorbidQ). It just doesn’t explicitly mention it, so I must admit that I hadn’t noticed at first, and downloaded another one from somewhere else :-)

One thing that is very often lacking in documentations (in general, not just here!) is a simple diagram that helps the newbie understand what is going on, the “big picture”. So I hacked a quick one for this post, in case my explanations weren’t clear…

Hyakutake general architecture

Of course, the coloured arrows are just data flows, they don’t indicate where the three servers (daemons, in Unix parlance) are actually installed. Conceptually, all permutations are possible: web2py could be on the 770, and Orbited on the laptop. Or in reverse, if you’re in a playful mood :-). MorbidQ comes with Orbited, but could be sent to the other device, or to a third one. Then again, you can group everything at one place: on the laptop, on the tablet (if you’re in no hurry), on a third, separate device… it’s just a matter of configuration files, IP addresses and/or URLs.

In my case I used the (Windows) laptop to host all development and testing, then moved all the server parts to a Linux box that already runs my other web2py stuff, leaving the clients to be… just clients.

10 2

First, I quickly realised there would be no “cut ‘n paste, roll-my-own” here. COMET is a complex beast, covering a good handful of different techniques, relying heavily on javascript code and AJAX, and accordingly having to negotiate with browsers to work around incompatibilities. The only way someone like me could use that stuff would be through a ready-made package to encapsulate the complexity of setting up the communication channels between clients and servers, on both sides.

Speaking of which, I also discovered that I would have to add another web server to my toolbox. You see, http is a stateless protocol, by design: a client sends a request to a server, the server responds, and both happily forget about each other right away. There is normally no way for a client to stay connected to a server indefinitely, much less for a server to send data to a client that didn’t ask for anything… Pushing the boundaries of the protocol specs that hard requires active participation of the server, to say the least. So COMET servers tend to be specialized and dedicated to their function — feeding their clients. If these also have need of standard, request/response http services, they must ask somewhere else.

So, if a picture is starting to build in your mind, you may see where this is leading: if one client communicates with a “normal” web server (say, web2py), and the result of that interaction is pushed to another client by an unrelated COMET server, where is the missing link?

The answer is that the two http servers must communicate between them through a “message queue”, using a protocol which adds yet another entry to the alphabet soup: STOMP (Streaming Text Orientated Messaging Protocol). A complete COMET package will also include a STOMP server, such as MorbidQ or RabbitMQ… it sort of piles up, doesn’t it ? :-)

Finally, I learnt that the various methods used by COMET servers to stream data to their clients all had their specific pros and cons: that’s why there are several, to choose the best one for a given job. Fortunately for me, those pros and cons mostly seem to revolve around the same couple of issues: scalability and performance with many clients, and/or fine-grained control over the appearance of the content being streamed to a client. As I only intend to have one client at a time, and basically just switch it from one Web site to another, this, at least, was something I wouldn’t have to worry about.

So I set out to summarily evaluate the prominent COMET packages, and rather quickly settled on the one we’ll see next: Orbited.

12 2

I was right in that my goal is, ultimately, achievable. But I was severely off-base in my estimation of what it would take to get there.

Like Alice in Wonderland, I found myself dragged further and further down into a whole, new and unsuspected world of unknown concepts, with its own rich alphabet soup of acronyms and protocols.

You see, it turns out that what I wanted to do is just not that easy — yet. It will be, in time, because among plenty of other things, the upcoming HTML5 standard defines a new javascript API, WebSocket, which is designed to solve just that problem: bidirectional communication between client and server.

This is all still just a draft, however, and is mostly not supported by common, currently available browsers. In the meantime, the usable solutions are — self-admittedly –  a hodge-podge of clever hacks and kludges. This mixture has been brewing and evolving behind the scenes since around 2006, and goes by several names. COMET, however, seems to have caught on as the catchy nickname  for this field of Web programming, much as AJAX emerged some years ago to sum up the various uses of javascript and XmlHttpRequest to enhance web interfaces.

This is no coincidence, as most COMET techniques rely heavily on AJAX and/or XmlHttpRequest. The name itself is not an acronym, but an AJAX-related pun.

The analogy further extends to the state of the field as a whole, which is strongly reminiscent of how AJAX simmered in the years before it was known by that name and went mainstream: elite, bleeding-edge, black-magic like. Full of obscure jargon and implicit concepts. Sparse, elliptic, scattered documentation. Tutorials with more holes than Swiss cheese. Mutually contradictory HOWTOs on the same page because of rapid API obsolescence and no versioning. Browser incompatibilities … you name it, it’s there.

For the unwary explorer who crosses this kind of New Frontier in his sandals and shorts, hoping for a “quick-in, quick out”, it means long hours of honing Google searches, wandering through a maze of links, cross-referencing, second-guessing, correlating, hoping something will start making sense at some point… like climbing every dune in the Sahara to see what’s behind, hoping it’s an oasis.

Normally I would flee such a state of affairs immediately, and let it mature for some years until it reaches “normal human” levels, like AJAX did. But this itch must have been unusually strong, for somehow I stuck in there, meandered through the swamp again and again, to finally emerge with a couple of trophies.

That’s when I decided that if I succeeded in my project, it would definitely bear the name of a comet :-)

8 4

The laptop I would turn into a wall-mounted or table-top display, using an enclosure similar to any number of the “photo frame” mods you can find on the Web. Only the 12″, 1024*768, non-tactile screen would appear.

Then I would write a web app (with web2py, of course) to serve a finger-friendly UI tailored for the 770 form factor, using mostly just big icons. These icons would form a sort of (rather shallow) menu, enabling one-touch access to all the mundane Web services that get used daily or frequently around the house:

• real-time schedules for the nearby bus stop (to the kid’s school) and train station (to work)
• real-time traffic maps for the area
• TV programs for tonight
• weather for tomorrow
• movies at the local theater
• current playlist on my favourite FM station
• anything else that might come up…

So far, nothing very original. The goal that made the project interesting was that pressing an icon on the 770 would not display the resulting page on the tablet itself, but in a full-screen browser on the laptop. In essence, the tablet would be a Web remote command for the laptop, itself acting as a remote display.

This way, I thought, anyone in the family could use the system easily, the information would be displayed on a larger screen and could be viewed collectively from a distance. When idle, the laptop screen would also serve as a large wall clock.

This “remote command” stuff was a totally pie-in-the-sky notion at the time, and the prospect of doing something new (to me) was of course part of the appeal. The http protocol is such a resilient beast, and has been made to do so many unforeseen things over the years, that I felt naively confident: an hour of googling would certainly yield enough prior art, clues and similar projects to get me started, and from there I would cut ‘n paste my own.

Boy, was I mistaken… :-)

9 1

You see, I just hate to see perfectly good, if slightly obsolete, hardware go unused. At the same time, I can’t bear to part with it, and sometimes even rescue “collectors” from corporate dumpsters. So of course, I always end up with more than I can use or manage, which breaks my heart.

Specifically in this case, I have too many tablets. At the time (last summer), I had one of each: 770, N800, N810. The N810 was my daily workhorse, always with me everywhere. The N800 was the stay-at-home mobile terminal, for news at breakfast and quick domestic lookups at any time (weather, TV programs, bus schedules, etc.). Both saw daily, even intensive use. The 770… well, good old 770 mostly sat around and served to test those Mer releases that had support for it, when I got around to it, which was not saying much. I needed some other use for it.

I also had an old laptop lying around, a victim of the netbook wave. It had been an expensive ultra-light in its day, still sleek and in good working order, just not up to modern standards any more. I wanted it put into active retirement, but none of the “mod” ideas I could find on the Web (like a digital photo frame or torrent box) appealed to me.

Then one day I happened to think of both devices at the same time, and something clicked into place: I would kill both birds with one stone. I would write an app that would make them both useful again — together.

I saw it all so clearly in my mind…

10 2

If you recall, Season One was about a presentation I gave at the first (Berlin, 2008) Maemo Summit, revisited and extended. Season Two will be about a presentation I would have given at the 2009 Summit in Amsterdam… if I had actually managed to attend, that is. This way I can salvage some of it, and maybe give some readers useful tips or ideas for new uses of their tablets.

This presentation would have been quite different from the first, if only for the fact that it was to be a 5-minute “lightning talk”. This format does not lend itself well to philosophical ramblings, so I intended to do what suits it best: a demo of an application, which I had created last summer.

As is usually the case, the idea for this app stemmed from an “itch to scratch”. Of course, as you may already have guessed, there is plenty of Python and Web protocols in this story, and some web2py, but in the end I had to dig into deeper and more complex stuff than I had initially anticipated, so that is not all of it.

In hindsight, the five-minute demo would probably have been quite a challenge to set up, as it would have involved at least two devices (one of them a tablet) communicating over three protocol stacks.

Hyakutake is the name of the resulting product. For the curious, it is also the name of an atypical comet which crossed the solar system in 1996. It’s a bitch to spell, but I find it has a nice ring to it :-)

The nature of the itch I’ve been scratching, and the reason for choosing a comet’s name, will be made clear in the following posts…

8 2

I finally received my own N900, barely four days ago. I’m still getting to know it, and come to terms with the new UI, the new capabilities; also, of course, the news bugs, shortcomings and sometimes regressions that will keep us all waiting for firmware updates…

One thing I wanted to test right away, as you can imagine, is the web2py stuff I have running on my N810.

First, find the xterm icon hidden somewhere. From the way the terminal springs to life on the screen you already know this hardware is wicked fast compared to the N8x0 generation. Just typing ‘python’ confirms that the language runtime is not preinstalled, as expected.

Next stop, App manager. The ‘extras’ repository is present, but disabled by default. Enable it, update, and here is PyGTKeditor by my fellow Frenchman Benoît. It is both a very nice code editor, and the simplest way to get Python installed as a dependency.

Back to xterm : ‘python’ now works. ‘import pygame’ doesn’t, however, meaning it is no longer installed by default like it was before. It’s probably just a separate install however, as the SolarWolf pygame shooter is available for Fremantle.

So, on to the test : on the N810, I just tar and compress the web2py directory; send the .tgz file over to the flash card on the N900 (using File manager and Bluetooth); uncompress it there and run the startup script. No need to be root, change permissions or install packages.

Output in the console stops so fast that I immediately think there has been some kind of error: but no, web2py has just reached the end of its startup messages, and is waiting to serve requests. Again, this is blazing fast compared to the N810. Also, it means that the python runtime itself is more complete than on Maemo4: no import errors, no need to install a complementary “python-devel” package. Yay.

Last, the browser: even MicroB manages to feel almost zippy on this platform, despite its crazy UI. I enter the localhost URL for the slowest of my web2py apps, which routinely takes anywhere from one to three seconds between screens on the N810, probably due to lousy ‘database joins’ which I’m too lazy to refactor: here’s my good old app, except now it churns along as if it had been tuned by a professional dba… The age-old, traditional IT ‘fix’ at work: throw hardware at it :-)

So, the ‘porting” of my entire app portfolio (and their data) took all of five minutes. Looks like Maemo5 and the N900 will be a good home for them, allowing for more liberal use of CSS, javascript and Ajax. Also, distribution will be less of a problem. And the pymaemo team is still working to make their distribution better, so there’s more to come…

Next up : Season Two !

17 0

♦

As a user, you might scoff at web interfaces, and as a matter of principle prefer slick native GUIs, even for mundane-but-custom utility apps (like to-do lists, shopping lists, book lists etc.). As I mentioned earlier, it is perfectly possible and reasonable for a Python programmer to write a Maemo app, using PyGTK or PyQt/PySlide. Coding on a PC and testing on a tablet is less fluid (many visual differences), and the app, though still portable, will be more Maemo-specific; but it’s still heaps easier than regular cross-platform compilation, even if packaging is still something of a black art. There are plenty of nice and useful such apps for Maemo4.

They all share the same drawbacks, however. One is their start up time: when you launch them, you first load the Python interpreter, which in turn loads the Python bindings to the underlying binary graphic libraries (PyGTK or PyQt), and finally your own code. This tends to add up. For the same reasons, even trivial apps have large memory footprints, mostly due to PyGTK or PyQt, as these libs have huge numbers of classes.

This has a direct impact on usability: on the 770 PyGTK apps took too long to start and were too memory-hungry to be more than curiosities, even though they ran fast enough once loaded. On the N8x0 start up time is down to bearable levels, especially for apps that tend to be used for extended sessions (like Canola), or on specific occasions (like PyGTKEditor). Maybe on the N900 it may not matter, but the memory footprint will remain an issue: you don’t want to have too many of these loaded at the same time.

This is especially bad for “mundane utilities”, because if they’re any use at all, they tend to be used casually, frequently, on the spot, on a whim, quick in, quick out. If they can’t be kept active at all times, and don’t load fast either, they get forgotten quickly enough, deep down in the ‘Extras’ menu.

♦

Now contrast this with the way our Python web apps exist on the tablets.

The start-up time penality you pay only once, while the system boots, as it loads a Python interpreter and a server instance of web2py. If you’re anything like me, you only ever reboot your tablet when you absolutely have to, which means really seldom, so this is not comparable to waiting each and every time you launch an app.

Memory-wise, the cost is also one-time and permanent: one Python process, one web2py process. As there are no bindings for huge GUI libs, the footprint is again not comparable to loading just one PyGTK or PyQt application, let alone several. I have been running my N810 like this for almost two years, and there really is no impact.

The third big difference is that web2py, like most web frameworks, allows you to serve any number of different applications from a single instance: applications are just subdirectories, and are accessed through URLs with the same names. They don’t cost anything while not being used, but are still always close at hand, just a bookmark away. Most of the time you will already have a browser open for something else, so when the need for an app arises, you can pull it up in two taps, use it, and resume what you were doing before.

This mode really befits casual use of day-to-day utilities. There is no penalty in having plenty of simple, specialized apps (or applets), each for one specific task, with a common styling and UI design, and skipping from one to another as needed. A local ecosystem of web apps, in a way.

♦

Finally I’d like to thank those readers who took time to post comments, encouragements, suggestions or corrections. I’m not entirely convinced these posts are worth the time they took me to write (to the detriment of other activities), but I hope that they at least brought entertainment or ideas to some of you. Also, don’t hesitate to send me ideas and suggestions about subjects you’d like to read about here — in between the cracks.

9 2

♦

First we need a custom launcher script, for several reasons. One is that just running web2py.py in xterm launches web2py in ‘interactive’ mode: it prompts for an administrator password, then runs in the foreground, printing all output to the console. While good for a first test run, this is not what we need for background (daemon) execution. The solution is to specify the password on the command line with the relevant option. Another reason is that logging is enabled by default for the web server, which in our case is useless, will only eat up storage and slow things down. Disabling this is done by specifying an “empty” log file name. Then there is the usual ‘listener pool’ to allow several clients to connect at once; here it is 5 by default, we can scrimp a few bytes by lowering it, as there normally will be only one client. So if you have unzipped the web2py archive to the “mmc1″ memory card, your launch script would look something like this:

cd /media/mmc1/web2py
python web2py.py -a password -n 2 -l “” &

Once we have the server running, there are a couple of basic precautions we can take in coding our apps for performance. Server-side, it’s best to avoid database requests that generate complex SQL “joins”, as these seem to be costly. Browser-side, especially in MicroB, served pages should be kept light in terms of graphics, CSS and javascript.

Finally, in the TTW admin, you will see a ‘compile’ option in the app’s menu. This ‘compilation’ translates your template files (views), which are normally parsed and rendered for each request/response, into static python files which are executed directly. On the tablets this is probably the biggest speed boost of all; you should do it when your app is mostly finished, as templates are not editable in their compiled form (but you can always ‘uncompile’, edit and recompile).

♦

All this works just fine for the bricoleur, who is also at heart a coder, and not afraid of the command line. Extending the results of his work to casual users, however, still lacks user-friendliness in some spots; this could be cured with a proper Maemo installation package, in three different ways:

web2py installation: we’ve seen that Python is readily available for Maemo, and that once you have web2py running, installing web apps is a snap. The vexing part is installing web2py itself, which involves some tinkering in the console as above. A standard install through App Manager would be much better of course, especially if it came with a ready-made launcher script and a menu entry… one could even dream of a simple dialog, or Control Panel applet, to set the admin password and start/stop the daemon.

size reduction: one little quirk of web2py is that it insists on having an ‘examples’ application installed the first time it is run; it can be safely removed later but causes an error if not initially present. It also accounts for over half the size of the entire installation… Building the Maemo package from a freshly-installed, run-once web2py instance, with the examples app removed, would save space and time when installing on tablets.

python dependencies: Python is already installed on many tablets as a dependency of another app, but an installation package with a proper dependency would insure that it does get installed if it isn’t already. Moreover, on Maemo4 web2py also requires the ‘python-devel’ package, because it uses some standard library modules that the INdT team did not include in the pymaemo runtime. So that could be made a dependency too, or even better, the package itself could include the few missing python modules.

While we’re at it, we could also include templates and styles to replace web2py’s defaults, which are sometimes on the heavy side with CSS/javascript/Ajax. In a comment to a previous post, Josef Assad suggested this could lead to look ‘n feel consistency between apps, and better integration with the platform style, which is something I guess Apple achieved with its iPhone.

Anyone game ? :-)

♦

Tomorrow we’ll conclude this series with various odds ‘n ends, then take a much-needed rest while I ponder the ifs and whats of Season Two :-)

4 7

just add Python: this is the equivalent of Python’s “batteries included” mentality. If you don’t intend to use another database than sqlite (which is perfectly adequate for tablet use), then the default web2py package comes with everything you need, and only requires a Python 2.5 runtime to get going: a web server, MVC framework, DAL, templates, form building and validation, error and ticketing management, javascript library, multiple language support… There is also an administrative interface (through SSL) to create and manage apps, and optionally code, which we’ll come back to (you can try it online here).

no installation: well, none to speak of. The common cross-platform package is just a zip file that expands to a web2py directory with everything underneath, including a python script to start the server. Works just the same on Windows, a Linux box, or a tablet. While this may not be ideal for an end user (more on that later), it is great for the developer because the entire environment, including apps and data, can be copied or moved in one go. It also means that unlike normal Maemo apps, everything can be implanted in internal or external Flash memory, because no file really needs the ‘executable’ bit set: only the launcher script should be run from the Linux file system.

fast: at least fast enough to run useful apps acceptably on the N8×0 series, with some tweaks and recommendations we’ll see in the next post. I haven’t actually seen a real N900 yet, but of course I expect it to fly. It’s fun to imagine that you can write code this way that will run on almost anything: tablets and smartphones, regular netbooks, laptops and PCs, servers and “big iron”, even “in the cloud” (Amazon EC2 and Google App Engine)!

great TTW admin: through-the-web administration is nothing new, but when proprely executed, can be a real boon. Not only does web2py’s TTW admin (which is itself a web2py app) competently serve its intended purpose, it also answers a few non-trivial questions I asked before. Consider the following items as sub-sections of this one:

easy app creation and management: when you create a new application TTW, you just give it a name, and a whole file hierarchy is created, populated with default stubs and supporting files. The skeleton app actually already runs, à la Ruby-on-Rails, only without shell commands. Afterwards the admin page acts as a dashboard, a page where you can see all your app’s components (models, views, controllers), modules and methods, translations, errors/tickets, static files, etc. You can also edit files there in a javascript editor (with syntax coloring); although you’ll probably prefer writing code in your preferred editor from the file system, this sometimes comes in handy to quickly correct a typo, or tweak a remote file on a server. Changes go live immediately, for swift edit/test cycles.

easy app deployment, upgrade and backup: there are two features on the admin page that may not be eye-catching, but are real game-changers for what we want to do. One is the “pack all” command in an app’s main menu: this creates a web2py.app.appname.w2p file (actually a renamed, standard .tar file) and downloads it to your browser (or wget…). This file contains your entire app: code, supporting files, and database. See ? Instant app remote backup… The other feature is the ability to create a new app by uploading a .w2p file, or providing an URL pointing to one. Combine the two features, and you also have an instant app distribution scheme… just put up an archive of your app with an empty database on a web page, and any web2py user can install and run it by pasting a link. The “appliances” available on the web2py site are just that.

There is much more to web2py of course, but the ability to use it simply and keep things simple is an endearing trait, and should be enough to tickle your curiosity. In the next instalment, we’ll see some basic tips & tricks to make it run smoothly on our tablets, and some not-implemented-yet ideas on how some Maemo-specific tweaking could make it even better for bricoleurs and users alike.

7 4

As announced, we will use the web2py framework as a yardstick and see how it measures up, its shortcomings and how it can be tweaked for best results on the tablets. Other frameworks, of course, may be similarly tallied and found to fit the bill.

As previously, we will first look at the “human factor”, then the technical ones.

I did not end up choosing web2py through any scientific process; naturally I did not test the entire list — which is just an excerpt anyway, there are many more on the Python Package Index, and new ones appearing all the time.

Years ago I approached a similar list and started with those offerings advertising themselves as intentionally “simple and easy to use” — as you would expect from me. I even tried web.py, which truthfully considers itself the “anti-framework”, and is actually more a well-designed toolkit or library. I also used Karrigell, still maintained by a fellow Frenchman, and the now discontinued Snakelets, my favourite at the time.

The experience was initially rewarding, but ultimately frustrating : true, these tools were easy and fast to get started with, didn’t “get in the way” — in the sense of imposing coding style and structure. But in this case, “simple/easy” really implied “limited in scope”: after a while I found myself repeatedly implementing low-level stuff that had nothing to do with my application logic, and that I thought should really be provided by the framework.

So when the next project took form I thought I should take a look at those “full-stack” solutions — with some trepidation, because I knew full well that if 900-pound gorillas like Django were justifiably #1 for their intended use, they were also total overkill for my own needs, and probably way over my head. I went at it on gut-feeling alone: skimming the web sites, the docs, the code samples, the forums… jumping board if after a reasonable effort my head swam or my eyes glazed… waiting for something to “click”.

web2py was the first one to click, so I stopped there. It had the qualities in like in Python itself : not intimidating, easy to get started with, but full-featured and not constrained; clean syntax, plenty of learning material, a buzzing and tolerant community.

I guess it helps that the author, Massimo di Pierro, is a University professor, and first designed web2py as a teaching tool for his students (somewhat like Andy Tanenbaum with Minix, in the old days). Also, he did it in typical academic style, top-down: first the specs, then the API, then the implementation. His tenet is that code may come and go, but the API and backward compatibility are “sacred”. This is in stark contrast with most grass-roots, bottom-up community projects, where agility and features rule, and drastic refactoring is allowed to break existing code.

In the life of an in-between-the-cracks tinkerer, knowing you can upgrade your app engine to the latest version with the guarantee that your code will still run, is a Good Thing™.

So, off I went. I guess that with any framework (not only web ones), the most difficult step at the start is understanding how it wants you to do things: if you fight it, you fail. For bricoleurs, the cost of that step might be a good metric of the framework’s fitness-to-purpose. With web2py, the compact API, and the variety of simple-to-complex examples, made it reasonably painless to get past that point.

After that it was like discovering Python so many years ago : first overdoing things out of habit, later understanding “the web2py way”of doing them, and marvelling at what little code is left.

With philosophy out of the way, we’ll take a deeper look at the technical merits of web2py… after the weekend !

7 3