Planet maemo: category "feed:db85272b5cc4c48c874836930690af4e"


N9, QML, Javascript and differential update.

2012-03-12 00:59 UTC  by  monkeyiq
I had a server app that had both a desktop and web interface. Though I discovered that the web interface was less than monkey finger friendly on the n9 and required some authentication each time. I thought this was a job for QML and javascript on the device. The good news is that I now have a client using just those tools, so no cross compilation needed at all ;)

The core of the UI is updated via a timer which runs an ajax call, parses the JSON response and gives that to updateModel(). The first implementation of the later method was to clear the listModel and repopulate from scratch each time. Rather wasteful, even though it was only being called once every 2-3 seconds, it was eating up 8-9% CPU of the n9 in top. A better solution is to iterate the existing items, performing update, remove, or add as you go. This needs a "key" field for each tuple and a stable total order on the model JSON data, either from the server or through an explicit client side sort.

I'll probably put this up on github soon, but in the meantime, this is the updateModel() which has the smarts I mentioned above. It takes 3-4% of CPU instead, and should maintain view state better on the device. I use objid as the key, and the "state" is cloned to objstate because I noticed some issues there with QML interaction using state as the property. Interesting what one can do with QML+JS in very short time for the n9.

function updateModel( model, data )
var existingIndex = 0;
var existingIndexMax = model.count;
var id;

for( id in data )
var found = false;
var name = data[ id ];
var col = data[ id ];
var k;
var row = {};
row[ "objid" ] = id;
row[ "name" ] = "v";
for( k in col )
if( k == "state" ) {
row[ "objstate" ] = col[k];
else {
row[ k ] = col[ k ];

// find it if it exists...
for( ; existingIndex < existingIndexMax; ++existingIndex )
if( model.get( existingIndex ).objid == id )
found = true;
model.set( existingIndex, row );
// This element from the local model no longer exists on the server,
// remove it. Do not advance the index as we have removed an element instead.
model.remove( existingIndex );

// it didn't exist, add it as new.
if( !found )
model.append( row );
Categories: kde

Libferris on both arms

2012-03-08 04:07 UTC  by  monkeyiq
I now have libferris on a 512mb ARM5t embedded device and a 1gb arm7 one (the Nokia n9). As part of the fun and games I updated the clucene and boostmmap index modules for libferris and created a QtSQL module which uses SQLite by default. Hey, when you can't choose which cake to have why not have a slice of them all... or a slice to fit the moment or device at hand. eg, desktop or server works nicely with the soprano or postgresql indexing modules, embedded is nicer on mmap or clucene.
Click to read 1012 more words
Categories: gcc

The Nokia N9: Going for a spin

2012-02-26 15:55 UTC  by  monkeyiq
In short, I now have a Nokia n9, and setup a scratchbox for it, and compiled & installed abiword, clawmotia, unison, and libferris for and on the device. If that didn't enlarge your page down key, the longer version now follows ;-)
Click to read 1364 more words
Categories: abiword

Mounting statusnet/identica as a filesystem

2012-02-22 17:13 UTC  by  monkeyiq
With the 1.5.12 release of libferris hot off the presses, I thought I'd blog about how to use the new statusnet mounting. I tried to do some testing against identica at the start, and managed to get myself silenced there ;) I was resurrected within hours once I put forward my case as not-a-trollbot^tm. Anyway, that led me to setup my own local statusnet server for testing against, and now libferris can mount and also one or more local servers if you so desire.... yay!

There is a little bit of setup required, web keys, secrets and oauth... the normal story.
The identica filesystem contains a directory for each server you have setup in ~/.ferris/identica. To update my public status I just write to the status file in my identica account:

$ echo "this is coming from the console via libferris, mounting the public statusnet ;-p" \
| ferris-redirect identica://identica/status

To see what has been going on I can list the timeline directory, using "-0" to get libferris to tell me what metadata is interesting for that directory instead of viewing the protection bits and other non interesting information.

$ fls -0 identica://identica/timeline
89868729 Welcome to, @monkeyiq! Welcome Bot 12 Feb 2 03:53

There is also support for direct messaging through the filesystem, seeing retweets, friends, and mentions.

Now for the details... Configuration happens through the virtual filesystem. Each statusnet server you want to mount has it's own directory inside ~/.ferris/identica. The configuration directory lists the base URL for performing REST requests on that server and the prefix of where the webkey and secret are to live. For the public server I have the following:

$ pwd
$ cat rest_base
$ cat token_prefix

And the webkeys go in a file in ~/.ferris prefixed with the token_prefix above:
$ cat ~/.ferris/identica-api-key.txt

I do not distribute web keys, so you need to go to and create one yourself.

If you have a private statusnet server foobar then you create ~/.ferris/identica/foobar with the REST url for it, and foobar as the token_prefix. Then put the web API key into ~/.ferris/foobar-api-key.txt.

To setup the oauth part, we use our old friend capplet-auth. Again for our foobar custom server replace the auth-with-site with foobar instead. This will give you a URL to visit to authorize your webapp (libferris) on your account. You'll be given a PIN to copy back onto the console and then you should be setup.

$ ferris-capplet-auth --auth-service identica --auth-with-site identica
Categories: kde

Searching again (400ms -> 2ms)

2011-05-02 03:12 UTC  by  monkeyiq
The problem statement: you want to search for a file using any arbitrary substring contained in the file url. This is deceptively hard, if you think in a relational db way, you want something "like %foo%" where the leading percent will force indexes to be considered redundant and a sequential scan to ensue. If you try instead to break up the file's URL into words and use a full text indexing solution you will find your old friend the leading percent again, like here for Lucene.

One easy way to use full text indexing style and allow substrings is to cheat. First disable word stemming (ie, dont turn diving and dive into a stemmed "dive" word). Then for each word to be added to the full text index, shift it left from 1..10 times. So wonderful, onderful, nderful, derful, erful etc are all added as words for the same URL. This is most simply accomplished using a postgresql function to do the shifting.

Then when you search for a substring foo you want to find to_tsquery('simple,'foo:*') as a fulltext query on your url column. As prefix searches are allowed in fulltext index engines like Lucene and postgresql's TSearch2 engine, this evaluates quite quickly.

For 200,000 URLs when using the raw regex match "~" in postgresql you might see 400ms evaluation times, the same data using a gin index on to_tsvector('simple',fnshiftstring(url)) might return in 2-3ms. Of course the index can't always handle your regular expression, but if you can tease out substrings which must be present from your regular expression, a shifted gin index could drop evaluation times for you. eg, .*[Ff]oob([0-9]... could use a shifted search for "oob" as a prefilter to full regular expression evaluation.

Getting down to a few ms evaluation allows GUIs to return some sample results as you type in your regular expression. This speed up will be available in 1.5.6+ of libferris.

A while ago I released an engine for maemo with statistical spatial indexing for regular expression evaluation. It's an interesting problem IMHO, and its also a very common one.
Categories: desktop search

Change Tracking: Why?

2011-04-24 17:40 UTC  by  monkeyiq
As some of my past posts have mentioned, the OASIS group is currently working out how it wishes to extend support for change tracking in ODF. The change tracking feature allows you to have an office application remember what changes you have made and associate them with one or more revisions. There are many examples where governments might want such traceability, but small businesses are likely to find this functionality valuable too.

Late last year there was an ODF plugfest held in Brussels where it was decided that an Advanced Document Collaboration subcommittee should be formed to work out how to serialize tracked changes into ODF.

There are currently two proposals. One which is generic and tracks how the XML of the ODF is modified over time, and an extension to the limited change tracking that already exists in ODF 1.2.

As my previous posts have mentioned, I'm involved in making abiword able to produce and consume ODF files which contain tracked changes in the format of the generic proposal.
See my github for abiword and its change tracking test suite. Ganesh Paramasivam has been working to make Calligra and KOffice support the generic proposal too, and Jos van den Oever has done some hacking to enrich WebODF for change tracking.

As I have been going along implementing things in abiword, I've been clarifying some points on the oasis office-collab mailing list, and participating in some of the general conversation there too.

Some of the differences between the two proposals can have a large impact to the complexity of implementing change tracking in applications. Folks who are involved in office applications which use ODF might like to read up on the proposals and have their say on the future of this feature before a decision is made for you.

I think it is important for ODT to include a good, complete change tracking specification because it would offer all people the ability to collaborate on documents and businesses the change to know the exact extent of modifications made to documents by each party.
Categories: Change Tracking

From Canon CR2 Raw files to the Web

2011-01-20 21:28 UTC  by  monkeyiq
A potentially interesting artefact of me playing around with a Canon DSLR is that libferris now has some initial support for CR2 "raw" files. Complements of course go to the dcraw library and tools for the heavy lifting. The upshot is that ferriscp ~/cam/foo.CR2 flickr://me/upload now works as expected. Once I update my maemo build, this should also let me use an n900 with 3G to upload from RAW while on the move.

I need to include stuff for ICC profiles and the like but at least the Web services get what they expect and everything runs smoothly, and it will for you too next release (TM). I really should look into mounting the kipi plugins as a filesystem sometime, they offer some awesome Web import/export functionality, ripe for a filesystem I tell's ya.

I still need to add "rgba-32bpp" metadata support for CR2 files. As you can imagine from the name, if you read this metadata for a jpeg, png or other supported image file, libferris decodes the image to 32 bit RGBA byte values for you. It might also be interesting to create another "file as filesystem" design where a bunch of virtual files are offered to see the small thumbnail, the roughly 1080p sized thumbnail or the ICC corrected rendering of the CR2 and others. Such virtual files are really handy for drag and drop to other filesystems ;)

I might hunt around for some Nikon raw files to add support for those too at some point. Though there is nothing like having a camera generating files to make you want to make your software support those files.
Categories: cr2

Sharing is Caring...

2010-11-15 15:46 UTC  by  monkeyiq
My repository now contains a steaming ferris-sharing-plugin_0.2_armel.deb which will hook up libferris to the Nokia sharing system on the n900. This lets you click the funky Brussels atom building "sharing" icon from many places and copy images to flickr, 23hq, facebook.

There isn't much feedback at the moment, that is planned for "the future". You might also like to install the ImageMagick from my repo. Simply expand it to /opt and make a link /usr/lib/ImageMagick-6.6.0 -> /opt/ImageMagick-6.6.0/lib/ImageMagick-6.6.0. This gives you proper handling of exif orientation during scaling prior to upload with libferris.

I also have some tweaks to libferris itself which will be in the upcoming 1.5.1 release to help with annotations etc during upload. The next step is to get vimeo and youtube upload support happening too, since libferris can already upload to those.
Categories: ferriscp

Pop Quiz Hot Shot - ssh

2010-11-12 14:17 UTC  by  monkeyiq
You have ssh'ed into a host via three others and just noticed a file(s) you would like on the laptop you started all this merriment from, what do you do? Of course you don't have a control connection or tunnelling already setup because its a "once off" multi hop connection.

A solution that is not quite as widely known as I'd thought is to use the sz and rz pair. Hurrah the zmodem programs are still just as useful today as they were in the years of yore. The snag is that your local terminal needs to have some knowledge of this, konsole works nicely and will pop up a dialog asking where to save files "sent" to it. So doing ssh server; date >/tmp/df; sz /tmp/df will result in a dialog appearing on your local display asking where to save the df file to. Note that this works over multiple intermediate ssh hops. Just perfect for when you have routed your way into a protected network via 4 hops and find out that /etc/foo.conf which is 8kb would be *really* nice to have on the laptop.

This is packaged as lrzsz for Fedora and I've made an lszrz tarball available in my n900 repository. Let the zmodem goodness rain down...
Categories: kde

Clawmotia and QML: Moving the TV

2010-11-08 18:20 UTC  by  monkeyiq
Some may know that Clawmotia is my MythTV remote control which started back in the n810 days. For a bit of fun while recovering from jet lag I decided to port Clawmotia over to using QML and take advantage of some of the n900's capabilities such as the accelerometer. Shown below is controlling the playback speed by moving the device. Apologies for the flicking video, it seems mythtv doesn't like the mov file in software playback much, but the Bunny is free to use so he made the audition.

Controlling MythTV with an n900 accelerometer from Ben Martin on Vimeo.

The normal functions from the older qedje remote are also available on the main page shown in the video below. I have now put the cursor keys into a submenu which is opened by both the joystick and menu buttons, the latter first getting mythtv to open the menu and then changing the remote configuration to show the cursor key submenu. This explains the vacant positions in the grid in the new design -- submenu shuffle has given me a few extra spots to play with. Also the settings can be set in the program itself now instead of the old edit the launcher file and set environment variables. I would like to make the volume adjustment use a popup slider and also route the hardware vol +/- keys to mythtv, but I'm still working out how to do the later.

Clawmotia QML: The Qt MythTV remote control from Ben Martin on Vimeo.

Interestingly, the big "oh no" moments came when I tried to do trivial things from QML such as reading a file (the accelerometer in /sys). I would have thought that QFile was available but it seems not. Perhaps it is from dedicated javascript files or I'm just doing it wrong^TM. I also notice that the simulator doesn't update that /sys file so you have to test that on the device :/

I also bumped into other things like hooking up signals bidirectionally which would need pass by reference or similar to make it useful in custom QML elements. I might also make some data bound classes to use with QSettings, it seems like an idea for a QString like class to update the QSettings automatically when you assign new data to it.

I also tinkered in Nokias SDK environment for this but reverted to emacs and js-mode because there are just too many things my hands are used to happening that don't in the SDK editor. Though for QML stuff I suspect I need to update my elisp to have better "add property" support so the get/set/signal/member declarations are all added from a "QString foo" input. I have part of that done but need to revisit it.
Categories: accelerometer

n900: The claw arrives!

2010-09-25 17:44 UTC  by  monkeyiq
Clawmotia is a MythTV remote control written using Qt and EFL technology. My repository contains a tarball ready to install clawmotia onto an n900. Desktop shortcut optional :)

Simply extract the tarball into /opt on the device. From there, you currently have to install the desktop file yourself to get it onto the menus. Finally, you'll need to edit to tell the application where your mythtv server is at and which client you want to control through that server. I plan to add dynamic client selection and a user configuration window in the app itself, but I seem to have more plans than the clock allows.
Categories: clawmotia

Camera to Web the eagle has landed!

2010-09-22 15:16 UTC  by  monkeyiq
Pop quiz, hot shot; an n900 sits before you and something wonderful is happening in the distance. You've love to take advantage of the rear camera and share images of the wonderful event on any flickr API website or some sort of facial book site. What do you do? The camera has all sorts of custom APIs to access it, and the web sites all need some authentication and use REST.

Well, for those who know me, or of me, the answer is quite simple... just "cp" the data from your camera to the Web. And I've now got this working on the n900. I tried the front camera too, but unfortunately it comes out quite underexposed on in house shots.

To read the camera, libferris uses gstreamer. Because there are a whole bunch of ways you might like to read information, frame rates, transcodings etc, you use a file in your ~/.ferris directory to tell libferris how to create virtual files for your input hardware like so:

cat ~/.ferris/gstreamer.xml
<file name="live.jpg">
v4l2src num-buffers=1
! videorate ! video/x-raw-yuv,framerate=\(fraction\)3/1
! ffmpegcolorspace ! jpegenc
! appsink name=sink

Those with a keen eye will notice that the live.jpg is generated with the same thing you can use in gst-launch to create a jpeg image. The final element is an appsink which is where libferris gets the bytes from. I made that last element explicit because you can tee the data in gstreamer and do other things to have multiple end points.

Then to take a photo and put it up on the net:

$ fcat gstreamer://capture/live.jpg >| /Card/tmp/out.jpg
$ ferriscp gstreamer://capture/live.jpg 23hq://monkeyiq/upload

Of course, the fcat line above hints that you can use ssh to grab an image from the n900 to your laptop or server and upload it from there. See the ferris-capplet-auth for how to authenticate with the web sites and also how to have libferris scale before upload (optional) and what privacy settings to add during upload.

Next up is using the vimeo and youtube mounting in libferris to stream video to the net.
Categories: camera